How to Evaluate AI Assistant Quality Across Languages
This guide is for whoever owns quality on a store that sells in more than one language: the founder, the ecommerce lead, or the engineer wiring up the assistant. It covers what breaks by language, the metrics that catch it, the techniques and tools to measure it, and the specific things that change when the model is a remote LLM you call over an API instead of one you host. WisWes runs this loop across storefronts in several languages, and the playbook below is the one we use.
Why English metrics lie in other languages
Almost every eval habit in the industry was built in English and quietly assumes it. Golden sets are written in English, rubrics reason about English grammar, and the judge model is strongest in English. Point that machinery at a German or Portuguese conversation and it keeps returning confident scores — they are just measuring the wrong thing, or nothing at all.
The result is a blind spot with real revenue attached. Non-English traffic is often your fastest-growing segment, and it is precisely the segment your instrumentation was never designed to see. Four failure modes live entirely in that blind spot:
| What breaks | What it looks like | Why English eval misses it |
|---|---|---|
| Understanding | Misreads an inflected or agglutinative query (Finnish, Turkish, Ukrainian), or code-switched text mixing English product names into another language. | The English test set contains no morphology-heavy or code-switched inputs, so the failure never appears in the sample. |
| Register & formality | Uses du where Sie is expected, tú instead of usted, or the wrong Japanese politeness level — reading as rude or unprofessional. | English has no formality distinction, so no English rubric even has a field for it. |
| Grounding | Hallucinates more in lower-resource languages, or answers from a wrong-language retrieval hit that sounds plausible. | Faithfulness is only checked against English context, or retrieval mismatch is invisible when everything is one language. |
| Language behaviour | Replies in English to a non-English question, or starts in the right language and drifts to English after a few turns. | When the whole test is English, “replied in English” can never be wrong. |
None of these show up as a crash or an error code. They show up as a shopper who quietly closes the tab — which is why they have to be measured deliberately rather than waited for.
The metrics that matter — per language
The single most important word in multilingual evaluation is segmented. Every metric below is worthless as a global average and valuable the moment you slice it by detected language. Group them into three families.
| Family | Metric | What it catches by language |
|---|---|---|
| Business | Resolution / containment rate | Share of chats handled without a human — the headline that most often diverges sharply between languages. |
| Checkout / conversion rate | Whether the assistant actually helps that language’s shoppers buy, not just chat. | |
| Escalation & thumbs-down rate | The cheapest early-warning signal you have; spikes in one language before anything else moves. | |
| Quality | Faithfulness / groundedness | Answer supported by the retrieved context — the direct measure of hallucination, and usually worse outside English. |
| Answer relevance | Whether the reply addresses the actual question, which degrades when understanding degrades. | |
| Language correctness | Replied in the user’s language, start to finish. A hard, countable pass/fail. | |
| Register / formality | Correct politeness level for the language and market. Invisible in English, load-bearing everywhere else. | |
| Operations | Latency p50 / p95 | Non-Latin scripts tokenize longer, so identical conversations run slower — track the tail per language. |
| Tokens & cost / conversation | The same chat can cost 1.5–3× more in Cyrillic, Arabic, or CJK. Budget per language or be surprised. | |
| Retrieval hit-rate | Did RAG return same-language, on-topic passages? Low hit-rate here is the root cause of most grounding failures. |
Techniques: how to actually measure it
1. Golden sets, authored natively — never machine-translated
A golden set is a fixed list of representative inputs with known-good expectations. The temptation is to write one in English and translate it into every target language. Resist it. Machine translation produces clean textbook sentences that no real shopper types; it launders out the slang, typos, regional wording, and English-brand-name code-switching that are exactly where assistants fail. Author 50–100 native items per language per top intent, or better, sample them from real conversations. A native set is harder to build and worth ten of the translated kind.
2. LLM-as-judge — the production workhorse, used carefully
You cannot pay humans to read every conversation in every language, so a strong model grades the answers instead. This is the technique that makes multilingual eval affordable — and the one most often done wrong. Five rules keep it honest:
- Judge in the target language. Give the judge the conversation and a rubric in the same language, and ask it to score specific dimensions — faithfulness, relevance, language correctness, formality — each with a one-line justification, not a single vibe score.
- Write the rubric with a native speaker. A rubric that reasons about English politeness is useless for German Sie or Japanese keigo. The rubric is where language expertise gets encoded.
- Validate the judge per language. Measure its agreement with native human labels (Cohen’s kappa). Trust it above ~0.7; below that, demote it to a triage tool that only picks what humans review.
- Match the model to the job. A strong model (Claude Opus 4.8 or Sonnet 5) for the graded rubric; a cheap one (Claude Haiku 4.5) for high-volume online scoring where you only need a rough signal.
- Remember the judge has the same blind spot. It is also weaker in low-resource languages. A judge you trust in French may be unreliable in Ukrainian — which is why per-language validation is not optional.
3. Reference metrics where you have references
For narrow, translation-shaped tasks — a fixed FAQ answer, a localized product blurb — you may have a known-correct reference string. There, classical metrics earn their keep: COMET and chrF for translation quality, BERTScore for semantic similarity. They are fast, cheap, and deterministic. They are also useless for open-ended chat, where there is no single right answer — so reserve them for the reference-having slice and use the judge for everything else.
4. Native humans, on the disagreements only
Human review does not scale, but it does not have to. Point your small budget of native-reviewer time at the highest-value cases: where the judge and an automatic signal disagree, where scores are low, and where confidence is thin. Those reviews do double duty — they fix real problems and they continuously re-calibrate the judge that grades everything else.
5. Online signals — the eval that runs itself
Live traffic emits quality signals for free if you bucket them by language: thumbs-down rate, conversation abandonment, repeated rephrasing (a strong proxy for “it didn’t understand me”), language-switch events, and escalations. None of these require a golden set. All of them move first in the language that is breaking.
The tooling
You do not need to build all of this. A production multilingual eval stack is four layers, each with mature open-source options:
| Layer | What it does | Representative tools |
|---|---|---|
| Tracing / observability | Capture every production turn with its retrieved context, detected language, tokens, and latency. | Langfuse, Arize Phoenix, OpenTelemetry |
| Offline eval suites | Run golden sets and LLM-judge rubrics as a repeatable, versioned test run. | Ragas, DeepEval, promptfoo |
| Language ID | Detect the language actually used, per turn, to tag traces and flag mismatches. | fastText lid.176, CLD3 |
| Reference metrics | Score reference-having tasks deterministically. | COMET, chrF / sacreBLEU, BERTScore |
The glue between them is your own per-language dashboard: a scorecard that joins the business, quality, and operational metrics on a single language key so you can read one column per language and spot the outlier at a glance.
What remote LLMs change
Most teams now call a hosted model over an API rather than running their own. That is the right call for cost and capability, but it reshapes evaluation in four concrete ways.
| Remote-LLM reality | Consequence for evaluation |
|---|---|
| You can’t see inside the model | All measurement happens at the API boundary. Instrument the request, the retrieved context, the response, the detected language, tokens, and latency — that trace is your only ground truth. |
| The model changes under you | A provider update can lift English and quietly regress Ukrainian overnight. Keep a frozen per-language regression set, pin versions where you can, and re-run on every model change. |
| Tokenization skews by script | Cost and latency for an identical conversation differ by language, so SLOs and budgets must be per-language, not global. |
| The judge is remote too | Your LLM judge is the same kind of black box with the same per-language weakness. Validate it against human labels in each language before you trust its scores. |
There is a fifth, quieter one: data residency. Evaluation logs are full of real customer text, so where you store and route them — especially for EU shoppers — is a compliance decision, not just an engineering one.
The tokenization point deserves a picture, because it is the one that turns up on the invoice. The same sentence, translated, does not cost the same to process:
Data drift: your eval decays even when the model doesn’t
There are two ways multilingual quality rots over time, and teams constantly confuse them. Model drift is the remote LLM changing under you. Data drift is the opposite: the model holds still while your traffic moves — new products, a seasonal rush, a promo that introduces unfamiliar terms, a freshly onboarded market, or simply shoppers phrasing things in new ways. Either one quietly pulls live quality away from what your eval last measured, and they need different fixes.
| Model drift | Data drift | |
|---|---|---|
| Cause | Provider updates the hosted model. | Live inputs move away from your golden set and your index. |
| Symptom | Scores shift overnight, often in one language only. | Scores erode gradually; retrieval hit-rate and rephrase rate creep up first. |
| What catches it | A frozen regression golden set, re-run on every model change. | A rolling set refreshed from recent real traffic; input-distribution monitoring. |
| Per-language twist | Low-resource languages regress hardest and least predictably. | Thin-traffic languages drift undetected longest — too few samples to notice. |
The multilingual sting is in that last row. A newly added language is, on day one, entirely drift relative to your English-centric assumptions — nothing in your history represents it. And your smallest languages, the ones with the least traffic, are exactly the ones where drift hides longest, because you never accumulate enough samples to see the trend until a quarter has gone by. So you monitor the inputs, not just the outputs: track the embedding distribution, the rate of never-seen terms, the language mix, and retrieval hit-rate — per language — and refresh golden sets from live traffic on a cadence rather than freezing them once and trusting them forever.
Here is what model drift looks like when you are watching per language: a provider update lifts nothing you notice in English and knocks a hole in another language — until your regression set catches it and a refresh pulls it back.
Putting it together: the production loop
None of the pieces above matter in isolation. The value is in the loop they form — a cycle that runs continuously and turns live traffic into per-language quality you can act on.
What running this across languages actually taught us
Reading about multilingual eval and living inside it are different things. Running WisWes across storefronts in English, several European languages, and Cyrillic- and Portuguese-language markets, the same lessons kept arriving — usually the hard way. Five of them changed how we build.
- Retrieval mismatch was the biggest silent killer. Before we split the index by language, a non-English query could match a confident-sounding passage from a different language’s content and answer from it. Nothing errored. The reply just quietly cited the wrong thing, and only a native reader would ever catch it. Per-language collections removed a whole class of failure we had been chasing symptom-by-symptom.
- Formality is a bug English can’t see. An assistant can be accurate, grounded, fast — and still address a German shopper with the familiar du where the market expects Sie, reading as faintly rude the entire conversation. No English metric has a slot for it, so it only surfaced when a native reviewer flagged it. It is now an explicit rubric dimension.
- The judge needed its own eval. We trusted an LLM judge because it was reliable in English and French, then found it markedly softer in a lower-resource language — waving through answers a native speaker rejected. Validating the judge per language, not just the assistant, stopped us from grading with a broken ruler.
- Machine-translated test sets flattered us. An early translated golden set scored beautifully and predicted almost nothing about live traffic, because real shoppers write with slang, typos, and English brand names dropped mid-sentence. Native-authored and traffic-sampled sets were harder to build and immediately more honest.
- A model update can move one language and no other. The drift above is not hypothetical in spirit: the failure mode where English holds steady and a single language slips is exactly why we keep a frozen regression set per language and re-run it on every model change, rather than trusting one global score to notice.
How WisWes runs it
The WisWes approach starts one layer earlier than most, at retrieval. Each storefront’s catalog and FAQs are indexed into a separate vector collection per language, so a query is always searched against same-language content. That design choice removes the most common silent multilingual failure — a confident answer pulled from the wrong-language index — and it is what makes per-language faithfulness measurable in the first place.
On top of that, a dedicated evaluation model grades the assistant’s answers, the serving model is configurable, and quality, business, and operational metrics are tracked per storefront and language rather than blended into one global figure. The goal is simple: no language gets to hide inside the average. If you want to see the cost side of the same usage, the AI agent cost calculator models it per conversation, and how to A/B test a conversational agent covers the flip side — proving a change is an improvement once your eval says it might be.
Frequently asked questions
Can I use machine-translated test sets to evaluate other languages?
No. Machine translation produces translationese — grammatically clean text that no real shopper writes. It strips the slang, code-switching, misspellings, and regional phrasing that actual users bring, so an assistant can score well on a translated set and still fail on live traffic. Author golden sets natively, or sample them from real conversations in each language.
How do I measure whether the assistant replied in the right language?
Run automatic language identification (fastText or CLD3) on both the user turn and the assistant turn, then compare. A mismatch — a German question answered in English, or a reply that drifts to English mid-conversation — is a hard failure you can count directly, no LLM judge required. Track it as a per-language metric called language correctness.
Is LLM-as-judge reliable in low-resource languages?
Less reliable than in English, so you must validate the judge per language before you trust it. Have a native reviewer label 50–100 items, then measure the judge’s agreement with those labels (Cohen’s kappa). Above ~0.7 you can lean on it for that language; below that, keep a human in the loop and use the judge only to prioritise what humans review.
Which metrics matter most for a multilingual shopping assistant?
Three per-language families: business outcomes (resolution rate, checkout rate, escalation rate), answer quality (faithfulness to retrieved context, relevance, language correctness, and register/formality), and operations (latency and cost per conversation, retrieval hit-rate). The non-obvious ones — language correctness, formality, and retrieval hit-rate — are exactly the ones English-only dashboards never surface.
Why does the same assistant cost more per chat in some languages?
Tokenization. Most tokenizers were optimised for English, so the same sentence can consume 1.5–3× more tokens in Cyrillic, Arabic, or CJK scripts. More tokens means higher API cost and higher latency for an identical conversation, which is why cost and latency budgets have to be tracked per language, not as a single global number.
How often should I re-run multilingual evals with remote LLMs?
Continuously in production plus on every model change. Remote providers update models under you, and an update can shift quality in one language while leaving English untouched. Keep a frozen regression golden set per language and re-run it whenever you change model, prompt, or retrieval — and monitor live per-language scores against a rolling baseline so silent drift triggers an alert.
What is the difference between data drift and model drift for an AI assistant?
Model drift is the remote LLM changing under you when the provider updates it — quality can shift overnight, often in one language only. Data drift is your live traffic moving away from what your eval assumes: new products, seasonal queries, a new market, or shoppers phrasing things in new ways. Model drift is caught by a frozen regression golden set re-run on every model change; data drift is caught by refreshing your test set from recent real traffic and monitoring the input distribution — retrieval hit-rate, never-seen-term rate, and language mix — per language.
What is the minimum viable multilingual eval setup?
Detect and tag the language on every production turn, keep a 50-item native golden set per top language, score it with a validated LLM judge on faithfulness plus language correctness, and watch three live per-language numbers: thumbs-down rate, escalation rate, and checkout rate. That is enough to catch the failures that matter before customers do.
How does WisWes handle retrieval across languages?
WisWes indexes each storefront’s catalog and FAQs into a separate vector collection per language, so a query is always searched against content in the same language it was asked in. That removes the most common silent multilingual failure — a right-sounding answer retrieved from the wrong-language index — and makes per-language faithfulness measurable.