Imagine deploying a medical AI assistant in Beijing that scores 95% on English licensing exams but fails basic triage questions in Mandarin. Or picture a coding copilot that writes perfect Python when prompted in New York but hallucinates variable names when asked in Berlin. These aren't hypothetical nightmares; they are the daily reality for engineers trying to scale large language models beyond the Anglosphere. While Large Language Models (LLMs) have mastered English, their performance often crumbles elsewhere due to skewed training data and culturally blind metrics. If you're building global products, relying on translated English benchmarks is like checking your car's speedometer while driving in reverse-you might get numbers, but they don't reflect reality.

Why English Benchmarks Fail Globally

Most developers assume that if a model works in English, it will work everywhere. That assumption is dangerous. The core issue isn't just vocabulary; it's the massive imbalance in training data. English dominates high-quality web crawls, curated books, and academic papers. When an LLM encounters a low-resource language or even a high-resource non-English language like Japanese or German, it lacks the nuanced exposure needed for true proficiency. Industry leaders like LILT report consistent quality drops in enterprise deployments. They see mistranslations of idioms, broken honorifics in Asian languages, and factual errors in localized contexts. It’s not that the model "doesn't know" the language; it knows it poorly compared to its English counterpart.

This degradation stems from technical hurdles too. Tokenization plays a huge role. Languages with complex morphology, such as Finnish or Turkish, or those using non-Latin scripts, often require more tokens to represent the same meaning. This increases sequence length, complicating attention mechanisms and making it harder for the model to maintain coherence over long contexts. Furthermore, many existing evaluations rely on machine-translated prompts. If you translate an English logic puzzle into Arabic, you lose the cultural context and idiomatic phrasing that native speakers expect. You’re testing translation ability, not reasoning capability in that language.

The Menlo Framework: Measuring Native-Like Quality

To address these gaps, researchers developed the Menlo framework, a second-generation approach to multilingual evaluation. Unlike traditional benchmarks that check for simple correctness, Menlo focuses on "native-like audience design." It evaluates whether an output feels natural to a specific locale, considering tone, politeness strategies, and local cultural references. The framework covers 47 language varieties and uses over 6,000 human-annotated prompt-response pairs. This scale allows for granular insights that broad-brush tests miss.

Menlo uses a four-dimensional rubric to judge responses:

• Language Quality and Coherence: Is the grammar correct? Does the flow make sense?
• Cultural Alignment: Does the response respect local norms and idioms?
• Localized Factual Correctness: Are facts accurate within the local context (e.g., local laws, geography)?
• Style and Helpfulness: Is the tone appropriate for the user relationship?

Crucially, Menlo demonstrates that zero-shot LLM judges-models used to grade other models without fine-tuning-struggle with this nuance. They perform better when comparing two responses side-by-side (pairwise evaluation) rather than scoring them individually. However, even then, they lag behind human annotators. To fix this, researchers trained reinforcement learning (RL) judges using human feedback. These RL-trained judges approximate human preferences much more closely, acting as generative reward models that can guide future LLM updates toward better multilingual performance.

Domain-Specific Stakes: Medicine and Code

General conversation is one thing; professional domains are another. In medicine, the stakes are life-and-death. Researchers tested LLMs on the Chinese National Medical Licensing Examination (NMLE). The results were stark. Models that passed English medical boards with ease often failed Chinese equivalents. Why? Because medical terminology doesn't always map 1:1 across languages. Abbreviations, local drug names, and regional treatment guidelines differ significantly. A model trained heavily on English PubMed articles might misinterpret a Chinese clinical vignette because it lacks grounding in local medical practice.

In software development, the problem shifts from safety to functionality. The "LLM of Babel" study from Delft University analyzed how models handle code tasks when prompts are given in non-English languages. Developers often describe bugs or features in their native tongue. The study found that models frequently misinterpret technical terms when they appear in non-English contexts. For instance, a French developer asking for a "variable globale" might get code where the variable name is literally translated, breaking convention. The researchers created an error taxonomy to categorize these failures, ranging from partial implementation to complete misinterpretation. They also used cosine similarity between embeddings to measure how well the generated code matched expected outputs, revealing that semantic drift increases significantly outside English.

Best Practices for Global Teams

If you're evaluating LLMs for a global audience, stop translating English datasets. Start building native ones. First, hire native speakers as annotators. They catch subtle tone issues that automated metrics miss. Second, use pairwise comparisons. Asking an evaluator to choose the better of two responses yields higher consistency than asking them to rate each on a scale of 1-10. Third, leverage domain-specific standards. Use local medical exams for healthcare apps and local legal codes for legal tech. Finally, adopt hybrid evaluation pipelines. Use RL-trained LLM judges for scalability during development, but validate final releases with human review. This balances cost with reliability.

Remember, tokenization matters. Monitor sequence lengths in your target languages. If a task requires 500 tokens in English but 800 in Korean, your context window usage doubles, potentially pushing out critical instructions. Adjust your prompts accordingly. Also, be wary of "translation artifacts." If your training data includes machine-translated text, your model may learn awkward phrasings that sound robotic to natives. Curate your fine-tuning sets carefully to ensure they contain organic, human-written content in the target language.

The Future of Multilingual AI

We are moving away from ad-hoc checks toward structured, locale-specific frameworks. The goal isn't just to make models speak other languages; it's to make them think in them. Research is now focusing on "transforming" chat models to new primary languages, preserving conversational alignment while shifting the linguistic base. This requires equally advanced evaluation protocols. As we expand to more languages, the gap between English and non-English performance will narrow, but only if we invest in diverse, high-quality evaluation infrastructure. Companies that ignore this risk deploying biased, unsafe, or simply annoying products in key international markets. The next wave of AI innovation won't just be about bigger models; it will be about smarter, more inclusive evaluation.