My team recently tested how well the newest AI models from leading frontier labs translate seven languages. Swedish has about 10 million speakers, while Hindi has about 600 million. You might expect the models to do better in Hindi. Instead Swedish got some of the best translations in the study, while Hindi was among the worst, along with Estonian and Thai. Across all seven languages, the number of speakers had almost nothing to do with translation quality.
We gave every model the same set of about 500 sentences per language, drawn from four common business domains, and scored the translations based on several standard industry metrics so no single number could skew the results. The languages were Estonian, Hindi, Indonesian, Brazilian Portuguese, Swedish, Thai, and Turkish. We tested two generations of each lab's model to see where each language stands today and whether it is improving.
This matters because people now use AI for banking, healthcare, and schoolwork, and anyone who builds these models into a product inherits their blind spots. A Hindi speaker gets a worse experience than a Swedish speaker, despite outnumbering them 60 to one. For years, the question was whether a model supported a language at all. Today’s models can produce text in hundreds of languages. The question now is how good that text is.
Some of the gap comes from the languages themselves. Brazilian Portuguese has fairly simple grammar and a large amount of available text, so the models handle it well. Swedish is related to German but has lost most of German's grammatical complexity. Estonian changes the form of its words heavily depending on how they're used in a sentence, and models struggle with that. Hindi lags because models haven't been trained on nearly enough Hindi text, or enough kinds of it.
Grammar and data explain only part of why the gaps persist. Translation is one task among many for labs building general purpose models, and it competes for attention with reasoning, coding, agents, and safety, so each release reflects where the lab spent its effort. Tokenization, the way a model splits text into pieces it can process, matters most for scripts like Hindi and Thai. Tokenization updates tend to follow product roadmap priorities.
The results show what happens when labs do the work. Languages that started out weaker usually improved the most from one model generation to the next, but that pattern doesn’t always hold true, and progress wasn’t the same from lab to lab. Newer models also often change formatting, verbosity, or punctuation in ways that move the metrics even when the meaning is intact. Either way, three or four companies' priorities in a given cycle can decide whether a language catches up.
The methods for closing the gap are known: more multilingual text, better tokenizers, synthetic data (text generated by models to fill gaps), and cross-lingual transfer, for example, a model trained on lots of Russian can extrapolate to Slovenian. What’s needed is more focus; labs can give underserved languages a dedicated owner and publish results for each language with every release. Companies building on these models can test them in the languages their customers speak and send those results back to the labs. We share our findings with the labs and work through the causes with their scientists, and they're receptive, because almost no one else brings them field data on translation.
It’s encouraging that this is starting to happen collectively. In September, a Gates Foundation coalition brought together frontier labs for a shared commitment to build AI support for under-resourced languages over the next five years. It’s a sign that the labs are beginning to treat these languages as a priority instead of an afterthought.
The gap is closing where labs put in the work, and the models will keep improving. Hindi, Estonian, and Thai will improve with them only if the labs, and the companies building on them, decide those languages count.
