Study finds post-training can weaken LLM grammar across 101 languages

Zhanyu Chen and Jaap Jumelet’s paper examines how post-training and language-resource levels relate to grammatical ability in multilingual language models. The authors submitted it to arXiv on 30 Sep 2026, and the paper is listed as an EMNLP Main 2026 paper. They compare base and post-trained models from six families using four evaluation methods. The tests use MultiBLiMP, a syntactic minimal-pair benchmark covering 101 languages.

The central finding is that post-training weakens grammatical competence, but the size of the decline varies unevenly with model scale. The authors report that low-resource languages bear the greatest cost. Their results therefore do not describe one uniform effect across all the models and languages they evaluated. Instead, the study treats model scale and language resource availability as factors that shape the reported outcome.

The paper also distinguishes grammatical knowledge from a model’s ability to express that knowledge through explicit prompting. The authors found that post-trained models could retain knowledge that direct prompting did not reveal. They could measure this gap in high-resource languages, but not in the same way in low-resource settings. There, baseline performance was close to chance, leaving little observable room for hidden knowledge to explain. This means an evaluation result can reflect both what a model knows and what a particular test can elicit.

For low-resource languages, the study found that prompting in the language being tested can recover otherwise hidden competence. The authors say direct probing through unprompted probabilities works only for high-resource languages. They conclude that multilingual grammar evaluations should account for language differences and use multiple evaluation approaches to avoid understating low-resource abilities. The study does not claim that one prompting method or benchmark resolves every evaluation challenge.

For AI builders, the findings mean that relying on a single evaluation approach may give an incomplete picture of grammar across languages, particularly those with fewer resources. A practical implication is to include language-informed, varied tests when assessing multilingual models, rather than treating one measurement as definitive. The paper’s results also make it important to distinguish a model’s underlying grammatical competence from what it can demonstrate under a given prompt or test.