Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
Sourabrata Mukherjee · Kalika Bali · Sunayana Sitaram
Given the same task in another language, tool-using agents take measurably different actions: they retain only 71-73% of their own action self-consistency, a constant hidden by four confounds that each flip a conclusion.
摘要
When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks, because it compares final answers and throws the intermediate actions away. For an agent those actions are the product: the route fixes cost and latency, decides how the system fails, and is the only part of its behaviour a reviewer can audit, so two language versions that agree on every answer can still differ in price, in failure mode, and in whether a safeguard written against one of them holds for the other. We therefore make the action policy itself the measured object, across 8 models, 6 parallel benchmarks and 41 languages (2.38M agent rollouts). The naive measurement does not work, because five confounds sit between raw trace similarity and any defensible claim, each large enough on its own to change a conclusion: short traces score higher than long ones, empty traces score perfectly, unrelated traces already agree by chance more than half the time, the gap is capped by each model's own reproducibility, and, worst of all, a model asked the same question twice in one language does not answer it the same way, so there is no reference point to measure against. We rebuild the measurement to remove all five, and every correction makes the effect larger rather than smaller. Cross-lingual divergence then proves structural rather than sampling noise: it survives greedy decoding in every cell we test and stays flat as temperature rises, even as models become far less self-consistent. Normalising by each model's own reproducibility, four very different frontier models converge on nearly the same value under greedy decoding, each keeping 71-73% of its action policy when the language changes, and which model it is explains only 5.7% of the variance. Below roughly 10B parameters that regularity breaks down, and the apparent ordering among smaller models is largely an artifact of a chance floor that we measure by permutation rather than assume. Asking why, we find that agents route non-English tasks through English, that this pivot is causally load-bearing, with a prediction registered in advance and confirmed across four models, and that models will not abandon it when instructed to. Finally, a single trace-extraction regex, not the model, manufactured an apparent multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.