Fantine Eri Huot

To handle ambiguous queries, AI assistants must decide (a) when to guess the user's intent and answer directly, (b) when to enumerate and respond to multiple plausible intents, and (c) when to ask a clarifying question. No strategy is always superior; the right decision depends on *contextual* factors such as user preferences or modality. For example, enumerating several possible interpretations can be cumbersome on small screens or voice-based settings, while a more interactive style may be dispreferred in desktop settings. In this work, we propose training *steerable policies* for managing such uncertainty via self-play. Given two agents, one simulating a user and the other an AI assistant, we generate conversations where the user issues a potentially ambiguous query, and the assistant must choose how to respond. The assistant receives as input the *numerical cost* of asking a clarification question, and of generating each word, and is trained to select the conversational action that maximizes its final reward: accuracy penalized by these costs. We apply Reinforced Self-Training (ReST) to optimize this objective and show that it yields a steerable policy that adapts its behavior based on the provided costs, improving both reward and accuracy. Moreover, the resulting policy generalizes to numerical cost values unobserved during training.

We present a scalable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communication about private information. This enables an interactive scaling analysis, in which a fixed token budget is divided over a variable number of turns. We find that in many cases, language models are unable to use interactive collaboration to improve over the non-interactive baseline in which one agent summarizes its information and the other agent immediately acts -- despite substantial headroom. This suggests that state-of-the-art models still suffer from significant weaknesses in planning and executing multi-turn collaborative conversations. We analyze the linguistic features of these dialogues, assessing the roles of sycophancy, information density, and discourse coherence. While there is no single linguistic explanation for the collaborative weaknesses of contemporary language models, we note that humans achieve comparable task success at superior token efficiency by producing more coherent dialogues. The proactive management of private information is a defining feature of real-world communication, and this work is designed to drive further progress on this capability.