Yejin Choi

Instruction Hierarchy (IH), the structured prioritization of system prompts over user prompts, has emerged as a key security mechanism for language models (LMs). Despite its importance for flexible steering and robust safety control, current LMs offer limited support and often fail to enforce system-level specifications when these conflict with user instructions. In this work, we introduce HieraSuite, a full-stack toolkit for building steerable and secure system-user IH for LMs. HieraSuite encompasses four key components: (1) HieraInstruct, a large-scale and diverse collection of 221K system–user instruction pairs spanning four real-world application domains (system constraints, privacy and security, steerability, and task execution); (2) HieraConsReasoner, an effective and compact reasoner model, paired with training data, that elicits contextualized rubrics to specify what constitutes valid responses under hierarchical instructions; (3) HieraCRO, an iterative response optimization approach, grounded in constitutional rubrics, that enhances LM compliance with instruction hierarchy; and (4) HieraBench, a unified benchmark that integrates ten tasks to assess controllability, steerability, customizability, and security of system-user instruction hierarchy. Together, these components form an end-to-end solution that yields consistent gains across model families and scales, including up to 66.9% improvements on HieraBench tasks and over 306.3% gains in overriding conflicting user instructions. Systematic testing of alignment recipes further identifies design choices that balance user instruction-following, system instruction-override, and general capabilities. This work provides a principled framework and practical toolkit for LM user-system instruction hierarchy, laying the foundation for future studies on “instruction un-following” and advancing steerability and security in LM alignment.

Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that achieve better loss scaling: not only lowering loss at finite compute but especially as compute approaches infinity. We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss on the web data, despite the synthetic data coming from an entirely different distribution. With optimal mixing and epoching, loss and benchmark accuracy improve without overfitting as the number of synthetic generations grows, plateauing near $1.48\times$ data efficiency at 32 rephrases per document. We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer *megadocument* instead of many short documents. We show two ways to construct megadocs: stitching synthetic rephrases from the same web document or stretching a document by inserting rationales. Both methods improve i.i.d. loss, downstream benchmarks, and especially long-context loss relative to simple rephrasing, increasing data efficiency from $1.48\times$ to $1.80\times$ at $32$ generations per document. Importantly, the improvement of megadocs over simple rephrasing widens as more synthetic data is generated. Our results show how to design synthetic data algorithms that benefit more from increasing compute when data-constrained.

Can humans identify AI-generated (fake) videos and provide grounded reasons? While video generation models have advanced rapidly, a critical dimension -- whether humans can detect deepfake traces within a generated video, i.e., spatiotemporal grounded visual artifacts that reveal a video as machine generated -- has been largely overlooked. We introduce DeeptraceReward, the first fine-grained, spatially- and temporally-aware benchmark that annotates human-perceived fake traces for video generation reward. The dataset comprises 4.3K detailed annotations across 3.3K high-quality generated videos. Each annotation provides a natural-language explanation, pinpoints a bounding-box region containing the perceived trace, and marks precise onset and offset timestamps. We consolidate these annotations into 9 major categories of deepfake traces that lead humans to identify a video as AI-generated, and train multimodal language models (LMs) as reward models to mimic human judgments and localizations. On DeeptraceReward, our 7B reward model outperforms GPT-5 by 34.7% on average across fake clue identification, grounding, and explanation. Interestingly, we observe a consistent difficulty gradient: binary fake v.s. real classification is substantially easier than fine-grained deepfake trace detection; within the latter, performance degrades from natural language explanations (easiest), to spatial grounding, to temporal labeling (hardest). By foregrounding human-perceived deepfake traces, DeeptraceReward provides a rigorous testbed and training signal for socially aware and trustworthy video generation.

This position paper argues that in many realistic (i.e., complex, contextualized, subjective) scenarios, one LLM is not enough to produce a reliable output. We challenge the status quo of relying solely on a single general-purpose LLM and argue for multi-LLM collaboration to better represent the...