Liwei Jiang

Instruction Hierarchy (IH), the structured prioritization of system prompts over user prompts, has emerged as a key security mechanism for language models (LMs). Despite its importance for flexible steering and robust safety control, current LMs offer limited support and often fail to enforce system-level specifications when these conflict with user instructions. In this work, we introduce HieraSuite, a full-stack toolkit for building steerable and secure system-user IH for LMs. HieraSuite encompasses four key components: (1) HieraInstruct, a large-scale and diverse collection of 221K system–user instruction pairs spanning four real-world application domains (system constraints, privacy and security, steerability, and task execution); (2) HieraConsReasoner, an effective and compact reasoner model, paired with training data, that elicits contextualized rubrics to specify what constitutes valid responses under hierarchical instructions; (3) HieraCRO, an iterative response optimization approach, grounded in constitutional rubrics, that enhances LM compliance with instruction hierarchy; and (4) HieraBench, a unified benchmark that integrates ten tasks to assess controllability, steerability, customizability, and security of system-user instruction hierarchy. Together, these components form an end-to-end solution that yields consistent gains across model families and scales, including up to 66.9% improvements on HieraBench tasks and over 306.3% gains in overriding conflicting user instructions. Systematic testing of alignment recipes further identifies design choices that balance user instruction-following, system instruction-override, and general capabilities. This work provides a principled framework and practical toolkit for LM user-system instruction hierarchy, laying the foundation for future studies on “instruction un-following” and advancing steerability and security in LM alignment.

Current LLM safety alignment techniques improve model robustness against adversarial attacks, but overlook whether and how LLMs can recover helpfulness when benign users clarify their intent. We introduce CarryOnBench, the first interactive benchmark that measures whether LLMs can revise their interpretation of user intent and recover utility, while remaining safe through multi-turn conversations. Starting from 398 seemingly harmful queries with benign underlying intents, we simulate 5,970 conversations by varying user follow-up sequences, evaluating 14 models on both intent-aligned utility and safety. CarryOnBench yields 1,866 different conversation flows of 4--12 turns, totaling 23,880 model responses. We design Ben-Util, a checklist-based metric that evaluates how well each model response fulfills the user's benign information need using atomic items. At turn one, models fulfill only 10.5--37.6\% of the user's benign information need. When the same query includes the benign intent upfront, models fulfill 25.1--72.1\%, confirming that models withhold information due to intent misinterpretation, not limited knowledge. With benign clarifications in multi-turn conversations, 13 of 14 models approach or exceed this single-turn baseline, yet recovery cost varies across models. We identify two failure modes invisible to single-turn evaluations: unsafe recovery, where a model updates at disproportionate safety cost, and redundant recovery, where a model recycles prior responses rather than providing new information. Moreover, conversations converge to similar harmfulness levels regardless of how conservative the model starts. These findings expose a gap that single-turn evaluations miss---whether a model is appropriately cautious or simply unresponsive to clarified user intent.