Shujian Liu

Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors. As tasks grow more challenging, the logic structures embedded in natural language instructions becomes increasingly intricate. However, how well LLMs perform on such logic-intensive instructions remains under-explored. We propose **Logic** **I**nstruction **F**ollowing (**LogicIF**), a task requiring models to precisely follow and simulate every logical step within complex instructions. To generate these instructions, we construct **LogicIFGen**, a **scalable**, **automated** framework for generating **verifiable** instructions from **code functions**, which can naturally express rich logic such as conditions, loops, nesting, and function calls. Using this framework, we curate a collection of complex code functions to construct **LogicIFEval**, a benchmark comprising 426 verifiable, human-examined, logic-rich instructions. Our experiments demonstrate that current state-of-the-art LLMs still struggle to correctly follow the instructions in LogicIFEval. Most LLMs can only follow fewer than 60\% of the instructions, revealing significant deficiencies in their capacity to handle instructions that involve complex logical structures. To further demonstrate the efficacy of our framework, we collect an extensive set of code functions across a wide range of logical difficulties to generate **LogicIFTrain**, a fully synthesized training set featuring 27,324 verifiable instructions. Our results show that a model trained on LogicIFTrain using reinforcement learning significantly outperforms the base model on both in-domain and out-of-domain benchmarks. All the data, codes and models are released at https://github.com/mianzhang/LogicIF.

Large language models often hallucinate, producing content that is factually incorrect or not grounded in the sources. Reliable faithfulness verification is critical for trustworthy deployment. In the provided-source (closed-world) setting, existing verifiers either classify whole passages in one...