Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
Zhen Zhang · Changyi Yang · Zijie Xia · Zhen Yang · Chengzhi Liu · Zhaotian Weng · Yepeng Liu · Haobo Chen · Jin Pan · Chenyang Zhao · Yuheng Bu · Alkesh Patel · Zhe Gan · Xin Eric Wang
LenVM frames LLM generation length as a token-level value estimation problem, providing a scalable, annotation-free signal for precise length modeling.
摘要
Tokens are the fundamental units of computation in modern autoregressive models, and generation length directly influences both inference cost and reasoning performance. Despite its importance, existing approaches model length primarily at the coarse sequence level. We introduce the Length Value Model (LenVM), a token-level framework that estimates the remaining generation length at every decoding step. By formulating length modeling as a value estimation problem and assigning a constant negative reward to each generated token, LenVM predicts a bounded, discounted return that is a monotone proxy for the remaining generation horizon. This value formulation provides annotation-free, dense, unbiased, and scalable supervision. Experiments on LLMs and VLMs show that LenVM supports exact control, continuous performance--efficiency steering, length prediction, and interpretation. On LIFEBench-token, it raises the exact-length score of Qwen2.5-7B-Instruct from $30.9$ to $64.8$ in one pass; combined with eight-round LCG, it further raises the score from $71.3$ to $83.6$ while reducing average absolute deviation from $14\%$ to $6\%$. On GSM8K near $200$ tokens, LenVM retains about $63\%$ Pass@1 versus $6\%$ for a hard token budget, and it outperforms budget-aware prompting and EOS calibration at matched lengths on MATH500. Prompt-boundary prediction improves consistently with scale, reaching MREs of $9.8\%$, $14.9\%$, and $17.1\%$ on math, code, and instruction following at 32B; against entropy-guided predictors, LenVM reduces prompt MRE from $32.56$ to $26.90$ and progressive MRE from $26.71$ to $15.10$. Its token-level values also identify pivots between longer and shorter trajectories. Results demonstrate that LenVM supports a broad range of applications, including length control, prediction, and interpretation of generation dynamics. They suggest that generation length can be effectively modeled as a token-level value signal, highlighting the potential of LenVM as a general framework for length modeling and as a length-specific value signal that supports future RL training.