SPEED: Specialized Position Experts for Efficient Speculative Decoding
Langlin Huang · Chengsong Huang · Jixuan Leng · Di Huang · Jiaxin Huang
We propose specialized position experts for EAGLE-style speculative models, resolving their feature deviation problem. SPEED greatly improves the drafting accuracy, resulting in faster generation with either Transformers or vLLMs.
摘要
Speculative decoding accelerates Large Language Model (LLM) inference by using a small draft model to predict multiple tokens, and a large target model to verify these tokens in parallel. Recent studies leverage features of target and draft models to enhance draft accuracy, but suffer from the degrading accuracy of draft tokens at later positions, due to accumulated deviation in draft model-generated features. Worse still, features of inconsistent levels of deviation provide a noisy training signal at training stage. In this paper, we propose SPEED, applying specialized position experts to draft tokens at preassigned positions. Position experts substantially improve acceptance rate at later draft positions, as each expert only needs to focus on handling a certain level of feature deviation. Experiment results on six datasets demonstrate that SPEED effectively improves over baselines on average acceptance length and speed-up ratio. Our codebase is available at https://github.com/shrango/PosS.