一作OSDI 2026Oral· Operating Systems & Distributed Systems
ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs
Guangda Liu · Wenhao Chen · Chengwei Li · Zhenyu Ning · Jing Lin · Yiwu Yao · Quan Chen · Shixuan Sun · Jieru Zhao · Minyi Guo
摘要
Native sparse attention has emerged as a promising approach for efficient long-context LLM inference without compromising accuracy. While it significantly reduces the attention computation and KV cache access costs, the KV cache size exhibits steeper linear growth with the context length. As a re...