作者位次 #8/8OSDI 2026Oral· Operating Systems & Distributed Systems
Strata: Hierarchical Context Caching for Long Context Language Model Serving
Zhiqiang Xie · Ziyi Xu · Mark Zhao · Yuwei An · Vikram Sharma Mailthody · Scott Mahlke · Michael Garland · Christos Kozyrakis
摘要
Long-context large language models (LLMs) enable applications that reason over hundreds of thousands to millions of tokens, but serving these workloads efficiently is challenging. Modern systems cache key-value (KV) states and rely on hierarchical context caching across GPU HBM, CPU memory, and S...