Delta-Prefill Switching: Adaptive Routing for Speculative Decoding in Multi-Turn LLM Serving
0
Multi-turn LLM applications with prefix caching are increasingly common in production deployments. Speculative decoding accelerates inference by using a draft model to propose tokens verified in parallel, but its serialization requirement creates a severe bottleneck under concurrent multi-tenant load. We propose Delta-Prefill Switching (DPS), a simple routing policy that uses incremental prompt growth (∆L)—the new tokens added since the last turn—to route requests between speculative and greedy decoding servers. When ∆L is small, cached computation dominates and speculation provides speedup; when ∆L is large, speculation’s serialization becomes costly under concurrency. On ToolBench and BFCL benchmarks, DPS achieves 21–22% speedup over greedy decoding in sequential mode, matching always-on speculation. Under concurrent load (c ≥ 4), DPS achieves +64–80% speedup over always-on speculation by routing to the concurrent-capable greedy server. DPS is robust to threshold selection and requires no model modifications.
Discussion (0)
to join the discussion.
No comments yet. Be the first to discuss this paper.