feat(cache): add exact hybrid checkpoints for agent continuations - #1558
Open
sufubao wants to merge 5 commits into
Open
feat(cache): add exact hybrid checkpoints for agent continuations#1558sufubao wants to merge 5 commits into
sufubao wants to merge 5 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
混合注意力模型的旧缓存把线性状态终点绑定在 hash 页边界,难以复用 Agent 的非对齐输入和已生成前缀。本 PR 增加默认关闭的精确检查点缓存:CPU KV 仍按固定容量页搬运,线性状态按实际 token 长度保存,GPU KV 淘汰后仍保留 CPU 检查点目录。
例如,12000-token 输入生成16个 token 后可以保存
S@12015;下一轮复用输入和已计算的输出,只补算最后一个尚未生成 KV/state 的 token 及新增内容。完整输入命中使用保存的 hidden 执行HEAD_ONLY,按当前请求参数重新采样。实现
origins贯穿 CPU、GPU radix 和 PD,避免混用不同计算历史。vanilla_with_att/eagle_with_attadapter、精确 accepted 状态、私有 draft 尾槽,以及 P→D 来源传递、D→P 缺页回流和首 token owner 协议。CPU 数据预算按 TP rank 计算;页容量不限制状态长度。新模式不能与旧
--enable_cpu_cache/--enable_disk_cache同开。详见设计文档。H100 验证与性能
原 MTP 负载保持不变:Qwen3.5-0.8B、MTP step3、8并发、257输入/8输出、4GiB CPU预算、小页256、大页8192(256×32)、GPU KV容量65536。对照 upstream
1eb4810c;最新 R15 两次各20轮,每侧320次warm,704个HTTP请求、1312项比较通过,未剔除样本。剩余 TPOT 回退未解决。 R15 的2.5369ms与历史R12的2.5366ms基本相同;同轮baseline更快,因此当前差距20.8%,不能把历史13%的比例继续当作最新结果。总延迟降低24.6%,该短输出测试不能代表饱和吞吐。
本次确认的收益限于恢复搬运:实际CPU cache/state方法的H100微测,8请求只补末尾KV的恢复4.857→4.687ms,8193-token多页恢复23.416→22.855ms;多页GPU临时空间峰值224→约112MiB。使用真实张量形状、合成数据,未加载模型权重。随机数据、非连续目标槽位、跨页、clear/lease及copy异常通过逐字节验证;CPU专项42场景/310检查和真实CPU cache 759检查通过。不能用此微测宣称服务TPOT改善。
所有实验通过
exp -m记录,保留不可变源码、启动参数、逐token原始请求及失败结果。本次部署tar SHA2563924999ad1712246ee50abdf5ae5b31ea55c6a5fb31eeab9cfec7c4bd8546df2;最终仅增加docstring澄清,已核对执行AST一致。详见完整记录。当前限制
CPU onload 和 PD 发布仍有同步等待,新请求查找可能等待复制持有的缓存锁;尚非全异步 CPU cache。27B R8 的8并发 TPOT 也仍有10.5%回退,后续改动未重新跑该模型,不能宣称全面优化。
只缓存已计算、可核验的状态;不额外 seal 最后采样 token,不保证迟到 abort/字符串 stop 的精确末态。D 暂仅复用完整输入或差一个 token 的检查点;回流使用 CPU HTTP,路由不保证 owner 亲和。多模态、prompt logprobs/routed experts 回退,其他不支持组合和权重身份约束详见设计文档。
共享请求和PD结构有变更,部署需统一更新并重启相关进程。