fix(cache): improve hybrid attention snapshot eviction - #1551
Open
sufubao wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
混合注意力模型匹配或插入更长前缀时,当前实现会刷新沿途所有节点的时间,使未实际使用的历史快照也被视为近期使用。快照容量紧张时,这可能挤掉其他对话的有效恢复点。另外,KV 回收达到目标后立即停止,可能在删除尾部快照后,仍保留一段已经无法独立恢复的 KV。
本次修改:
验证:开发期间的 13 项针对性 CPU 控制逻辑测试通过,覆盖快照新旧顺序、大小页恢复与回退、引用计数和回收边界;临时测试文件未纳入本 PR。Flake8 和
git diff --check通过。尚未进行 GPU 推理或多轮对话性能实测。