feat: support multi-token KV cache pages - #1522

Open

hiworldwzj wants to merge 17 commits into

mainfrom

Commits on Aug 31, 2026

feat: support multi-token KV cache pages
hiworldwzj
committed
06a44f8
refactor: preallocate KV cache during scheduling
hiworldwzj
committed
4071248
refactor: derive KV indexes in infer state
hiworldwzj
committed
cac576f
fix: assert prompt cache page alignment
hiworldwzj
committed
e7280d6
refactor: align KV allocation needs to pages
hiworldwzj
committed
c3b81c4
refactor: reduce MTP KV reservation window
hiworldwzj
committed
cba88a6
refactor: move token scheduling control to infer backend
hiworldwzj
committed
ba4afe2
refactor: remove duplicate KV page reservation
hiworldwzj
committed
5008bd0
fix: restrict fixed KV cache to single-token pages
hiworldwzj
committed
92c4cb6
refactor: split PD queues by inference stage
hiworldwzj
committed
5e5ac6c
fix: reserve request capacity for MTP overlap
hiworldwzj
committed
3d681d4
fix: reserve page capacity for request admission
hiworldwzj
committed
7889557
refactor: simplify request peak token estimation
hiworldwzj
committed
befb8c1
refactor: simplify PD prefill token estimation
hiworldwzj
committed
af9dc62
refactor: simplify PD decode token estimation
hiworldwzj
committed
72bfde1
test: adapt adaptive cache release to KV reservations
hiworldwzj
committed
504a91e

Commits on Sep 1, 2026

fix: complete paged KV cache support
hiworldwzj
committed
70f6dbf