Skip to content

fix(vllm): window long deduplicated KV loads - #362

Open
ketor wants to merge 1 commit into
dingodb:mainfrom
ketor:fix/vllm-long-load-dedup-windows
Open

fix(vllm): window long deduplicated KV loads#362
ketor wants to merge 1 commit into
dingodb:mainfrom
ketor:fix/vllm-long-load-dedup-windows

Conversation

@ketor

@ketor ketor commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Problem

For same-host replicated-MLA loads, native GPU dedup publishes fetched results only after the enclosing batch_get_auto_sg completes. A 1M-token TP8 load contains 15,624 objects and takes much longer than the default follower wait, so all eight ranks time out and independently fetch the same KV data.

Change

  • add opt-in load_window_keys to bound each native GET publication
  • add load_window_min_keys so short requests retain one native GET
  • validate both settings at connector construction
  • preserve key/result ordering and one outer logical-load metric
  • document sizing against the dedup arena and wait deadline

Defaults remain disabled and backward-compatible.

Validation

  • ruff check integration/vllm/src/dfkv_vllm/worker.py integration/vllm/tests/test_worker_lifecycle.py
  • vLLM GLM-5.3 image: 13/13 lifecycle/window tests passed
  • xb01-0064, GLM-5.3, TP8, actual 1,000,000 input + 500 output:
    • default: TTFT 39.35 s, 22,756.73 total tok/s, ~8 remote copies
    • 128-key windows, 4,096-key threshold: TTFT 22.92 s, 36,458.56 total tok/s, ~1 remote copy, zero dedup fallback
  • 65,536/500, C10, 100 prompts with the same threshold: 35,622.47 total tok/s, 100/100 success

@ketor
ketor force-pushed the fix/vllm-long-load-dedup-windows branch from 6ad885a to 1569adc Compare August 29, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant