Skip to content

fix(hicache): discover SGLang context-parallel shards - #355

Merged
ketor merged 1 commit into
dingodb:mainfrom
ketor:fix/sglang-cp-hicache-namespace
Aug 27, 2026
Merged

fix(hicache): discover SGLang context-parallel shards#355
ketor merged 1 commit into
dingodb:mainfrom
ketor:fix/sglang-cp-hicache-namespace

Conversation

@ketor

@ketor ketor commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • discover SGLang attention-CP and DCP coordinates from the initialized runtime context when dynamic HiCache extra_config omits them
  • include the physical CP coordinate in every pool key instead of aliasing all ranks at pcp0/dcp0
  • elect one replicated-MLA writer per physical PCP/DCP shard rather than one global TP-rank-zero writer

Why

GLM-5.3-Flash TP8/attention-CP8 produced identical dfkv keys on all eight CP ranks. A restarted target observed some existence hits but no contiguous prefix (prefix=0/1023), issued zero GETs, and recomputed the entire prompt.

Validation

  • standalone HiCache key/coordinate suite: 16/16 passed
  • xb01-0064, 8x B200, GLM-5.3-Flash, SGLang PCP8 + MTP5 + dfkv, C10 / 64K input / 500 output / 100 prompts
  • before: zero dfkv GETs after restart
  • after: 522,060 GET hits, 217,627,889,664 bytes read, zero GET/PUT I/O errors
  • cold total throughput 24,850 tok/s; restart+dfkv hot 40,763 tok/s (+64%)

@ketor
ketor force-pushed the fix/sglang-cp-hicache-namespace branch from 99d7f88 to 61dcf10 Compare August 27, 2026 18:24
@ketor
ketor added this pull request to the merge queue Aug 27, 2026
Merged via the queue into dingodb:main with commit cfa9131 Aug 27, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant