Skip to content

arm PR106.x: EXL3 DFlash2 kit reference series (PR106.0 external / PR106.1 native 3.5bpw single-GPU) - #114

Open
ddvnguyen wants to merge 1 commit into
baselinefrom
fork/pr106-exl3-reference-arm
Open

arm PR106.x: EXL3 DFlash2 kit reference series (PR106.0 external / PR106.1 native 3.5bpw single-GPU)#114
ddvnguyen wants to merge 1 commit into
baselinefrom
fork/pr106-exl3-reference-arm

Conversation

@ddvnguyen

Copy link
Copy Markdown
Owner

Arm series spec derived from the MiaAI-Lab Qwen3.8-27B-DFlash2-EXL3-5.0bpw deployment kit (forked exllamav3 engine; EXL3 3.5 bpw target + MTP or DFlash2 5.0 bpw draft + ~4.5-bit KV lane; batch-1, always-reasoning). llama.cpp cannot load EXL3, so the family splits into an external reference and a llama.cpp-native translation.

What this PR contains

Docs only (docs/arms/pr106-exl3-dflash2-reference.md):

  • PR106.0: run the kit on the 5060 Ti alone (12.3 GB target + draft + 4.5-bit KV fits 16 GB), DRAFT=mtp then dflash2; same prompt set as arm102; reasoning/content split in metrics; measurement arm, no pass/fail bars; validates the kit's ~15% DFlash2-over-MTP claim
  • PR106.1: llama.cpp-native equivalent — UD-Q3_K_XL (~3.5 bpw) full offload on 5060 Ti alone (-dev CUDA0, -sm none, no UM/RPC), 747.0 reference flags otherwise; roofline insight: single-GPU 3.5 bpw raw ceiling (~36.5 t/s no-MTP) exceeds the VRAM-pinned 2-GPU 5 bpw ceiling (~32.5 t/s); bars >= 45 t/s target vs the 40.1 PR105.0 bar; hard quality gate: llama-perplexity KL vs UD-Q5_K_M
  • PR106.2 (optional, gated on PR106.0): DFlash2 draft EXL3->GGUF for --model-draft

Test plan

  • PR106.0 reference table produced (mtp + dflash2 cells)
  • PR106.1 executed after PR105.0 bars are banked; bars + KL gate checked
  • Results into arm report; quant-vs-split strategy decision recorded

Independent of PR103.0/PR104.x. Risks noted: sm_120 EXL3 kernel build unverified on the 5060 Ti; fork engine maturity.

AI usage disclosure: YES — arm spec drafted by AI (opencode) from the kit README and local flag verification; launch line flagged for on-rig confirmation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant