Proposal: Approximate Speculative Decoding (ASD) acceptance for MTP - an acceptance-laxity dimension complementary to LightSpec

[Kissmetothemoon]

Summary

I would like to contribute ASD (Approximate Speculative Decoding, arXiv:2608.03447)
as an optional acceptance policy for LightLLM's MTP speculative decoding.

Strict greedy verification discards the whole draft suffix at the first mismatched
token. ASD instead admits a draft token x_i when its local regret against the target
logits, r_i = max_v z_i(v) - z_i(x_i), stays within a bounded per-request budget:
(1) cumulative request-level regret <= B; (2) r_i / q_i <= g with suffix value
q_i = K - i; (3) at most m mismatches per draft block. The first infeasible position
stops acceptance and the target greedy bonus token is committed as usual. B = 0
recovers strict greedy verification exactly (bitwise acceptance equivalence).

Relationship to LightSpec

LightSpec (great work!) dynamically decides how much to verify (verify budget and
draft depth). ASD decides how strictly to accept. The two are orthogonal and
composable: the planner only consumes accept_len statistics, so ASD simply shifts
the observed acceptance distribution upward without touching the planner.

Measured results (algorithm level, greedy)

• Qwen3-14B + DSpark block7 draft, 8xL20: strict 66.21 TPS / GSM8K 79.682%;
  ASD q25 (B=2.0625): +13.34% TPS / -0.227pp; B=0 arm token-identical to strict
  on all 1319 GSM8K requests.
• DeepSeek-V4-Flash-DSpark: +5.44% TPS.

Reference implementation: github.com/Kissmetothemoon/ASD (Apache-2.0). The PR will
implement the rule natively in LightLLM (no external dependency), gated behind new
server args, default off with zero behavioral change.

Draft design

• New Triton kernel variant of mtp_verify in
  lightllm/common/basemodel/triton_kernel/mtp_utils.py that also outputs committed
  token ids (draft ids at relaxed-accepted rows) and carries per-request cumulative
  regret in a device buffer owned by ReqSamplingParamsManager (same lifecycle as
  req_to_next_token_scores).
• verify_mtp_tokens in mtp_speculative/utils.py dispatches strict vs ASD based
  on start args; the three decode call sites pass model_output.logits through.
• v1 is greedy-only (falls back to strict verification for sampled batches).

Happy to align on arg naming and whether you prefer server-level args now and
per-request sampling params later. I work in the same AI-infra research community
(BUAA) and would love to collaborate.