feat(autotune): separate general and decode attention warmup

[hiworldwzj]

Problem

The existing autotuner uses a single model warmup phase. That works for general/prefill kernels, but decode attention is difficult to tune correctly:

• decode kernels are normally reached while CUDA Graphs are initialized;
• the graph initialization requests have very short real sequence lengths and do not represent production decode workloads;
• autotuning cannot run inside CUDA Graph capture;
• treating every kernel reached by decode initialization as tunable creates unnecessary configuration searches.

Changes

This PR separates autotuning into GENERAL and DECODE_ATTENTION warmup phases.

Before each decode CUDA Graph is captured, LightLLM runs one ordinary decode forward under the decode-attention warmup scope. Only kernels marked as DECODE_ATTENTION may search configurations in this scope. The selected configuration is then used by the following CUDA Graph capture.

The autotuner also gains:

• a nestable warmup context that restores the previous phase;
• an optional rebuild_input_func for constructing representative benchmark inputs outside graph capture;
• an optional warmup_all_exist_config switch for stateful kernels that cannot safely replay every cached configuration;
• support for key functions that reference omitted optional arguments;
• invalidation of cached nearest-key matches after a new configuration is saved.

Decode attention treats FORCE_AUTOTUNE as adaptive tuning so identical kernels used by many layers do not repeatedly retune. To force a fresh decode search, delete that kernel's cached configuration directory before restarting.

After all startup argument normalization, an enabled decode CUDA Graph also caps graph_max_batch_size at running_max_req_size. Larger graph batches cannot be reached by the scheduler and only consume extra capture time and GPU memory. The original value is retained when CUDA Graph is disabled.

Covered decode kernels

Representative-input tuning is added for:

• FA3 KV-cache decode (num_splits);
• Triton GQA flash decode stage 1;
• MTP diverse attention stages 1 and 2;
• INT4 KV flash decode stage 1;
• INT8 KV flash decode stage 1;
• MTP fused recurrent gated delta rule linear attention.

The representative KV length is controlled by:

LIGHTLLM_DECODE_ATTN_AUTOTUNE_SEQ_LEN=32768

The default is 32K. KV-length run keys are bucketed where appropriate to reduce redundant searches while normal execution still selects configurations using the real request length.

Decode autotuning is supported only during service startup warmup. It is not performed for live runtime requests. The MTP recurrent linear-attention kernel therefore reuses the startup dummy state pool and disables cached-configuration prewarming; cloning its full persistent state pool would substantially increase memory use and distort benchmark timings.

Intermediate stage outputs that are fully overwritten are also excluded from mutates_args to avoid unnecessary clones during tuning.

Validation

Unit and kernel validation:

• 240 targeted autotuner, CUDA Graph, FA3, GQA, INT4/INT8, MTP attention, and MTP linear-attention tests passed;
• the two formerly colliding autotune test modules now have unique names, and their default pytest collection and execution pass: 65 tests;
• git diff --check, Python compilation, Black, and Flake8 pass;
• tests cover representative 8K/16K inputs, the 32K default selection, configuration persistence and reuse, CUDA Graph capture/replay, and comparisons against FP32 references.

Service validation used Qwen3.5-27B, native MTP step 3, BF16 weights, FP32 SSM state, FA3 full attention, Triton linear attention, NVIDIA H200 GPUs, and decode CUDA Graphs:

The service matrix was run with a 16K decode autotune target before the default changed to 32K. The new 32K default is covered by the targeted autotune key and input-construction tests; it has not been rerun through the full service matrix.

• TP 1/2/4;
• concurrency 16/32/64;
• 200 fixed MMLU requests and 200 fixed long-context TriviaQA requests per case;
• adaptive tuning on/off, for 7,200 measured requests in total.

All requests completed successfully with finite log probabilities. Tuning disabled produced no operator configuration files; adaptive tuning produced 12 expected FA3 and MTP linear-attention configuration files.

The measured throughput effect was workload-dependent rather than a universal improvement. Across the initial matrix, output throughput ranged from -10.7% to +6.1%. The TP2/MMLU/concurrency-32 regression was repeated three times and averaged -5.5%. With prefill CUDA Graph enabled, the same case averaged -0.03%. Score changes across the 200-sample comparisons ranged from -2.0 to +0.5 percentage points; the sample size and batching-related output variation do not establish a tuning-induced accuracy change.

The full service matrix was run at commit 361fdcd5. Later changes that reuse stage-2 inputs, avoid cloning overwrite-only outputs, document the stateful warmup exception, and rename the pytest modules were covered by their targeted kernel and graph tests.