feat(ENG-RESIDENCY-CONFIG): the device budget is a config key, and --offload-config reaches vllm-cli and the pooling server (#1127, #1135)

[localai-bot]

Two of ENG-RESIDENCY-CONFIG's own ## Owed entries, closed together because
they edit the same two files: server_main.cpp's argument block and
weight_residency.cpp's parser. Splitting them would have put two branches on
both.

The device budget becomes a config key (#1127)

VT_DEVICE_WEIGHT_BUDGET_BYTES is the escape hatch for the load-time device-fit
refusal that landed in #1132, and it was the odd one out once mmap, prefault,
expert_stream, slots and slot_bytes had all become keys under vllm_cpp.
It joins them:

--offload-config '{"vllm_cpp":{"device_fit":{"weight_budget_bytes":68719476736}}}'

Same terms as its five siblings. Environment variable > config > device probe,
pinned by a test in both directions. An absent field means unchanged, at the
parse and at the install, pinned by two DIFFERENT partial documents in one
process — the shape that caught #1133 H2, because a second document that restates
the first's field cannot show a dropped field. A misspelling anywhere in the new
level is refused at startup by name, with a case per spelling
(device_fitt, device-fit, weight_budget_byte, device_weight_budget_bytes).

device_fit is an object, not a scalar sibling of mmap and
expert_stream. Two reasons. vllm_cpp maps a name to an OBJECT today and the
parser's ExtObject walk assumes it, so one scalar would make the level
heterogeneous for one field. And the device-fit check is a third knob FAMILY
rather than a member of either existing one; the field name drops the family
prefix, which is the transformation VT_GGUF_MMAP → mmap.enabled and
VT_MOE_EXPERT_STREAM_SLOTS → expert_stream.slots already use.

ZERO is legal here and nowhere else in the struct. CheckDeviceWeightFit
reads a zero budget as UNKNOWN and decides nothing, so 0 is the documented
spelling of "suppress the refusal and get the late failure back" — exactly what
VT_DEVICE_WEIGHT_BUDGET_BYTES=0 already means. slots and slot_bytes are
sizes and keep refusing a zero, because a slot count that silently became 64 is a
cache the operator does not have. So the budget parses through its own
ExtNonNegativeInt, a negative value is refused, and the message says "must not
be negative" rather than "must be positive", because the second sentence would be
false about a value the parser accepts.

DeviceWeightBudgetBytes keeps its name and its callers and delegates to
ResolveDeviceWeightBudgetBytes, now the sole reader of the variable. That is
what stops the install-time override announcement from drifting away from what
the resolver does with a value: 0 wins and is announced, while a signed, spaced
or garbage value is ignored and falls through — to the config now, where it used
to fall through to the probe. A run with no config therefore resolves
byte-for-byte as before.

Three entry points, decided rather than plumbed (#1135)

The server parses --offload-config ONCE, ahead of the architecture branch.

vllm-cli gets the flag. It is a thin ABI client and
vllm_model_params.offload_config already parses both halves, so this is one
assignment and both tiers arrive — the mirrored uva/prefetch half included,
refusal and all. The gate runs the built binary and reads the install line off
its stderr, because a test that called the ABI would re-prove the ABI and would
have passed on the day the flag was missing.

The pooling/embedding path gets both halves. It builds EngineParams and
calls the same LoadedEngine::FromModelDir, which installs the residency
document and the weight offloader ahead of every path and weight operation, so
nothing about pooling makes either half inapplicable. Before this, both were
dropped there in silence.

The transcription-only path gets NEITHER, and refuses a non-empty document at
startup. ParakeetTranscriber::FromDir builds no EngineParams and calls no
LoadedEngine, so that path runs no SetWeightResidencyConfig, no
CreateWeightOffloader, no GGUF mapping and no expert slot store: there is no
field of either half that any code on it could read. Wiring it means giving the
transcription stack a loader seam, which is larger than #1135. AGENTS.md says to
refuse an unimplemented arm with a message that names the missing part, and a
warning would leave a server running while it holds a placement instruction it
does not follow — the failure #1135 was filed about. The wiring is owed as #1195
and listed under ## Owed.

The review repair

A fresh review executed every behavioural attack and all came back clean, so
nothing about what the code DOES changed. What it said about itself did.

include/vllm.h still described the pre-#1127 schema, in the one file
AGENTS.md names as the place a shipped capability is exposed. It omitted
device_fit.weight_budget_bytes from the schema block, omitted
VT_DEVICE_WEIGHT_BUDGET_BYTES from the precedence list, and its refusal list
said "a non-positive slots or slot_bytes" while never saying that
weight_budget_bytes refuses a NEGATIVE and ACCEPTS 0 — the single asymmetry
this wave exists to introduce. Measured:
git grep device_fit aed3aa5e9 -- include/vllm.h src/capi/ matches nothing, while
include/vllm/config/weight_residency.h has seven hits at the same commit, so the
key existed everywhere except the one file that is the public surface. That is
sharper than ordinary drift, because vllm-cli — this wave's own new entry point —
is a pure ABI client passing exactly the document the header called invalid, and
the loader accepted it. All three statements are there now.

A test that claimed to SKIP actually FAILED.
cmake -DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_BUILD_EXAMPLES=OFF is a legal
configuration; there VLLM_CLI_BINARY is undefined, the file defines it as "",
and all four cases ran REQUIRE(std::string(kCliBinary) != "") — four FAILED
cases on a build that simply did not include the binary. No CI job hits that
combination today, so it was latent rather than red. The tree already had the
idiom: vllm_cpp_add_test sets SKIP_RETURN_CODE 77 on every test. The four
REQUIREs become one RequireCliBinary() that exits 77, CTest reports
Skipped, and both CMake comments now say what happens. Returning early
instead would have printed doctest's assertions: 0 ... Status: SUCCESS!, which
is the #463 trap the 77 convention exists for. The repaired shape also cannot be
half-reverted: dropping only the calls leaves SkipGate unused and
-Werror=unused-function refuses the build, which is how the first attempt at
the proving mutation came back INVALID rather than green.

The setter now refuses what the parser refuses.
SetWeightResidencyConfig is declared in a public header and takes the struct, so
a hand-built config reaches the process-global having run no parser — and
ResolveDeviceWeightBudgetBytes casts the configured int64_t to size_t, so an
installed -1 resolved to SIZE_MAX: an effectively infinite budget that switched
the device-fit guard OFF silently, which is the exact failure that guard's own
refusal text names. 0 still installs, because it is the suppression spelling.

docs/USAGE.md quoted a refusal message the parser had stopped printing. W2
added device_fit to the vllm_cpp level, so RejectUnknownKeys enumerates
three names; the sample output still read (expected one of: mmap expert_stream),
in the section whose subject is that a typo is named rather than ignored. Found by
scripts/check-doc-checkpoint.py refusing the repair commit — the gate was owed a
real edit and this was it. No separate issue: the defect was introduced by this
unlanded pull request and is repaired inside it.

The counts were wrong and are corrected: #1206 below, six "five" statements
the sixth knob invalidated (weight_residency.h:62, :261, :312, :315,
weight_residency.cpp:618, test_serve_residency_config.cpp:58), a "These two"
above three cases, and the spec's attribution of three
test_gguf_device_fit_reach cases to a single mutation when it takes three.

Filed in flow

• The transcription-only server path has no seam a weight-offload or weight-residency document could reach #1195 — the transcription-only path has no seam either half could reach.
  Owned by ARCH-ONE-SURFACE, listed under ## Owed.
• The server's pooling path builds EngineParams from 8 of the engine flags and drops the rest, --device included #1196 — the pooling block drops six other EngineParams fields, device
  among them, so vllm-server --device cuda on an embedding model runs the
  accelerator-first probe. Same shape as --offload-config does not reach three entry points: vllm-cli, and the server's pooling and transcription paths #1135 over a wider set, predates this
  row, owned by SERVE-POOLING-ENDPOINTS, listed under ## Owed.
• docs/USAGE.md gives ./build/vllm-server and ./build/vllm-cli, which the build does not produce #1206 — THREE docs/USAGE.md command lines named ./build/vllm-server or
  ./build/vllm-cli, and both binaries are built under build/examples/. Fixed
  in this flow. The issue, the index row, this body and the spec first said FIVE;
  the number was never counted. grep -c 'build/vllm-' docs/USAGE.md is 3 at this
  branch's base fd64c76ee and at the merge base 5af6e763 (lines 2354, 3835,
  3848), and 0 at head. The FIX was always complete; only the count was wrong, and
  every place is corrected — the append-only index row while this pull request is
  still open, because after the merge it cannot be.

Gate

Every number below names the instrument that produced it, because two of them were
previously reported as one run and they are not the same lane.

Red first for the budget key (./build/tests/test_weight_residency_config,
whole binary). With ResolveDeviceWeightBudgetBytes stubbed to return the probe:
rc 1, 24 cases / 7 failed, 271 assertions / 15 failed — the seven are exactly
the seven cases this adds. With the implementation: rc 0, 24 cases, 315
assertions.

Red first for the setter guard, same binary, on the repair: rc 1, 25 cases
/ 1 failed, 324 assertions / 7 failed, the failing message logged as
ACCEPTED (no throw); with the guard, rc 0, 25 cases, 324 assertions. The
suite's case count moves 24 → 25, which is the check that the case was added rather
than silently skipped.

Mutations. Each applied alone from a pristine byte copy, with the file's sha256
printed before and after so a never-applied edit cannot read as a pass, the build's
exit status beside every result so a non-building mutation is INVALID rather than a
pass, a NON-ZERO doctest case count required, the LAST test cases: line taken, and
the tree restored by byte copy plus touch plus a rebuild — cp -p preserves
the mtime, ninja then skips the rebuild, and a stale mutated binary produces a kill
it has not earned. The twelve original mutations include the three call-site
deletions and the reachability mutation that removes SetWeightResidencyConfig. The
repair adds five, three of them to establish WHICH mutation reds WHICH of the three
test_gguf_device_fit_reach cases — the spec had attributed all three to one, and it
is M7 for the config-key case, M12 alone for the zero case and M4 alone for the
precedence case. One of the five came back INVALID rather than green and is recorded
as such. Both tables are in the spec's ## Evidence.

Full ctest gate, twice, because origin/main moved while the first was
running. Both are the declared recipe, cmake -S . -B build -G Ninja with no build
type so asserts stay live. On 65d6cdaed: build rc 0, ENOSPC 0, ctest --test-dir build -j 6 rc 0, 100% tests passed, 0 failed out of 523,
466.09 s. Merged onto c20018f8d and rerun: build rc 0, ENOSPC 0, ctest rc
0, 100% tests passed, 0 failed out of 523, 587.09 s. The second is the one
that counts, because a clean merge is not a merge that builds. It is slower for
contention only — the box's load average was above 200 for much of it.
test_modelopt_mixed_precision_checkpoint and test_voxtral_e2e are Skipped in
both, as they are on this host. The count is 523 rather than the 516 recorded
earlier because origin/main added test binaries in between.
scripts/check-symbol-anchors.py, which arrived in that interval, reports OK with
618 citations and 0 stale, and scripts/agent-preflight.sh is rc 0 with no failing
gate.

tests/scripts/test_cpu_x86_llamacpp_floor.py is a DIFFERENT instrument and is
not one of those tests. It is a Python harness contract run by ci.yml:210 and
scripts/agent-preflight.sh:145; ctest -N | grep -i floor matches nothing. It
fails on this box under load, and the isolation showing the change is not the cause
is NOT the concurrent A/B this body previously reported. That A/B was
self-poisoning: each arm is the other's contention and the failure lands on
whichever loses the race, which is why a rerun put the failure on the control arm
and inverted the result. The 3-versus-2 claim is withdrawn. What isolates it is
two things that do not race. The three files the harness reads are byte-identical on
this branch and on origin/main (358927d17, bd7a51925, 94f8509c4 on both
sides). And a SINGLE-arm sweep on this branch alone, three runs on one tree: rc
1 at load 172→194, where the harness's own quiet gate fires
NO_QUIET_WINDOW after 15s and the run takes 114.7 s; rc 1 again at load
92→107, same gate, 86.0 s; and rc 0 at load 53→49, Ran 10 tests in 3.338s / OK. scripts/agent-preflight.sh is rc 1 exactly when this harness is, and rc 0
when it is not. All of it is in the spec's ## Evidence.

A doctest filter trap, recorded because it cost a reading. Two case names here
contain a comma and --test-case= splits its argument on commas, so filtering to
one of them ran 0 cases and printed Status: SUCCESS!. Every result above comes
from running a whole binary and reading its case count.

Closes #1127. Closes #1135. Closes #1206.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]