fix(cache): default TOKENFUSE_CACHE to off, not shadow - #326
Merged
Merged
Conversation
Shadow mode ran SemanticCache::get on every non-streaming eligible call even when no operator had ever configured the cache: one global Mutex over the whole store, a retain() TTL sweep, then a linear cosine walk against every stored entry. Issue #319 measured 79-92% of gateway CPU in SemanticCache::get under a 4-core load test, with throughput dropping from 1102 req/s to 177 req/s (with 403s) once the cache carried load, all for a feature nobody had turned on. TOKENFUSE_CACHE now defaults to off, unset => Off, off => Off, shadow => Shadow, on => On, and any other value => Off with one warn line naming it - the same "measurement feature, safer default" shape as TOKENFUSE_TOOLS_PRUNE (invariant 61). The parse moves out of main.rs into tokenfuse_gateway::defaults::cache_mode_from, a pure, unit-testable function, same shape as tools_prune_mode_from. Adds CLAUDE.md invariant 63, features/the-cache-defaults-off.feature (three scenarios, each bound), and three tests in gateway::defaults, the first run red against the unfixed default (left: Shadow, right: Off) before the fix landed. components.json, README.md and PROGRESS.md test counts updated to match. Refs #319 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CLAUDE.md invariant 63's unit tests prove cache_mode_from is correct in isolation; they cannot prove serve() actually calls it rather than keeping its own inline match (with the old Shadow fallback). Reverting main.rs to that old match leaves every defaults::tests test green. crates/gateway/tests/cache_default_startup.rs closes that seam by running the real built binary (same technique as tests/stub_wire_mismatch.rs) and reading the one piece of evidence main.rs exposes for this decision, the startup line tracing::info!(?cache_mode, "semantic cache"): - unset_cache_resolves_to_off_in_the_real_binary - a_typo_cache_value_resolves_to_off_and_warns_in_the_real_binary (also asserts the warn line names the variable and the value; this binary's tracing setup has no separate stderr routing by level, so the warn line is one of the captured stdout lines, not stderr) - cache_on_is_honoured_in_the_real_binary, the negative control Run red first against main.rs reverted to the old inline match: both non-control tests failed with "got: ... semantic cache cache_mode=Shadow" where "cache_mode=Off" was expected; the negative control stayed green on both sides. Recorded in CLAUDE.md invariant 63's gate marker. Gateway test count: 834 -> 840 (three defaults:: unit tests already counted, three more here). Workspace total: 1489 -> 1492. Refs #319 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Owner
Author
Addendum: seam test proving
|
TAIPANBOX
added a commit
that referenced
this pull request
Sep 24, 2026
Conflicts were documentation only: invariants 63 and 64 kept in order, the test count is the sum of both branches (1509, core 359, gateway 840), checked by stated-numbers.sh after cargo test --all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
TOKENFUSE_CACHEnow defaults tooffinstead ofshadow. Unset oroff=>
Off;shadow=>Shadow;on=>On; any other value =>Offwithone warn line naming it. The parse moves out of
main.rsinto a new purefunction,
tokenfuse_gateway::defaults::cache_mode_from(andcache_mode_from_env), the same shape astools_prune_mode_from(invariant 61).
Why
Issue #319: with
TOKENFUSE_CACHEunset the gateway ran the semanticcache in
Shadow, and every non-streaming eligible call ranSemanticCache::getthrough one globalMutex, aretain()TTL sweepover the whole partition, then a linear cosine walk against every stored
entry, whether or not anyone had ever configured the cache. Measured on a
4-core box: 79-92% of gateway CPU in
SemanticCache::get, throughputdropping from 1102 req/s to 177 req/s (with 403s) once the cache carried
load. A measurement feature should not cost every request by default;
shadowandonstay one variable away.This PR is the gateway-side default only. The three launchers that set
TOKENFUSE_CACHE=offexplicitly are a separate, already-in-flight change.Red-first
defaults::tests::cache_is_off_when_nothing_is_configured, run against aversion of
cache_mode_fromwhose fallback wasShadow(the unfixedbehaviour):
Reverting the fallback back to
Offturns it green, along with the othertwo new tests (
every_named_cache_mode_is_honoured,an_unrecognised_cache_value_is_off_not_a_guess).Tests
3 new (all in
gateway::defaults), 1489 total across the workspace, allgreen (
cargo test --all).Gates
cargo fmt --all -- --check,cargo clippy --all-targets,cargo test --all,cargo test -p tokenfuse-gateway --features cluster --test cluster_backend, everyscripts/*.shin CLAUDE.md's gate list(including
stated-numbers.sh, updated for the new count, andcompat-surface.sh, unaffected sinceTOKENFUSE_CACHE's default is notpart of the frozen 1.0 surface, only the variable name is), and
gates-have-teeth.shafter committing: 65 cases, all as expected.Also added:
features/the-cache-defaults-off.feature(3 scenarios, eachbound) and CLAUDE.md invariant 63.
NOT proven
This changes only the gateway's own default; it does not touch the three
launcher repos that were already being updated to set
TOKENFUSE_CACHE=offexplicitly, and it does not change anything aboutShadoworOnmode behaviour itself (that is PR B,fix/cache-lookup-without-a-global-walk).Refs #319
🤖 Generated with Claude Code