Skip to content

fix(cache): default TOKENFUSE_CACHE to off, not shadow - #326

Merged
TAIPANBOX merged 2 commits into
mainfrom
fix/cache-default-off
Sep 24, 2026
Merged

TAIPANBOX merged 2 commits into
mainfrom
fix/cache-default-off

Conversation

@TAIPANBOX

Copy link
Copy Markdown
Owner

What

TOKENFUSE_CACHE now defaults to off instead of shadow. Unset or off
=> Off; shadow => Shadow; on => On; any other value => Off with
one warn line naming it. The parse moves out of main.rs into a new pure
function, tokenfuse_gateway::defaults::cache_mode_from (and
cache_mode_from_env), the same shape as tools_prune_mode_from
(invariant 61).

Why

Issue #319: with TOKENFUSE_CACHE unset the gateway ran the semantic
cache in Shadow, and every non-streaming eligible call ran
SemanticCache::get through one global Mutex, a retain() TTL sweep
over the whole partition, then a linear cosine walk against every stored
entry, whether or not anyone had ever configured the cache. Measured on a
4-core box: 79-92% of gateway CPU in SemanticCache::get, throughput
dropping from 1102 req/s to 177 req/s (with 403s) once the cache carried
load. A measurement feature should not cost every request by default;
shadow and on stay one variable away.

This PR is the gateway-side default only. The three launchers that set
TOKENFUSE_CACHE=off explicitly are a separate, already-in-flight change.

Red-first

defaults::tests::cache_is_off_when_nothing_is_configured, run against a
version of cache_mode_from whose fallback was Shadow (the unfixed
behaviour):

thread 'defaults::tests::cache_is_off_when_nothing_is_configured' panicked at crates/gateway/src/defaults.rs:257:9:
assertion `left == right` failed
  left: Shadow
 right: Off

Reverting the fallback back to Off turns it green, along with the other
two new tests (every_named_cache_mode_is_honoured,
an_unrecognised_cache_value_is_off_not_a_guess).

Tests

3 new (all in gateway::defaults), 1489 total across the workspace, all
green (cargo test --all).

Gates

cargo fmt --all -- --check, cargo clippy --all-targets,
cargo test --all, cargo test -p tokenfuse-gateway --features cluster --test cluster_backend, every scripts/*.sh in CLAUDE.md's gate list
(including stated-numbers.sh, updated for the new count, and
compat-surface.sh, unaffected since TOKENFUSE_CACHE's default is not
part of the frozen 1.0 surface, only the variable name is), and
gates-have-teeth.sh after committing: 65 cases, all as expected.

Also added: features/the-cache-defaults-off.feature (3 scenarios, each
bound) and CLAUDE.md invariant 63.

NOT proven

This changes only the gateway's own default; it does not touch the three
launcher repos that were already being updated to set
TOKENFUSE_CACHE=off explicitly, and it does not change anything about
Shadow or On mode behaviour itself (that is PR B,
fix/cache-lookup-without-a-global-walk).

Refs #319

🤖 Generated with Claude Code

Shadow mode ran SemanticCache::get on every non-streaming eligible call
even when no operator had ever configured the cache: one global Mutex
over the whole store, a retain() TTL sweep, then a linear cosine walk
against every stored entry. Issue #319 measured 79-92% of gateway CPU
in SemanticCache::get under a 4-core load test, with throughput
dropping from 1102 req/s to 177 req/s (with 403s) once the cache
carried load, all for a feature nobody had turned on.

TOKENFUSE_CACHE now defaults to off, unset => Off, off => Off,
shadow => Shadow, on => On, and any other value => Off with one warn
line naming it - the same "measurement feature, safer default" shape
as TOKENFUSE_TOOLS_PRUNE (invariant 61). The parse moves out of
main.rs into tokenfuse_gateway::defaults::cache_mode_from, a pure,
unit-testable function, same shape as tools_prune_mode_from.

Adds CLAUDE.md invariant 63, features/the-cache-defaults-off.feature
(three scenarios, each bound), and three tests in gateway::defaults,
the first run red against the unfixed default (left: Shadow, right:
Off) before the fix landed. components.json, README.md and
PROGRESS.md test counts updated to match.

Refs #319

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CLAUDE.md invariant 63's unit tests prove cache_mode_from is correct
in isolation; they cannot prove serve() actually calls it rather than
keeping its own inline match (with the old Shadow fallback). Reverting
main.rs to that old match leaves every defaults::tests test green.

crates/gateway/tests/cache_default_startup.rs closes that seam by
running the real built binary (same technique as
tests/stub_wire_mismatch.rs) and reading the one piece of evidence
main.rs exposes for this decision, the startup line
tracing::info!(?cache_mode, "semantic cache"):

- unset_cache_resolves_to_off_in_the_real_binary
- a_typo_cache_value_resolves_to_off_and_warns_in_the_real_binary
  (also asserts the warn line names the variable and the value; this
  binary's tracing setup has no separate stderr routing by level, so
  the warn line is one of the captured stdout lines, not stderr)
- cache_on_is_honoured_in_the_real_binary, the negative control

Run red first against main.rs reverted to the old inline match:
both non-control tests failed with "got: ... semantic cache
cache_mode=Shadow" where "cache_mode=Off" was expected; the negative
control stayed green on both sides. Recorded in CLAUDE.md invariant
63's gate marker.

Gateway test count: 834 -> 840 (three defaults:: unit tests already
counted, three more here). Workspace total: 1489 -> 1492.

Refs #319

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@TAIPANBOX

Copy link
Copy Markdown
Owner Author

Addendum: seam test proving main.rs actually calls cache_mode_from

Follow-up requested in review: the unit tests in gateway::defaults prove
cache_mode_from is correct, but nothing proved main.rs's serve()
actually calls it rather than keeping its own inline match. Reverting
serve() to the old match would leave every one of those tests green.

Added crates/gateway/tests/cache_default_startup.rs, which runs the real
built binary (same technique as tests/stub_wire_mismatch.rs) and reads
the startup line tracing::info!(?cache_mode, "semantic cache"):

  • unset_cache_resolves_to_off_in_the_real_binary
  • a_typo_cache_value_resolves_to_off_and_warns_in_the_real_binary (also
    asserts the warn line names the variable and the value; this binary
    routes every tracing level to the same stdout stream, so the warn is one
    of the captured stdout lines, not stderr)
  • cache_on_is_honoured_in_the_real_binary, the negative control

Red first, against main.rs reverted to the old inline match
(_ => CacheMode::Shadow):

thread 'unset_cache_resolves_to_off_in_the_real_binary' panicked:
TOKENFUSE_CACHE unset must resolve to Off in the real binary; got:
... semantic cache cache_mode=Shadow
```//
same failure shape for the typo-value test; the negative control (`on`)
stayed green on both sides, as a guard should.

CLAUDE.md invariant 63's gate marker updated to name this file. Gateway
test count 834 -> 840 (3 already-counted unit tests + 3 new here);
workspace total 1489 -> 1492. All gates re-run clean: fmt, clippy (including
`--all-features -- -D warnings`), `cargo test --all`, the cluster feature
test, `audit.sh`, every other `scripts/*.sh`, and `gates-have-teeth.sh`
(65 cases, all as expected).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@TAIPANBOX
TAIPANBOX merged commit 6a8294e into main Sep 24, 2026
10 checks passed
@TAIPANBOX
TAIPANBOX deleted the fix/cache-default-off branch September 24, 2026 11:50
TAIPANBOX added a commit that referenced this pull request Sep 24, 2026
Conflicts were documentation only: invariants 63 and 64 kept in order, the
test count is the sum of both branches (1509, core 359, gateway 840), checked
by stated-numbers.sh after cargo test --all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant