52 integration tests in opencli-core fail, and had never run

[charliewwdev]

cargo test -p opencli-core --test all — 595 pass, 52 fail, 15 ignored.

They are not new. Every job that would have run them was scheduled onto a self-hosted runner group belonging to the project this was forked from, so from the first commit here they failed to schedule and reported nothing. The unit tests (--lib, 1,036 of them) pass and always have; it is the integration suite that was invisible.

By area:

Count	Area
7	truncation
6	remote_models
5	otel
5	compact
4	shell_serialization
4	rmcp_client
4	cli_stream
3	personality
2 each	tools, prompt_caching, list_models, compact_resume_fork
1 each	user_shell_cmd, unified_exec, tool_harness, model_tools, model_info_overrides, grep_files

Spot-checking suggests most are stale expectations rather than broken behaviour, in two families:

• Wording drift. The truncation tests expect a marker reading "N tokens truncated"; the output says "chars truncated".
• Fixture neutralisation. Model slugs that carried capabilities, default efforts and context windows were replaced with a test-model that carries none, and the assertions still describe the old model. Four tests in suite::client were repaired this way already — see the commit that fixed them for the pattern.

Until this is cleared, the Tests jobs in rust-ci are red on every commit.

[charliewwdev]

Ran against 6ec6b9c: 53 failures, against the 52 recorded above.

The extra one is mine and is described here rather than left to be found:

suite::compact::auto_compact_allows_multiple_attempts_when_interleaved_with_other_turn_events
now issues the six requests it expects — the count assertion passes — and
fails on first request should contain the user input, so something about
their order has moved. It followed the compaction changes in b5b1444
and 6ec6b9c; the fixture is unmodified.

Two others that were failing for the same reason are fixed in 6ec6b9c,
along with the design fault behind them: a compaction was being judged
against the size of the whole request rather than the size of the
conversation, so the fixed per-request overhead counted as growth.

Worth adding to the description above: these are the tests that would have
caught both regressions the moment they were introduced. They did — several
hours later, and only because the suite was run by hand.

[charliewwdev]

The count above is measured the way I measured it, and is inflated by seven.

suite::rmcp_client (4) and three of suite::truncation need a helper
binary, test_stdio_server, that lives in opencli-rmcp-client. Because
it is in another crate, CARGO_BIN_EXE_test_stdio_server is never set for
these tests; they fall back to looking in target/debug, which
cargo test -p opencli-core --test all does not fill. CI runs
cargo nextest run --all-features over the whole workspace and builds it,
so these seven have always passed there.

To reproduce what CI sees:

cargo build -p opencli-rmcp-client --bin test_stdio_server
cargo test -p opencli-core --test all

or just cargo nextest run --all-features.

The real figure is 41, at 5394086:

Count	Area
6	remote_models
6	compact
5	otel
4	cli_stream
3	personality
2 each	tools, shell_serialization, prompt_caching, list_models, compact_resume_fork
1 each	user_shell_cmd, tool_harness, model_tools, model_info_overrides, grep_files

truncation is now clear. Four were real and are fixed in 5394086:
three matched a marker production cannot emit — every model's policy is
byte-based, and TruncationPolicyConfig::tokens is constructed in one
branch that nothing ever reaches — and one hard-coded the number of
characters removed.

That last point deserves its own line: token-mode truncation is
unreachable. tool_output_token_limit is a token budget silently
converted to bytes. Either it gets wired up or it should be removed;
leaving it means a config option that does not do what its name says.

[charliewwdev]

39 at 5d84c90. truncation is clear; two more in compact were wording
that had drifted for a reason worth keeping, and are fixed.

The remaining six under compact and compact_resume_fork are one finding,
recorded here so whoever picks this up does not have to rediscover it:

auto_compact_counts_encrypted_reasoning_before_last_user
auto_compact_allows_multiple_attempts_when_interleaved_with_other_turn_events
multiple_auto_compact_per_task_runs_after_token_limit_hit
auto_compact_triggers_after_function_call_over_95_percent_usage
compact_resume_after_second_compaction_preserves_history
compact_resume_and_fork_preserve_model_history_view

All six turn on how many compactions happen in a task, and that changed
deliberately in b5b1444/6ec6b9c. Being over the limit is no longer
sufficient: a compaction also has to have somewhere to go, meaning the
conversation has grown past where the last one left it by at least a tenth
of the limit. Without that rule a session summarised itself before eleven
consecutive answers, throwing the thread away each time, because the fixed
per-request overhead kept the total over the limit no matter how much was
summarised.

These fixtures run in a 100-token window with short mock replies, so the
growth between turns is small in absolute terms and the second compaction
is suppressed. Two possibilities, and I have not established which:

1. The tests encode the old contract and should assert the new one.
2. The rule is too strict at small scale — the gain is limit / 10, which
   is 7 tokens here, and something in the accounting is not clearing it.

Anyone starting on these should settle that question first; changing the
fixtures before knowing would hide the answer either way.

[charliewwdev]

27 at fd4053c, from 52. Three findings account for most of it, and each
is a thing worth knowing rather than a test to fix:

The harness argued with a finished server. A mock serves one response;
everything after gets a 404, which the client retried with backoff up to the
provider's limit. Tests waiting for the turn to end waited through all of it
and timed out — the failure reading as broken behaviour rather than as the
harness. Retries are now off by default in test_opencli, as two other test
binaries here already did. Five went green and the suite runs six seconds
faster. One test wants them and now asks: the websocket turn-state flow
reconnects within a turn.

Two capabilities the build cannot reach. Personality templates need a
model carrying model_messages, and model_messages: Some(..) appears
nowhere. Token-mode truncation needs a model in token mode, and
TruncationPolicyConfig::tokens is constructed in one branch that requires
one — so tool_output_token_limit is a token budget silently converted to
bytes, a config option that does not do what its name says.

core/models.json is empty. Zero models, so no default model, no
built-in presets, nothing for a remote entry to replace. Six tests are about
a catalogue that is not there.

The last two are the same decision in two places: wire it up, or take it out.
Neither should stay as red tests nobody can act on, and neither is fixed by
putting somebody else's model catalogue back.

What is left, all of it needing a look rather than a rename:

Count	Area
4	compact — the compaction-frequency change, described in the comment above
4	cli_stream
2 each	tools, shell_serialization, prompt_caching, list_models, compact_resume_fork
1 each	user_shell_cmd, tool_harness, otel, model_tools, model_info_overrides, grep_files

[charliewwdev]

Correcting one of the three findings above.

I wrote that the empty core/models.json was "arguably right". It is not
arguable — it is the design. Models come from configuration and from
whatever runtime is on the machine, and the feature that fills the bundled
list, Feature::RemoteModels, is already off by default with a comment
saying why: it fetches OpenAI's catalogue, no other provider serves one, and
leaving it on made every startup fire a doomed request and log a 401 nobody
could act on.

So those six are not "a catalogue that is missing". They cover a feature that
is off, reading a list that is empty on purpose. Marked for that reason
instead, in fd4053c's follow-up.

That also removes one of the two "wire it up or take it out" decisions I
listed. Only one remains, and it is the sharper of the two:
tool_output_token_limit is a token budget silently converted to bytes,
because token-mode truncation is unreachable. A config option that does not
do what its name says is worth deciding about whatever happens to these tests.