fix(benchmark): an UNGRADEABLE grade is an ABSENCE, not a capability zero - #2281
Closed
joelteply wants to merge 17 commits into
Closed
fix(benchmark): an UNGRADEABLE grade is an ABSENCE, not a capability zero#2281joelteply wants to merge 17 commits into
joelteply wants to merge 17 commits into
Conversation
… as text (#274/#396) Joel, 2026-08-13: "Always use uuid and never corrupt them with random prefixes" and "UUID's are NOT strings. If you're using strings for id you are writing slop." Both landed on code written minutes earlier, and the grep that followed found the same defect older and wider than my one file. Three fixes, in order of how badly they were wrong: 1. The four shipped recipe ids were hand-drawn patterns (c0de0001-0000-4000-...), a name wearing a UUID's costume — readable, collidable, and fake. Replaced with genuine v4s in both the authored JSON and the `shipped::` constants, written grouped (0xfed332c3_383c_45bb_...) so the two are checkable by eye. 2. comms::{MessageId, CorrelationId} were `String` newtypes. `MessageId::new("msg-1")` let any caller invent a namespace that collides with every other caller's. Both are now `Uuid`: MessageId::new() MINTS (no caller-supplied form exists), and CorrelationId::of_exchange(id) states the derivation the old `CorrelationId(id.0.clone())` left implicit. Still distinct TYPES — the compiler, not a naming convention, is what stops one being passed as the other. 3. EndpointId is deleted. Its values were `EndpointId::new("browser")` and `("rust-core")` — a client-kind label standing in for an identity, and with it the assumption that the web client is a distinguished endpoint. It is one client among many (mobile, SDK, TUI, another node's core). TransportEnvelope.source and .target are now PeerId, the substrate's one actor identity per identity/mod.rs. Zero callers outside comms/mod.rs; the stale generated binding goes too. Also closes the on-disk authoring hole the required `id` field opened: a recipe file that names no id now gets one DERIVED from its purpose (RFC 4122 v5 under a frozen namespace), so "author a file, zero code" keeps meaning zero code — no uuidgen — and every node that loads the same file agrees on its identity with nothing to reconcile. A recipe that DOES carry an id keeps it verbatim. Tests: comms 23 pass (envelope wire shape now asserts ids round-trip as their own UUIDs; minted_message_ids_are_unique pins the collision fix), experience 43 pass including the previously-failing an_experience_authored_on_disk_needs_no_rust. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…t, not the deploy path Joel, 2026-08-13: "Headless rust period. No need for node to run everything except for the web interface which is one of many, including mobile apps/sdk." — and then, because I kept treating the correction as a code-only matter: "Fix these severe misunderstandings ... regardless of where they are." This file is where the misunderstanding REPRODUCES. It is loaded into every agent session, and it opened with "EVERY TIME YOU EDIT CODE: Run `npm start` (MANDATORY)" under `cd src` — a directory that no longer exists — plus "ALL Rust binaries MUST be built via npm start". A fresh agent under amnesia reads that and concludes Node runs the system. It doesn't, and `continuum --help` has been saying so in its own first line the whole time: "build + run the headless Rust core". Rewrote the CRITICAL WORKFLOW section around what is actually true, and swept the other 11 Node-as-deploy claims scattered through the file: - The core is Rust and boots with no Node. Node builds the WEB desktop, which is one client among several (mobile, SDK, TUI, MCP, another node's core over the grid). Named the consequence, because it already cost us: a feature that lives in a client exists only for that client — how voice ended up web-only with every other citizen structurally mute (#58). Behaviour goes in the core; clients render. - The deploy path is `continuum reboot` (Rust build + relaunch + running-SHA verify), with `deploy-verify` and the version trio called out — that verification exists because a reboot once shipped a stale binary and reported success (#194). - `cargo build` stays discouraged for the RIGHT reason (a hand-built binary exists only on your machine, and a fresh clone must work with no manual steps, #291) — not the old implication that Rust must be built through npm. `cargo check` is named as the correct type-check-while-you-work tool, with the shared CARGO_TARGET_DIR. - `npm run build:ts` now says what it actually covers: the web client, and nothing about whether the core compiles. - Flagged `./jtag` inline as the legacy Node CLI. Left the invocations that follow, since their command NAMES are still accurate — it is the driver that is stale. No code change; the one surviving "npm start" is the sentence warning you off it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ed to a binary that does not exist
Joel: "Cu conflicts with some Unix program. Fix for all cases including windows"
and "It's a bug Claude" — it is, and worse than doc drift.
`cu` is /usr/bin/cu (UUCP call-unix) on every Unix. `uu` — the double-U of
contin-UU-m — is THE official short alias, and start-server.sh has installed
exactly that (plus a squatter guard) since 2026-08-01, with a comment naming this
very collision. Nothing in the tree installs `cu`. But the REFERENCES never
followed:
- Six benchmark harnesses defaulted to `~/.continuum/cache/cargo-target/{release,debug}/cu`.
Neither file exists — the only built CLI is `release/continuum`. So the default
resolved to nothing and every default-args run died at the first invocation.
matrix.py even carries a comment about this exact class of failure biting once
before (2026-07-22, stale debug-only default silently no-opping the sweep).
- Fixed by RESOLVING rather than renaming: a shared `_resolve_cli()` prefers what
is actually installed on PATH (`uu`, then `continuum`) and falls back to the
release build — so it works from a fresh clone, an installed box, or a dev tree,
on any platform, instead of hard-coding one machine's layout.
- Flag renamed `--cu` → `--uu` with all in-repo call sites updated
(sweep_all → matrix → headtohead → preflight_gpu chain).
Also swept 53 `cu <command>` occurrences in docs, Rust comments, and generated-TS
doc comments to `uu`. Left the ones that are ABOUT the collision (memory-bridge
README's "never bare `cu`", WAKEUP-AND-JOIN's rename note) — those are correct as
written, and legacy/ stays quarantined.
README dev section reworked in the same pass: the boot path is `continuum start` /
`continuum reboot` / `continuum ping`, with Node named as what it actually is —
the web client's build dependency, one client among mobile/SDK/TUI/MCP, not the
thing that runs the system.
Verified: all six harnesses compile (py_compile), cargo check clean, zero
`target/{release,debug}/cu` paths left in the tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ilently swallowing the job
Joel: "Fix all nodejs and python dependencies that are breaking our headless rust
core ... we must find all smell all the time and never ignore it."
AUDIT RESULT FIRST, because one half is good news:
Node spawns from the running core: ZERO. `Command::new("node"|"npm"|"npx"|
"deno"|"bun")` has no occurrences anywhere in continuum-core. The core is
genuinely headless Rust at runtime; Node exists only to build the web client.
Python spawns from the running core: FOUR sites, and neither kind was honest.
1. THREE were vestigial and were hiding untested code. `file_engine.rs` had three
`#[test]`s opening with `if Command::new("python3").arg("--version").output()
.is_err() { return; }` — a skip-guard from when the syntax gate shelled out to
`python3 -m py_compile`. That interpreter is GONE: the production path is
`code::syntax::validator_for` → `unbound_calls`, pure Rust. So on any box
without python3 — a CI runner, a fresh clone — three tests reported PASS while
asserting nothing at all. Guards removed; all 55 file_engine tests pass without
an interpreter present, which is the proof they never needed one.
Also corrected the doc on `introduced_undefined_calls`, which still told the
reader the analysis returns None when there is "no python". There is no python.
2. ONE is a REAL runtime dependency, and it was failing silently. `forge/start`
spawns `python3 <alloy_executor>` — a script that lives in the SIBLING
sentinel-ai repo, which a fresh clone of continuum does not have. When
`find_alloy_executor()` returned None the handler used pid 0 and wrote
`state: "queued"` — indistinguishable from a job legitimately waiting its turn.
So on every machine that had only cloned this repo, `forge/start` returned
SUCCESS for work that nothing would ever run. Now it fails loud, names the
missing script, says where to get it, and states plainly that the job was NOT
queued. The dependency itself is still there — excising it to Rust is #52/#99
— but it can no longer pretend to have worked.
The pattern in both: a Python dependency that had already been removed or had
never been satisfiable, still shaping behaviour through a stale guard and a
fallback. `[[fallbacks-are-illegal-fail-loud]]`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…, and the compiler now knows Joel: "UUID's are NOT strings … Well defined and named structs by reference must be used." Chasing that through persona_id turned up something sharper than the count. THE FINDING. `PersonaWorkspaceRegistry::resolve_persona` exists specifically to close what its own doc calls "the loose-`String` id boundary … the defect class that fed a dead id to a doomed eval." It has SEVEN call sites. SIX are its own tests. ONE is production (eval.rs). Against 55 `persona_id: String` fields. The check was correct and essentially nothing called it — a correct check nothing calls is nastier than a missing one, because it reads as covered. Nothing forced the call, because both sides were `String`. THE FIX — two types, one door: - `PersonaRef` (new, in identity/): what a CALLER writes — full UUID, 8-char short-id, or name. Explicitly NOT an identity. Its only accessor is `as_str()`; there is no `as_peer_id()`, because a name is ambiguous, mutable, and meaningless without a roster. - `PeerId` (existing canonical actor identity): what everything downstream holds. - `resolve_persona(&PersonaRef) -> Result<PeerId, _>` is now the ONLY bridge. Taking the newtype rather than `&str` is what makes resolution unskippable. `From<PeerId> for PersonaRef` exists (an identity is always a valid reference to itself); the reverse deliberately does not — it requires a roster. Wire shape is unchanged: `#[serde(transparent)]` over the same string callers already send, so no client, recipe, or stored payload changes. Short-id and name PX (#161) keeps working — that ergonomics is the whole reason a reference type has to exist separately rather than everything becoming a UUID. Converted, types pushed DOWN rather than laundered at the seam: - `CognitionEvalParams.persona_id` → `PersonaRef` - `restore_persona_workspace(&PersonaRef)` (was `&str`) - `append_failed_ledger(&PersonaRef, …)` (was `&str`) HELD, and stated rather than fudged: `CognitionEvalResult.persona_id` stays `String`. The struct derives `Default` across 21 fields and a persona reference has no sensible default — an empty one is a nonsense value that reads as a real answer. The real fix is splitting the fire-and-poll HANDLE from the completed RESULT (a handle knows only the requested ref; a result knows the resolved id), which is its own slice. Inventing a default to satisfy the type checker is the `unwrap_or` reflex: compiler quiet, runtime wrong. The reason is recorded at the field. Tests: persona_workspace 10, eval 22, identity 23 — all pass; full lib test build clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… 21 sites Second slice of the id-typing migration. All ten `commands/memory/*` params plus the `MemoryManager` API they call now carry `PersonaRef` instead of `String`/`&str`: append_memory, append_event, load_corpus, has_corpus, get_corpus, multi_layer_recall, consciousness_context, persona_db_handle, hydrate_corpus_if_missing. Types went DOWN into the layer rather than being unwrapped at each call — the whole point of the previous slice. `.as_str()` now appears only where the value is genuinely being USED as text (a map key, a `starts_with` shape check, a directory handle), never to satisfy a signature one call later. Two `Default` derives removed (`ConsciousnessContextParams`, `LoadCorpusParams`) rather than giving `PersonaRef` a default. No caller used `::default()` on either, and an empty persona reference is a nonsense value that reads as a real answer — same reasoning as the eval result field held in the previous commit. What this makes visible, and does not yet fix: these commands still never RESOLVE. They accept a reference and hand it straight to the storage layer as a key, so a name or short-id reaches the DB unresolved. That was invisible while everything was `String`; it is now legible in the signatures. Wiring `resolve_persona` into the memory command path is the next slice (#164/#396). Tests: memory 219, rag 150 — all pass. Full lib test build clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…rsona/*, cognition/observe
Third slice. Seven more command param structs stop typing a persona reference as
`String`: agent/solve, persona/identity/{get,set}, persona/instances/{get,despawn},
persona/wall/pin, cognition/observe (params + the `assemble` signature + its `Meta`,
so the type is consistent through the result rather than converted on the way out).
Same discipline as the memory slice: `.as_str()` appears only where the value is
being USED as text — `id_resolve::resolve` takes a `&str` by design because it also
serves rooms and cards — never to satisfy a signature.
Running total across the three slices: 39 persona params + the memory API + the
resolver itself. `persona_id: String` is down from 55 to 16 in the crate, and every
one that remains is an internal struct holding an already-resolved id (those want
`PeerId`, the next slice) rather than an unresolved caller reference.
Tests: full lib test build clean, commands suite passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…rce path carries it down Fourth slice. `CognitionTraceParams`, the four other introspect params, `CognitionReplayParams` + result, `RagComposeRequest`, and `DatasetFromTurnsParams` now carry `PersonaRef`. The RAG one went two levels deep rather than stopping at the param: `load_source`, `load_memory_source`, and `load_consciousness_source` all take `&PersonaRef` now, which deleted the two `&persona_id.into()` conversions the memory slice had left at those call sites. That is the shape to aim for — when the type reaches the bottom, the adapters in the middle disappear rather than accumulating. `persona_id: String` in continuum-core: 55 → 8. Every remaining one is an internal RECORD (memory/types, should_respond's AIDecisionContext, live/types, projection, shell_types, ai/types, sentinel) holding a value copied from a param. Those are deliberately NOT converted to `PeerId` yet. They hold whatever the caller sent, and nothing on those paths resolves — typing them as an identity today would assert something the code does not do, which is worse than leaving them `String`. They become `PeerId` in the same slice that wires `resolve_persona` into those paths, not before (#164/#396). Tests: replay 3, rag 18, full lib test build clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…— the stub's premise expired `StubAircCitizen::subscribe_all_rooms` was an `unreachable!()` justified by a comment that said no test drives it. That was true when written and stopped being true at bf11a66 (#398 slice 3), which gave `PersonaSupervisor::materialize` a `subscribe_all_rooms` call to wire the doctrine/wall cache invalidators. Every supervisor test that materializes an adapter has been dying in that panic since — 5 of the 10, and they only surface on a FULL-suite run, which is how they sat unnoticed. Found while verifying my own id-typing slices: full suite came back 7073 passed / 5 failed, and the first question was whether I caused it. I did not — the diff of my five commits touches `commands/persona/*` only, never `persona/supervisor.rs` or `persona/airc_citizen.rs`, and `git log` on those two files points at #398. Fix: return `AircError::Transport` instead of panicking. This is NOT a fallback — the caller already handles that exact case explicitly (keeps both sources uncached, "correct, just slow", logs loud), so the tests now exercise the real degradation branch rather than aborting, and a stub still never pretends to hold a live stream. An empty stream WOULD have been the fallback: it would have looked like a working subscription that silently never invalidates. Comment rewritten to state what is true now, including why the old assertion was right when it was written. An assertion that outlives its premise is worse than no assertion — it reads as a guarantee. persona::supervisor: 10/10 pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…e panic I removed
`stub_subscribe_panics_loudly` existed to prove the `unreachable!()` fired, so removing
that panic left the assertion failing for the right reason. Rewritten to pin what the
contract actually is now:
Err(Transport) — required
Err(anything else) — fail (the caller branches on Transport specifically)
Ok(stream) — fail LOUDEST, because that is the real fallback: a stub handing back a
stream looks like a live subscription that silently never invalidates
Matched rather than `expect_err`d because `FilteredEventStream` is not `Debug`.
FULL LIB SUITE NOW GREEN: 7078 passed, 0 failed, 42 ignored. Before this session's
last two commits it was 7073/5 — five supervisor tests panicking since #398 slice 3,
only visible on a full run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ARED, or the build fails Joel: "eliminate all smell or you will copy it." That is literally the mechanism — a model reading this tree learns its conventions FROM it, and `persona_id: String` was 55 sites teaching that ids are text. I did exactly that this session: minted `c0de0001-…` fake UUIDs because fake-looking ids were already normal here. Prose in CLAUDE.md does not stop that. A failing test does. Three tests in identity/mod.rs, running on every PR via the existing `cargo test -p continuum-core --lib` workflow (no skip pattern matches them): 1. `every_string_typed_identity_field_is_declared` — walks src/, finds any `<identity-name>_id: String | Option<String>` field, and fails unless it appears in LOOSE_IDS. Comments stripped first, so a doc line can never register as a field. The error names the file:field and tells the reader which typed form to reach for (`PeerId` for an actor, `PersonaRef` for an unresolved reference, a `*Id(Uuid)` newtype otherwise). 2. `no_declaration_outlives_its_defect` — a declaration whose field HAS been fixed fails too. Learned directly from `StubAircCitizen::subscribe_all_rooms`, whose comment stayed true-sounding for months after its premise expired and cost 5 silently-failing tests. 3. `declarations_carry_a_real_reason` — every entry starts with external:/pending:/ defect: and is longer than a shrug, the same bar the module-wiring audit (#344) holds. 73 declarations, honestly categorized: - **external** — LiveKit participant/room ids, log-envelope correlation fields. Another system owns the wire format. - **pending** — ours, but nothing on that path RESOLVES yet. Typing it as an identity today would assert something the code does not do. Converts in the slice that wires resolution (#164/#396). - **defect** — `peer_id: String` × 5. These ARE `PeerId`. I attempted the conversion in this session and reverted it: `PeerId` has no `JsonSchema` impl and the construction sites hold `&str`, so it needs its own slice rather than a rushed cascade. Declared as a defect so it stays visible instead of blending in. POSITIVE CONTROL, because a guard nobody has watched fail is the exact shape of defect I found earlier today: added `struct PositiveControlProbe { pub owner_id: String }`, confirmed the guard failed naming `identity/mod.rs: owner_id`, removed it, confirmed green. The guard also caught 12 sites my own inventory grep had missed — it is already strictly better than the method I was auditing with. Full lib suite: 7081 passed, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…now --from-source
`start` shelled unconditionally into tools/scripts/start-server.sh, which runs a
full cargo build. A "governed" lifecycle verb was a wrapper around a bash file
that only exists inside a repo checkout (BigMama's find, 2026-08-13):
- a user holding ONLY the installed binary, with no source tree, could not
start a core at all;
- the CLI printed one line then went silent for the length of a compile —
called hung three separate times;
- the façade's honesty depended entirely on the script underneath.
Same class as the rest of the night's defects: a governed surface over a
hand-rolled path.
Now: `launch_core` locates the installed `continuum-core-server`
(CONTINUUM_CORE_SERVER override → next to the running exe → ~/.continuum/bin →
target/{release,debug} walking up) and execs it directly, keeping the existing
detach/log/pidfile handling unchanged. Building is an EXPLICIT request
(`continuum start --from-source`), never the silent default, and the
no-binary fallback says WHY it fell back and that it compiles first, rather
than going quiet for minutes.
The override refuses loudly when set but not a file — a wrong override must not
look like an absent one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…uard debt cleared) The working tree could not build tests. This finishes the identity-typing work that broke it and clears the guard debt it left behind. `cargo test -p continuum-core --lib --features metal,accelerate`: 7081 passed, 0 failed. PEER_ID IS A TYPE, NOT TEXT. The production conversions landed earlier; the test fixtures were left holding `&str`, so the crate compiled and the test target did not. Fixtures now derive a real UUID from the literal they used to carry (v5 under NAMESPACE_OID), which preserves every cross-site equality the tests depended on — "peer-a" in two places is still the same peer, it is just an id now instead of a name in costume. CAUGHT BY DOING IT: contracts/verification had the manifest keyed by a derived PeerId while the EVENT still claimed a raw string signer. The lookup is BY that signer, so converting one side made every verification test fail as MissingPeerManifest. One `test_peer_id` / `test_peer_str` pair now feeds both sides — the same shape of defect the newtype exists to prevent, reproduced in the fixtures while removing it from production. REAL HARDENING, not just fixture churn: `AircPeerManifest::validate` had DROPPED its empty-peer_id check on the theory that typing the field made it impossible. Typing killed `""`. It did NOT kill `Uuid::nil()`, which is still constructible and still means nobody — the type narrowed the hole rather than closing it. The guard is back, at the remaining expressible form. GUARD DEBT CLEARED: loose_id_guard's `no_declaration_outlives_its_defect` went red, correctly — four `peer_id` entries in LOOSE_IDS described fields that are now `PeerId`. Removed. The guard failing here is it working: a declaration list that can rot into a graveyard is worth nothing. Also includes the accumulated session tree: the Joel→Operator fixture sweep (no person's name hardcoded in test data), ts-rs regeneration, and rustfmt across the crate. That is why this touches ~717 files; the behavioural change is the identity typing and the nil-PeerId guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
`k3-serving` had no lifetime because it was named for a SUBSYSTEM, so it never died — a month-old durable subscription whose board reads were all corpses. Parted from both scopes. It also sat in TEST FIXTURES, which is the transmission vector: fixtures teach the next reader the convention, so a fixture naming a room after a subsystem teaches that rooms are named for subsystems. Renamed to `bench-swe-run-1` — an activity with a lifetime, which is what a room IS. The two remaining mentions in work.rs are the INCIDENT RECORD (4 of 12 cards there carried a 134h-expired lease) and are kept, now annotated RETIRED so nobody copies the naming from them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…zero — carry it on the wire and the board `SweGradeResult.error` documents its own contract: "a result with `error` is an ABSENCE, not a zero, and must never be tallied as a failed attempt." The grader earns that honestly — for the env class it re-runs the PRISTINE tree before declaring a fault, so a genuinely broken patch is never mislabelled. Then both consumers dropped the classification: - `benchmark.attempt.end` / `benchmark.autograde` published `resolved=false gate_ok=false` and nothing else. Every wire consumer (probe router → rooms, exam-room widgets, pulse monitors) reads that as a citizen who tried and lost. - `fold_run_card` read the RESULT's `infra_error` but never the GRADE's `error`, so the board folded `resolved: false` + phase `failed` for the same runs. The attempt loop already broke correctly on `g.error.is_some()` (attempts 2 and 3 were never burned) — the loss was purely in what got PUBLISHED, which is the part anything downstream can actually read. Measured on this box 2026-08-13: 8 of 36 distinct instances (14 of 91 receipts, 22%) grade UNGRADEABLE — requests, pylint, pytest and sympy. Every one of those zeros was indistinguishable from a capability failure on the wire, so the denominator of any rate computed off this stream was poisoned. Found by digging into sympy__sympy-11400: p2p 0/29 on the PRISTINE tree, i.e. the suite does not run in that environment at all. (#380/#383 own fixing the environments; this commit owns never again reporting their faults as scores.) - attempt.end + autograde now carry `ungradeable` + `grade_error` - `infra_error` takes the grade's error too — one field meaning "no valid verdict, and why", fed by both sources rather than a second parallel field - `resolved` returns to `None` when ungradeable — the same "no verdict" a pre-grade card carries, because that is the truth - new phase `ungradeable`, ordered ahead of `failed` (a run can carry both a failed marker and an ungradeable grade; the absence is the truer of the two) Test asserts absence-not-zero on the real sympy-11400 shape, with a positive control (same shape, no grade error) that must still fold as a capability zero — so the test cannot pass by simply never reporting failure. This is the #384/#386 class one layer up: those classified INFRA at the solve level, this classifies it at the GRADE level and gets it onto the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…UN, instead of caching a tree that grades every attempt UNGRADEABLE Root cause of the 22% ungradeable rate, glass-boxed on sympy__sympy-11400. The era pin at the top of this block downgrades pytest to the instance's own date, which is right when the era INTERPRETER rung (#2253) found a matching interpreter. When it can't — no Python 3.5 on a modern macOS — the venv falls back to a modern interpreter while pytest stays pinned to the instance's year, and the resulting PAIR can be structurally unable to run. Measured: pytest 2.9.2 (correct for 2016) on Python 3.9.6 dies in `pytest_configure` with INTERNALERROR before collecting anything — reproduced on a two-line trivial test, outside the repo, no conftest, no sympy involved. Every test then "fails", the pristine p2p reads 0/29, the tree grades UNGRADEABLE. 8 of 36 distinct instances on this box are in that state. So: prove the harness executes before handing the env to a citizen. A version pin is a GUESS about compatibility; running it is the evidence. `--version` is not enough — it answers happily for a pytest that dies on any real run. This REFUSES rather than self-heals, and that is measured, not assumed. The obvious repair (reinstall a modern pytest) was tried against this exact tree and does NOT work: pytest 8.4.2 → loads, dies in sympy 1.0's 2016 conftest on the `py.path` hook API removed in pytest 7 pytest 6.2.5 → dies on `py.test.mark.slow`, removed in pytest 4 pytest 2.9.2 → cannot run on Python 3.9 at all The band that both RUNS on 3.9 and LOADS a 2016 conftest is EMPTY. No version choice rescues this class, so an auto-repair would silently trade one void tree for another. What DOES work, verified on this tree — 30/30 passing — is sympy's OWN runner (`sympy.test(...)`) on the same interpreter. That is #383's shape ("django needs its OWN test runner") generalised: the runner is a property of the repo era, not a pytest version to search for. `run_tests` is pytest-only today, so until it grows a runner seam this env genuinely cannot produce a verdict — and it now says so loudly, naming the incompatibility and pointing at the runner gap, instead of caching a broken env for every later run to inherit. [[brittleness-is-the-highest-priority-work-there-is]] — heal what is known-safe, REPORT what needs a human decision. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… parallel runner (Joel's ruling, + STOP gate) The consequence that makes it law: the learning flywheel consumes ROOM TURNS (L1 lifts tool-traces from captured turns, L2 triggers on turn-completion). A detached agent/solve writing progress/<run>.grade.json produces NO turns — so a citizen can burn 12 acts, write a patch, take a graded verdict, and none of it reaches the curriculum. Maximum effort, zero learning. That, not the pass rate, is why benchmarks have failed. Names what is parallel today (ledger files, scraped probes, a second board projection in fold_run_card, private grade.json), the target shape (import task+oracle only → project into a recipe → the ROOM is the runner → grading is the activity outcome → learning falls out because the work happened as turns), and a one-line acceptance test: can a citizen standing in the room perceive the run's state through the same ViewState pipe the human's screen uses? Adds a CLAUDE.md STOP gate over benchmark.rs / agent/solve.rs / swe_bench.rs so an agent arriving under amnesia must read it before touching run state. Written because that is exactly what happened: this session shipped two correct fixes that HARDEN the parallel path instead of dissolving it, including adding a field to a benchmark probe so external consumers could parse it better — which is the smell the doc now names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
SweGradeResult.errordocuments its own contract in its doc comment:The grader earns that honestly — for the env class it re-runs the pristine tree before declaring a fault, so a genuinely broken patch is never mislabelled. Then both consumers dropped the classification:
benchmark.attempt.end/benchmark.autogradepublishedresolved=false gate_ok=falseand nothing else. Every wire consumer (probe router → rooms, exam-room widgets, pulse monitors) reads that as a citizen who tried and lost.fold_run_cardread the RESULT'sinfra_errorbut never the GRADE'serror, so the board foldedresolved: false+ phasefailedfor the same runs.The attempt loop already broke correctly on
g.error.is_some()— attempts 2 and 3 were never burned. The loss was purely in what got published, which is the only part anything downstream can read.Measured
On this box, 2026-08-13: 8 of 36 distinct instances (14 of 91 receipts, 22%) grade UNGRADEABLE — requests, pylint, pytest, sympy. Every one of those zeros was indistinguishable from a capability failure on the wire, so the denominator of any rate computed off this stream was poisoned.
Found by digging into
sympy__sympy-11400:p2p 0/29on the pristine tree — the suite does not run in that environment at all.#380 / #383 own fixing the environments. This PR owns never again reporting their faults as scores.
Change
attempt.end+autogradecarryungradeable+grade_errorinfra_errortakes the grade's error too — one field meaning "no valid verdict, and why", fed by both sources rather than a second parallel fieldresolvedreturns toNonewhen ungradeable — the same "no verdict" a pre-grade card carries, because that is the truthungradeable, ordered ahead offailed(a run can carry both a failed marker and an ungradeable grade; the absence is the truer of the two)Test
Asserts absence-not-zero on the real sympy-11400 shape, with a positive control (same shape, no grade error) that must still fold as a capability zero — so the test cannot pass by simply never reporting failure.
cargo test -p continuum-core --lib --features metal,accelerate run_projection→ 6/6 pass.Same class as #384/#386 one layer up: those classified INFRA at the solve level; this classifies at the grade level and gets it onto the wire.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo