Correct the weak-model skills delta to −2 - #59
Merged
Conversation
AGENTS.md carried -3 for eleven days. Both -3 readings are from 13 August and no run since has reproduced them: eight of the nine later runs read -2, one read 0. 13 August is the credit-outage day this same file documents under costs, where jobs were scored without the agent acting — and the figure was never re-derived after that cause was identified, which is exactly the mistake the "identify a bad population by its cause" trap was written about. Adds the recomputation inline so the next person derives it rather than quoting the paragraph, and a warning against reading Loop 2's Outpost-only +2 as a contradiction: five scenarios across three models is not fifteen scenarios on one, and the two were conflated once into a claim that the sign had flipped, which no run supports. Scenario counts corrected while here: twenty-two scenarios, nineteen benchmark, and nine of nineteen discriminate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AGENTS.md carried −3 for the weak model's skills delta since 13 August. It is −2.
Both −3 readings are from 13 August and nothing since has reproduced them:
13 August is the credit-outage day AGENTS.md documents under costs — thirty-seven Codex jobs failed and twenty-two earlier runs were scored with the agent never making a tool call. Whether these two snapshots contain affected rows needs the job start times against the outage window, which is the method that file prescribes and which was never applied to this figure. Flagged in the text rather than asserted.
The sign is replicated and unaffected. Only the magnitude was wrong.
What else changed
results/runs/instead of quoting the paragraph. That is how it went stale.Related, not in this PR
#2 has the full recomputation. The composition of the delta has changed completely:
transform-001-reshape-payloadis worse with skills in all ten runs and is the only durable one; the other three originals dissolved; andverification-002inverted to a skills win holding for eight consecutive runs with no skill change to explain it (the submodule pin has not moved since 12 August).#24 updated with the same correction.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA