EVALS_PUBLISH is set to false as of 27 August. The matrix still runs and still uploads artifacts; only the write to results/ is skipped, so the website cannot move.
This issue exists so the reason is not stuck in one person's head. The flag was built after a cron nearly published twelve failures that were ours, and cancelling it by hand needed somebody watching at the right moment. A hold whose rationale is undocumented recreates that problem in slower motion.
Why it is on
#62 added "Do not ask clarifying questions. Complete the task with the information provided." to the base prompt. That is the one string every cell in every experiment shares, so every row measured before 27 August was measured under a different treatment.
Publishing now would produce a snapshot that is part one treatment and part the other, with no way for a reader to tell which cells are which. Compounded by #60, where --merge carries forward rows a run did not re-execute — as of the 25 August snapshot, 22 rows dating from 13 August.
When to clear it
When a snapshot exists that was measured end to end under the current base prompt. Concretely: after the milestone-2 run, or after any full matrix where export-results --merge reports no carried rows with harness changed or no provenance recorded.
That report is new in #65 and is the check to use — it prints what the merge is carrying, by execution date, and which rows were measured under something else.
gh variable delete EVALS_PUBLISH # resume publishing
gh variable list # confirm
What happens meanwhile
Scheduled runs still execute and still upload transcripts, which is deliberate: transcripts are where every product finding has come from, and holding publication is not the same as skipping the week. The Monday 31 August weekly and the 1 September monthly matrix will both run, held. They are the first passes under the new prompt and are worth reading — a free check that nothing broke before the milestone-2 run is paid for.
Blocks the next release. Nothing should be tagged from a mixed-treatment snapshot.
EVALS_PUBLISHis set tofalseas of 27 August. The matrix still runs and still uploads artifacts; only the write toresults/is skipped, so the website cannot move.This issue exists so the reason is not stuck in one person's head. The flag was built after a cron nearly published twelve failures that were ours, and cancelling it by hand needed somebody watching at the right moment. A hold whose rationale is undocumented recreates that problem in slower motion.
Why it is on
#62 added
"Do not ask clarifying questions. Complete the task with the information provided."to the base prompt. That is the one string every cell in every experiment shares, so every row measured before 27 August was measured under a different treatment.Publishing now would produce a snapshot that is part one treatment and part the other, with no way for a reader to tell which cells are which. Compounded by #60, where
--mergecarries forward rows a run did not re-execute — as of the 25 August snapshot, 22 rows dating from 13 August.When to clear it
When a snapshot exists that was measured end to end under the current base prompt. Concretely: after the milestone-2 run, or after any full matrix where
export-results --mergereports no carried rows withharness changedorno provenance recorded.That report is new in #65 and is the check to use — it prints what the merge is carrying, by execution date, and which rows were measured under something else.
What happens meanwhile
Scheduled runs still execute and still upload transcripts, which is deliberate: transcripts are where every product finding has come from, and holding publication is not the same as skipping the week. The Monday 31 August weekly and the 1 September monthly matrix will both run, held. They are the first passes under the new prompt and are worth reading — a free check that nothing broke before the milestone-2 run is paid for.
Blocks the next release. Nothing should be tagged from a mixed-treatment snapshot.