Repository navigation
FEAT: observation lifecycle for surface scoring (scorer phase 11) - #3156
Merged
Richard Lundeen (richlundeen) merged 3 commits intoOct 14, 2026
Merged
Richard Lundeen (richlundeen) merged 3 commits into
Richard Lundeen (richlundeen) merged 3 commits into
Conversation
A snapshot shows that content is present; it cannot show that this attempt wrote it. Lifecycle support is optional and lets a source record a baseline when an attempt starts and finalize a later snapshot against it, so attempts can share a workspace without losing write attribution. - SurfaceBaseline records what the named locations held, with the same bounded evidence and coverage a snapshot keeps. A finalized SurfaceObservationPayload carries its baseline, so replay gives the same verdict. - SurfaceLifecycleSource and AttemptObserver are runtime protocols. LocalFileSurfaceSource implements start_observing_async and finalize_async on the same bounded, confined read as acquire_async. - An attack asks its objective and auxiliary scorers that observe attempts to start once the attempt ID is allocated, before anything is sent, and to stop when the attempt ends, whatever the outcome. - FileWriteScorer records a baseline per attempt and routed condition, and scoring finds the attempt from the conversation's attack result. A location counts only when the attempt changed it and added the content: unchanged is a proven negative, the same bytes written again cannot be credited, an append is credited only for a match that reaches into the new text, and a location the baseline might have missed stays undetermined. Without lifecycle support the scorer keeps the snapshot.
- Observation starts after setup, inside the strategy's execution, so a failure to start records the attempt's ERROR result under its allocated ID. - Every observer is stopped even when one stop fails. A failed stop is logged and no longer replaces the attempt's error, cancellation or saved result. - The attack walks wrapped scorers (composite, inverter, fallback and other wrappers), so a FileWriteScorer inside one records its baseline with the expectation its wrapper routes to it. - A baseline cannot tell overlapping attempts apart. The docs keep the one-at-a-time requirement for a shared root, and FileWriteScorer leaves a change undetermined when another attempt observed the same location at the same time. SurfaceBaseline.concurrent carries this, so replay agrees. - Without a text matcher, a changed nonempty location is credited from its digest, so binary files and files beyond max_content_bytes are no longer undetermined. Each has a regression test.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1f6debee-86b3-43ff-a35f-40af5a3a2dd6
Richard Lundeen (richlundeen)
enabled auto-merge
October 14, 2026 06:12
Richard Lundeen (richlundeen)
approved these changes
Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Phase 11 of the scorer design plan, following the merged phase 10 (#3042). The contract is baseline-aware content-state comparison, not write-event detection or writer attribution.
ContentWritten.require_content_change=Falsepreserves presence scoring, including with lifecycle-capable sources. Setting it toTruerequires final content to differ from its baseline and satisfy the matcher. A required comparison without a baseline is undetermined, not snapshot success.SurfaceBaselineis retained inSurfaceObservationPayloadfor offline replay.SurfaceLifecycleSourceandAttemptObserverprotocols capture evidence before execution. Scorer-tree traversal preserves each child's routed expectation.The plan and paired notebook now state this contract explicitly. The public scorer and condition names remain compatible with phase 10; the condition selects the predicate rather than the source's capabilities selecting it implicitly.
Tests and Documentation
Added attack, persistence, and replay coverage for unrelated edits, appends preserving a match, identical rewrites with changed or preserved timestamps, transient writes followed by restoration, missing baselines, and explicit presence versus change criteria. Existing wrapper, overlap, startup/cleanup, binary-content, and acquisition-bound tests remain covered.
Validation for
1af5bc34e:uv run --no-sync pre-commit run --all-files— all hooks passed after notebook metadata normalization, including type checking.uv run --no-sync pytest tests\unit\score\test_file_write_scorer.py tests\unit\score\test_local_file_surface_source.py tests\unit\models\test_surface.py tests\unit\models\test_scorable.py tests\unit\models\test_condition.py -q --tb=short— 259 passed, 3 skipped (Windows symbolic-link privileges unavailable).uv run --no-sync jupytext --to ipynb --execute --set-kernel python3 doc\code\scoring\6_file_write_scorer.py— passed. Outputs retained; notebook and Python cell content match.