[no-ci] Register pytest.mark authorship markers, add matching AGENTS.md instructions#2245
Merged
Merged
Conversation
This was referenced Jun 23, 2026
mdboom
approved these changes
Jun 25, 2026
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds repository-wide
AGENTS.mdguidance for marking the provenance of newly added unit tests using pytest markers, and registers those markers in the pytest config roots.The marker system is intentionally minimal. The design goal is to keep agents and humans choosing from no more than three clear states:
@pytest.mark.agent_authored(model="<model>")for tests authored by an agent and not yet materially reviewed or rewritten by a human.@pytest.mark.human_reviewedfor agent-authored tests that a human has materially reviewed or rewritten.@pytest.mark.human_authoredfor tests authored by a human, or rewritten enough that the authorship is primarily human.The guidance keeps markers local to the tests they describe: markers should be placed immediately above each test function, with class-level markers only when every test method in the class has the same provenance. It also says not to use module-level
pytestmarkfor authorship provenance, because that is easy to miss in large files and makes later per-test provenance changes ambiguous.The pytest marker registrations include sync comments so future edits keep the copies aligned across pytest config roots.
Relationship to #2232
This is an alternative to #2232. PR #2232 uses lightweight comment tags; this version uses formal pytest markers so tests can be selected by provenance with
pytest -m, for examplepytest -m agent_authored.Purpose
The purpose of these markers is to preserve ground-truth test provenance at the moment when it is easiest and most reliable to capture.
Provenance can be useful when interpreting a test. If a test was carefully authored or materially reviewed by a human, we may reasonably give it more evidentiary weight when investigating ambiguous behavior, evaluating an API change, or deciding whether an assertion reflects deliberate intent. This is not a guarantee that the test is correct, nor does an agent-authored marker imply that a test is inferior; it is additional context that would otherwise be lost.
Recording the generating model also makes the annotations actionable over time. As newer models become available, we could run automated passes over tests written by older models to review assumptions, strengthen assertions, identify missing cases, or remove low-value duplication. The provenance markers provide a practical way to select those cohorts and record when meaningful human review has occurred.
The accumulated metadata may also support useful empirical analysis. In the future, we could examine whether test provenance correlates with factors such as defect frequency, regressions detected, coverage, flakiness, or maintenance cost. We do not yet know which correlations will prove meaningful, but such analysis will only be practical if we capture the ground truth while it is readily available. Reconstructing provenance later from commit history or pull-request discussion would be incomplete and unreliable.
For now, the markers are informational. They do not automatically change CI behavior or assign a quality score. Their purpose is to preserve lightweight, machine-readable evidence that future reviewers, tools, and studies can use.