Skip to content

[no-ci] Register pytest.mark authorship markers, add matching AGENTS.md instructions#2245

Merged
rwgk merged 3 commits into
NVIDIA:mainfrom
rwgk:agent_human_authored_pytest_mark
Jun 25, 2026
Merged

[no-ci] Register pytest.mark authorship markers, add matching AGENTS.md instructions#2245
rwgk merged 3 commits into
NVIDIA:mainfrom
rwgk:agent_human_authored_pytest_mark

Conversation

@rwgk

@rwgk rwgk commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Summary

This adds repository-wide AGENTS.md guidance for marking the provenance of newly added unit tests using pytest markers, and registers those markers in the pytest config roots.

The marker system is intentionally minimal. The design goal is to keep agents and humans choosing from no more than three clear states:

  • @pytest.mark.agent_authored(model="<model>") for tests authored by an agent and not yet materially reviewed or rewritten by a human.
  • @pytest.mark.human_reviewed for agent-authored tests that a human has materially reviewed or rewritten.
  • @pytest.mark.human_authored for tests authored by a human, or rewritten enough that the authorship is primarily human.

The guidance keeps markers local to the tests they describe: markers should be placed immediately above each test function, with class-level markers only when every test method in the class has the same provenance. It also says not to use module-level pytestmark for authorship provenance, because that is easy to miss in large files and makes later per-test provenance changes ambiguous.

The pytest marker registrations include sync comments so future edits keep the copies aligned across pytest config roots.

Relationship to #2232

This is an alternative to #2232. PR #2232 uses lightweight comment tags; this version uses formal pytest markers so tests can be selected by provenance with pytest -m, for example pytest -m agent_authored.

Purpose

The purpose of these markers is to preserve ground-truth test provenance at the moment when it is easiest and most reliable to capture.

Provenance can be useful when interpreting a test. If a test was carefully authored or materially reviewed by a human, we may reasonably give it more evidentiary weight when investigating ambiguous behavior, evaluating an API change, or deciding whether an assertion reflects deliberate intent. This is not a guarantee that the test is correct, nor does an agent-authored marker imply that a test is inferior; it is additional context that would otherwise be lost.

Recording the generating model also makes the annotations actionable over time. As newer models become available, we could run automated passes over tests written by older models to review assumptions, strengthen assertions, identify missing cases, or remove low-value duplication. The provenance markers provide a practical way to select those cohorts and record when meaningful human review has occurred.

The accumulated metadata may also support useful empirical analysis. In the future, we could examine whether test provenance correlates with factors such as defect frequency, regressions detected, coverage, flakiness, or maintenance cost. We do not yet know which correlations will prove meaningful, but such analysis will only be practical if we capture the ground truth while it is readily available. Reconstructing provenance later from commit history or pull-request discussion would be incomplete and unreliable.

For now, the markers are informational. They do not automatically change CI behavior or assign a quality score. Their purpose is to preserve lightweight, machine-readable evidence that future reviewers, tools, and studies can use.

@rwgk rwgk added this to the cuda.bindings next milestone Jun 23, 2026
@rwgk rwgk self-assigned this Jun 23, 2026
@rwgk rwgk added documentation Improvements or additions to documentation P1 Medium priority - Should do cuda.bindings Everything related to the cuda.bindings module labels Jun 23, 2026
@github-actions github-actions Bot added cuda.core Everything related to the cuda.core module cuda.pathfinder Everything related to the cuda.pathfinder module labels Jun 23, 2026
@rwgk rwgk merged commit 5e9911a into NVIDIA:main Jun 25, 2026
29 checks passed
@github-actions

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

@rwgk rwgk deleted the agent_human_authored_pytest_mark branch June 26, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.bindings Everything related to the cuda.bindings module cuda.core Everything related to the cuda.core module cuda.pathfinder Everything related to the cuda.pathfinder module documentation Improvements or additions to documentation P1 Medium priority - Should do

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants