Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ body:
id: expected
attributes:
label: What the documentation says should happen
description: Quote the sentence, with a link. Every sentence in the README has a row in `docs/docs/CLAIMS.md`; if you found the row, name it.
description: Quote the sentence, with a link. Every sentence in the README has a row in `https://ctrlrun.dev/docs/CLAIMS`; if you found the row, name it.
validations:
required: true
- type: textarea
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ body:
- type: markdown
attributes:
value: |
Every addition answers one question first: *does this improve control or correctness of an agent action that can affect the real world?* If it helps a model think, improves prompts, stores memory, retrieves documents, chooses models or visualises traces, it belongs somewhere else. `docs/docs/ROADMAP.md` lists what is planned and `VISION.md` what is not a build spec.
Every addition answers one question first: *does this improve control or correctness of an agent action that can affect the real world?* If it helps a model think, improves prompts, stores memory, retrieves documents, chooses models or visualises traces, it belongs somewhere else. `https://ctrlrun.dev/docs/ROADMAP` lists what is planned and `VISION.md` what is not a build spec.
- type: dropdown
id: guarantee
attributes:
Expand Down
4 changes: 2 additions & 2 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
- [ ] **Specification first.** The section this implements is named above; a change to a frozen name or a new entry point amends the spec in this same PR.
- [ ] **Tests first.** The acceptance tests were red before the implementation and are green after it; every polling or waiting test bounds its clock or iteration count.
- [ ] **Mutation table.** Each MUST in the touched sections was removed, its named test confirmed red, and the guard restored. The table is in the description, checked against the four shapes in CONTRIBUTING.md.
- [ ] **`docs/docs/CLAIMS.md`.** Every new or changed README sentence has a row with code and a test; every removed capability took its row and its sentence with it.
- [ ] **Docs audit green.** `python tools/docs_audit/snippets.py`, `lint.py`, `links.py` and `render_capabilities.py --check` pass; a capability table was edited in `docs/capabilities.yaml`, never by hand.
- [ ] **`CLAIMS.md`.** Every new or changed README sentence has a row with code and a test; every removed capability took its row and its sentence with it. The table lives in [CTRLRun/ctrlrun-docs](https://github.com/CTRLRun/ctrlrun-docs), so a README change here is a pull request there too.
- [ ] **Docs audit green.** In a `ctrlrun-docs` checkout with `CTRLRUN_SOURCE` pointing at this one: `python tools/docs_audit/snippets.py`, `lint.py`, `links.py` and `render_capabilities.py --check` pass. A capability table was edited in that repository's `capabilities.yaml`, never by hand. CI here runs the same checks against this commit, so a stale page is red on this pull request and not on somebody else's.
- [ ] **`scripts/check.sh` green** under the project's interpreter.
- [ ] **Independent review** requested for anything touching authorization, identity, delegation, the gateway, an adapter or the store.
- [ ] **Nothing in `src/` merges on green CI alone**; a maintainer reads it.
Expand Down
163 changes: 99 additions & 64 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,20 +78,6 @@ jobs:
CTRLRUN_REQUIRE_RELEASE_FIXTURES: "1"
run: ./scripts/check.sh

# The badge is written **after** `check.sh`, and the order is the whole claim: a count
# produced before the run would be the size of a suite that might be red. It is uploaded
# from one matrix leg only — two would race for the same artifact name — and published by
# the `badge` job, which runs on a push to `main` and on nothing else.
- name: The test-count badge
if: matrix.python-version == '3.11'
run: python tools/docs_audit/render_badges.py --write-count tests-badge.json

- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: matrix.python-version == '3.11'
with:
name: ctrlrun-tests-badge
path: tests-badge.json

# SPEC-v0.4 §5, T118. The composite action, run against this repository's own
# configurations, with `install: .` so it dogfoods the checkout rather than the last
# release. Two runs, because the second is the one that matters: the N/A path has to be
Expand Down Expand Up @@ -157,7 +143,10 @@ jobs:
# rather than regenerated. §5.1's rule holds across the job boundary: the badge, the job
# summary and the uploaded report all come from one verify run.
badge:
needs: [verify, check]
# `docs` as well as the two it always needed: the test-count artifact is produced there
# now, and a `badge` job that did not wait for it would download nothing and publish a
# stale count -- or fail on a race, which is the same bug on a good day.
needs: [verify, check, docs]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
Expand Down Expand Up @@ -258,22 +247,35 @@ jobs:
assert skipped == 0, f"{skipped} adapter tests skipped; the frameworks did not install"
PY

# The documentation audit (docs/STYLE.md). One job of its own so a docs PR gets a docs answer,
# and every step in it is now a hard failure:
# The documentation audit, and the test-count badge that only a green suite may produce.
#
# - nine generators, each with `--check`, so no rendered table, reference page, recipe
# directory, readiness block, badge row or study table can be hand-edited;
# - runnable snippets, the forbidden-words lint and the link check.
# The pages moved to `CTRLRun/ctrlrun-docs`, and the checks moved with them -- but what they
# check is *this* commit, so the job runs them there against a checkout of here. One
# implementation of every rule, two triggers: this job on a change to the library, and that
# repository's own CI on a change to a page.
#
# The last three carried `continue-on-error: true` from session 0 to session 6. They were
# wired in against the documentation as it stood, and the findings they raised that day were
# the baseline the writing sessions had to clear; a job that was red on arrival would have
# taught everyone to ignore it. The baseline is clear, so the flags are gone -- a red step
# that is allowed to be red is documentation, not a guard.
# Every step is a hard failure. Three of them carried `continue-on-error: true` from session
# 0 to session 6, while a baseline nobody had cleared yet would have made the job red on
# arrival and taught everyone to ignore it. The baseline is clear, so the flags are gone: a
# red step that is allowed to be red is documentation, not a guard.
docs:
# **The badge below is the reason for this dependency, and the dependency is the claim.**
# `--write-count` publishes the size of the suite, and a count produced before the suite
# ran would be the size of a run that might be red. It used to be a later *step* in the
# same job as `check.sh`; the audit lives in another repository now, so the ordering is a
# job dependency instead -- which is the stronger form, because a step's ordering is a
# convention and `needs:` is enforced.
needs: check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
path: ctrlrun

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: CTRLRun/ctrlrun-docs
path: ctrlrun-docs

- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
Expand All @@ -285,47 +287,80 @@ jobs:
# sample nobody ran.
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,gateway,otel,identity]"

- name: Rendered capability tables match docs/capabilities.yaml
run: python tools/docs_audit/render_capabilities.py --check

- name: The CLI reference matches click's help text
run: python tools/docs_audit/render_cli.py --check

- name: The receipt, event and error references match the code
run: python tools/docs_audit/render_schemas.py --check

- name: The Python API reference matches the docstrings
run: python tools/docs_audit/render_api.py --check

- name: The cookbook directories are what their pages show
run: python tools/docs_audit/render_cookbook.py --check

- name: The framework-probe page is the render of the published results
run: python tools/docs_audit/render_probe.py --check

- name: The soak page is the render of the published results
run: python tools/docs_audit/render_soak.py --check

- name: The readiness block is the same in all three places
run: python tools/docs_audit/render_readiness.py --check

- name: The badge row matches the generator
run: python tools/docs_audit/render_badges.py --check

# The three audits landed `continue-on-error` in session 0, because a baseline nobody had
# cleared yet would have made the job red on arrival and taught everyone to ignore it.
# The baseline is clear -- 96 runnable blocks, 182 documents, 0 findings and 0 broken
# links -- so the flags come off here. An audit that cannot fail is an audit nobody runs.
- name: Runnable snippets execute offline
run: python tools/docs_audit/snippets.py
pip install -e "./ctrlrun[dev,gateway,otel,identity]"
pip install griffe pyyaml

# `_core.py` raises when it cannot find the library, so this prints a path or the job
# stops here. A documentation check that ran against no source would be green and
# would have verified nothing.
- name: The audit can see this checkout
working-directory: ctrlrun-docs
env:
CTRLRUN_SOURCE: ${{ github.workspace }}/ctrlrun
run: python -c "import sys; sys.path.insert(0, 'tools/docs_audit'); from _core import CORE_ROOT; print(CORE_ROOT)"

# Nine generators, each with `--check`, so no rendered table, reference page, recipe
# directory, readiness block, badge row or study table can be hand-edited; then the
# runnable snippets, the forbidden-words lint and the link check.
- name: The audit
working-directory: ctrlrun-docs
env:
CTRLRUN_SOURCE: ${{ github.workspace }}/ctrlrun
run: |
set -eu
python tools/docs_audit/render_capabilities.py --check
python tools/docs_audit/render_cli.py --check
python tools/docs_audit/render_schemas.py --check
python tools/docs_audit/render_api.py --check
python tools/docs_audit/render_cookbook.py --check
python tools/docs_audit/render_probe.py --check
python tools/docs_audit/render_soak.py --check
python tools/docs_audit/render_readiness.py --check
python tools/docs_audit/render_badges.py --check
python tools/docs_audit/snippets.py
python tools/docs_audit/lint.py
python tools/docs_audit/links.py

# Uploaded from here and published by the `badge` job, which runs on a push to `main`
# and on nothing else.
- name: The test-count badge
working-directory: ctrlrun-docs
env:
CTRLRUN_SOURCE: ${{ github.workspace }}/ctrlrun
run: python tools/docs_audit/render_badges.py --write-count "$GITHUB_WORKSPACE/tests-badge.json"

- name: Forbidden words
run: python tools/docs_audit/lint.py
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ctrlrun-tests-badge
path: tests-badge.json

- name: Internal links and anchors resolve
run: python tools/docs_audit/links.py
# A change here can make a page there wrong without anything there changing, so a push to
# `main` tells that repository to re-run its checks against this commit.
#
# **It needs `DOCS_DISPATCH_TOKEN`**, a token with `contents: write` on CTRLRun/ctrlrun-docs;
# `GITHUB_TOKEN` cannot reach another repository. Without the secret this step warns and does
# nothing, and that is a gap rather than a design -- the backstop is the weekly `schedule:`
# run in `ctrlrun-docs/.github/workflows/ci.yml`, which turns a dispatch that stopped
# arriving into a red run within a week instead of into silence.
notify-docs:
needs: [check, docs]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- name: Tell ctrlrun-docs the library changed
env:
TOKEN: ${{ secrets.DOCS_DISPATCH_TOKEN }}
run: |
set -eu
if [ -z "${TOKEN:-}" ]; then
echo "::warning::DOCS_DISPATCH_TOKEN is not set; ctrlrun-docs re-checks on its weekly schedule instead"
exit 0
fi
curl -sS -f -X POST \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer $TOKEN" \
https://api.github.com/repos/CTRLRun/ctrlrun-docs/dispatches \
-d "{\"event_type\":\"library-changed\",\"client_payload\":{\"sha\":\"$GITHUB_SHA\"}}"

# The definition of done says the demo runs with no network and the README quick start
# works verbatim. Both were only ever checked by hand, which is how a README stops being
Expand Down
19 changes: 13 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Not every contribution carries every rule below. In rough order of what they ask
and cannot take for itself; so is telling us where the policy template for your sector is
wrong. Neither needs the suite installed.
- **Documentation** — a cookbook recipe, a `ctrlrun verify` snippet for a CI that is not GitHub
Actions — is held to `docs/STYLE.md` and the audit below, and not to the mutation table.
Actions — is held to `CTRLRun/ctrlrun-docs`'s `STYLE.md` and the audit below, and not to the mutation table.
- **Anything under `src/`** is where the rest of this file applies in full, and a maintainer
reads it whatever CI says.

Expand Down Expand Up @@ -103,8 +103,8 @@ mutants (say so in the table instead of claiming to have closed one).

## Every claim maps to a test

`docs/docs/CLAIMS.md` maps every sentence in the README to the code that implements it and the
test that proves it. A sentence with no row is cut. A row whose test disappears takes its
`CLAIMS.md`, in [CTRLRun/ctrlrun-docs](https://github.com/CTRLRun/ctrlrun-docs), maps every sentence in
the README to the code that implements it and the test that proves it. A sentence with no row is cut. A row whose test disappears takes its
sentence with it in the same commit. A test resolves every `file.py:NNN` in the table against
the line it cites and fails if the named symbol is not on it.

Expand All @@ -124,17 +124,24 @@ downgraded from prevention to attribution is renamed everywhere it was called a

## How documentation pull requests are checked

The words are held to the same standard as the code, by `tools/docs_audit/`:
The documentation is [CTRLRun/ctrlrun-docs](https://github.com/CTRLRun/ctrlrun-docs), and the words
there are held to the same standard as the code here, by that repository's `tools/docs_audit/`.
The checks read **both** trees -- a page that says the CLI prints X is only true if the CLI
prints X -- so they run in two places: on a pull request there, and from the `docs` job of
this repository's CI against the commit you are proposing. A change to the code that makes a
page wrong is red here.

What they check:

- every fenced block marked `runnable` is executed offline, with a socket guard, and must
exit 0; a sample either runs or is not marked;
- a forbidden-words lint refuses compliance and standards claims, sector products, and social
proof that does not exist, with an allowlist that carries a reason per entry;
- internal links and anchors must resolve;
- the capability tables in the README and the docs are rendered from
`docs/capabilities.yaml`, and a hand edit to a rendered copy fails CI.
`CTRLRun/ctrlrun-docs`'s `capabilities.yaml`, and a hand edit to a rendered copy fails CI.

`docs/STYLE.md` has the writing rules. Run the four checks from its last section before
`CTRLRun/ctrlrun-docs`'s `STYLE.md` has the writing rules. Run the four checks from its last section before
opening the pull request.

## Pull requests
Expand Down
24 changes: 10 additions & 14 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -11,22 +11,18 @@ include ctrlrun.example.yaml
include CHANGELOG.md
include SECURITY.md
include VISION.md
# `docs/` is the six specifications and the two implementation ones: the implementer's
# contract, and the thing `# SPEC:` comments in `src/` cite. They ship.
#
# What used to be here as well -- the documentation site, the tools that render it, its
# capability source and its rendered fragments -- is `CTRLRun/ctrlrun-docs` now, and the
# lines that carried it are gone rather than repointed. An sdist cannot include a file from
# another repository, and `test_the_sdist_carries_everything_the_tests_read` is what would
# have caught a line left behind naming a path that no longer exists.
recursive-include docs *.md
# The documentation audit reads these: `docs/capabilities.yaml` is the source the capability
# tables are rendered from and `docs/generated/` holds the renders, and
# `tests/test_docs_audit.py` compares the two. `tools/docs_audit/` is the generator and the
# three checks, imported by that test, so an sdist that ran its tests without them would be
# the same failure as shipping tests without their data.
recursive-include docs *.yaml *.mdx *.txt *.json *.svg *.png *.js *.mjs *.jsx *.css *.cjs
include docs/.mintignore
recursive-include tools *.py *.txt
# The README's header assets, and the sources they are rendered from.
# The README's header assets, and the sources they are rendered from. These stayed: the README
# is this repository's, and so is what it renders on PyPI.
recursive-include docs/assets *.svg *.png *.gif *.tape *.sh *.txt
# The website's media. Mintlify deploys from `docs/`, so a page's video has to live under it,
# and `test_the_sdist_carries_everything_the_tests_read` then requires it to ship. That test
# is checked against `git ls-files`, so the only way out would be to untrack the file, which
# would take it off the site. The sdist carries the megabyte.
recursive-include docs/images *.mp4 *.jpg
recursive-include tests *.py
# SPEC-v0.4 §4.3 — T115 validates `--junit` against a checked-in schema, and reads the README
# beside it for the schema's provenance and licence. A test that ships without its data is a
Expand Down
Loading
Loading