Skip to content

Comments that describe the present: sweep of files changed since August - #960

Merged
cailmdaley merged 51 commits into
developfrom
docs/comment-sweep
Oct 26, 2026
Merged

cailmdaley merged 51 commits into
developfrom
docs/comment-sweep

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

We rewrite comments, docstrings and repository prose to describe current behavior and the reasons for it, without changing executable code or configuration values.
Measured hazards and ordering constraints remain beside their implementation; repeated explanations point to one authoritative location.
Load-bearing local constraints use named @sc contracts.

Scope and compatibility

The sweep covers module documentation, workflow rules and helpers, profiles, configuration comments and user documentation.
It preserves the current segmentation inputs, defect diagnostics, epoch-cut persistence, singleton vignetmaker paths, cleanup eligibility and packed uint16 exposure-map implementation.
The exposure-map scripts retain their current format documentation rather than the retired uint8 description.
Scientific decisions and committed defaults are unchanged.

Comment-only doesn't mean resume-neutral.
Workflow helper source hashes can change when comments change, so deploy at a campaign boundary rather than assuming completed work will remain up to date.

Stripped-diff check

Against develop 89324f2c, the checker below reports 65 files changed, 0 with code changes.
Python ASTs match after removing declaration docstrings; Snakefile/rule token streams match after excluding comments and statement-level strings.
YAML matches both as parsed data and comment-stripped lines; other hash-comment formats match after stripping comments outside quotes.
Markdown, reStructuredText and contract prose are listed separately.
This proves executable equality under those comparisons, not the truth of every explanatory sentence.

— Codex on behalf of Cail.

Comments-only checker
#!/usr/bin/env python3
"""Prove a diff touches only comments, docstrings and prose.

Usage: comments_only.py <repo> <base-ref> [<head-ref>]   (head defaults to the working tree)

For every file changed between base and head:
  * .py                          -> ast.dump with docstrings removed must be equal
  * Snakefile, .smk, bin/sp      -> Python token stream without COMMENT/NL tokens and
                                    without statement-level string literals (docstrings)
  * .yaml/.yml                   -> yaml.safe_load equal AND comment-stripped lines equal
  * .ini .param .sex .psfex .conv .setools .sh .bash .cfg .toml Dockerfile and
    other '#'-comment formats     -> lines with '#' comments stripped (outside quotes) equal
  * .md .rst README* CONTRACTS *.txt -> prose, always allowed (listed)
Exits 1 if any non-prose file differs in code.
"""
import ast
import io
import subprocess
import sys
import tokenize
from pathlib import Path

PROSE_SUFFIX = {".md", ".rst", ".txt"}
PROSE_NAMES = {"CONTRACTS", "README", "AGENTS.md", "CLAUDE.md"}
SMK_LIKE = {"Snakefile"}


def git(repo, *args):
    return subprocess.run(["git", "-C", repo, *args], check=True,
                          capture_output=True, text=True).stdout


def read(repo, ref, path):
    if ref is None:
        p = Path(repo) / path
        return p.read_text() if p.exists() else None
    try:
        return git(repo, "show", f"{ref}:{path}")
    except subprocess.CalledProcessError:
        return None


def strip_docstrings(tree):
    for node in ast.walk(tree):
        if isinstance(node, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            body = node.body
            if body and isinstance(body[0], ast.Expr) and isinstance(getattr(body[0], "value", None), ast.Constant) \
                    and isinstance(body[0].value.value, str):
                node.body = body[1:] or [ast.Pass()]
    return tree


def norm_py(text):
    return ast.dump(strip_docstrings(ast.parse(text)), include_attributes=False)


def norm_tokens(text):
    toks = list(tokenize.generate_tokens(io.StringIO(text).readline))
    out, prev = [], tokenize.NEWLINE
    for i, t in enumerate(toks):
        if t.type in (tokenize.COMMENT, tokenize.NL):
            continue
        if t.type == tokenize.STRING and prev in (tokenize.NEWLINE, tokenize.INDENT, tokenize.DEDENT):
            # statement-level string (docstring): drop it if the statement is only strings
            j = i
            while j < len(toks) and toks[j].type in (tokenize.STRING, tokenize.NL, tokenize.COMMENT):
                j += 1
            if j < len(toks) and toks[j].type in (tokenize.NEWLINE, tokenize.ENDMARKER, tokenize.DEDENT):
                continue
        if t.type == tokenize.STRING and out and out[-1][0] == tokenize.STRING and prev == tokenize.STRING and out[-1][1] is None:
            pass
        out.append((t.type, t.string))
        prev = t.type
    # collapse docstring-removal leftovers: NEWLINE runs
    clean = []
    for tok in out:
        if tok[0] == tokenize.NEWLINE and clean and clean[-1][0] in (tokenize.NEWLINE, tokenize.INDENT):
            continue
        clean.append(tok)
    return clean


def strip_hash(line):
    q = None
    for i, c in enumerate(line):
        if q:
            if c == q:
                q = None
        elif c in "'\"":
            q = c
        elif c == "#" and (i == 0 or line[i - 1].isspace()):
            return line[:i].rstrip()
    return line.rstrip()


def norm_hash(text):
    return [l for l in (strip_hash(x) for x in text.splitlines()) if l.strip()]


def kind(path):
    p = Path(path)
    if p.suffix in PROSE_SUFFIX or p.name in PROSE_NAMES or p.name.startswith("README"):
        return "prose"
    if p.suffix == ".py":
        return "py"
    if p.suffix == ".smk" or p.name in SMK_LIKE:
        return "smk"
    if p.suffix in (".yaml", ".yml"):
        return "yaml"
    return "hash"


def main():
    repo, base = sys.argv[1], sys.argv[2]
    head = sys.argv[3] if len(sys.argv) > 3 else None
    rng = [base, head] if head else [base]
    files = [f for f in git(repo, "diff", "--name-only", *rng).split() if f]
    bad = 0
    for f in files:
        k = kind(f)
        a, b = read(repo, base, f), read(repo, head, f)
        if a is None or b is None:
            print(f"FAIL  {f}: added or deleted"); bad += 1; continue
        if k == "prose":
            print(f"prose {f}"); continue
        try:
            if k == "py":
                same = norm_py(a) == norm_py(b)
            elif k == "smk":
                try:
                    same = norm_py(a) == norm_py(b)
                except SyntaxError:
                    same = norm_tokens(a) == norm_tokens(b)
            elif k == "yaml":
                import yaml
                same = yaml.safe_load(a) == yaml.safe_load(b) and norm_hash(a) == norm_hash(b)
            else:
                same = norm_hash(a) == norm_hash(b)
        except Exception as e:  # noqa: BLE001
            print(f"FAIL  {f}: {type(e).__name__}: {e}"); bad += 1; continue
        print(f"{'ok   ' if same else 'FAIL '} {f} [{k}]")
        bad += not same
    print(f"\n{len(files)} files changed, {bad} with code changes")
    sys.exit(1 if bad else 0)


if __name__ == "__main__":
    main()

cailmdaley and others added 30 commits October 10, 2026 18:42
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
… documentation

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…ry overview

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
cailmdaley and others added 20 commits October 10, 2026 18:47
…ration prose

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…config

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Correct five false or unclear statements, move five in-function @sc
contracts into their docstrings (the record checker accepts # tags only
at module level), and demote completeness-report-unit-key to prose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JhKts6NHr9YjfnatBZPogK
@cailmdaley
cailmdaley merged commit 7c55601 into develop Oct 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant