Skip to content

An extended four-act demo, and the noise it exposed - #92

Merged
kmadan merged 1 commit into
mainfrom
feat/extended-demo
Aug 30, 2026
Merged

An extended four-act demo, and the noise it exposed#92
kmadan merged 1 commit into
mainfrom
feat/extended-demo

Conversation

@kmadan

@kmadan kmadan commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

GOPAL's demo is one beat because GOPAL decides things: swap the model, the build fails. AICertify gathers things, and the interesting thing about it is how honest it is about the part it cannot gather. That needs four acts.

Act Command What it shows
1 score-card bert-base-uncased a real card, 0.49, BELOW THRESHOLD, no setup
2 explain eu_ai_act 150 you declare, 14 an evaluator computes
3 init-contract + evaluate 29 policies, 0 green, because none of the 150 were asserted
4 metric gap report 22 of 27 supplied, and the five that are not

Act 3 ending on nothing passing is the point rather than an embarrassment. It is act 2's argument arriving as a verdict.

Act 4 is the one I would defend hardest: the tool ships a command that enumerates its own gaps, and every GAP line is a contribution target with a name.

Two artefacts, one recorder

A 48-second CSS-animated SVG for the README, where JavaScript never runs, and the full 93-second cast for the website page.

Commands are really run and their output captured verbatim. Only the timing is synthesised: the four take about three minutes between them, mostly downloading model weights, and a cast at real speed is unwatchable. The recorder refuses to write if bert stops scoring 0.49, if explain stops reporting 150 and 14, if an empty contract stops failing all 29, or if the gap report stops saying 22 of 27.

Two fixes the recording forced out

extraction.py made a successful run look like a crash. It logged an expected condition at error and dumped a pydantic validation report per policy. The schema path only recognises report_output, which four of gopal's policies define; every other policy fails it and the caller then reads the decision rules instead. The #78 fallback made that correct and left the log level behind.

The highlighter counted lines where it should have counted screen rows. Two 130-character warnings wrapped, and BELOW THRESHOLD was drawn across an unrelated score three rows below itself. Caught by replaying the cast through a VT emulator and checking every highlighted row against what the command actually printed; that check is how I would verify any future re-record.

Submodule

Pins gopal 0936496, which is the version the numbers on screen come from. That bump is also why the gap report says 22 of 27 rather than 21 of 26: gopal#95 declared metrics.toxicity.max_toxicity in a metadata block, and ContentSafetyEvaluator already supplied it.

216 tests pass.

GOPAL's demo is one beat because GOPAL decides things: swap the model, the
build fails. AICertify gathers things, and the interesting thing about it is
how honest it is about the part it cannot gather. That needs four acts.

  1. score-card bert-base-uncased    a real card, scored, no setup
  2. explain eu_ai_act               150 you declare, 14 an evaluator computes
  3. init-contract + evaluate        29 policies, 0 green, because you have
                                     asserted none of the 150
  4. metric gap report               22 of 27 supplied, and the five that
                                     are not, which is a contributor's list

Two artefacts from one recorder: a 48-second CSS-animated SVG for the README,
where JavaScript never runs, and the full 93-second cast for the website.

Commands are really run and their output captured verbatim. Only the timing is
synthesised, because the four take about three minutes between them and a cast
at real speed is unwatchable. The recorder refuses to write if bert stops
scoring 0.49, if explain stops reporting 150 and 14, if an empty contract stops
failing all 29 policies, or if the gap report stops saying 22 of 27.

Two fixes the recording forced out:

extraction.py logged an expected condition at error and dumped a pydantic
validation report per policy, so a successful evaluation looked like a crash.
The schema path only recognises report_output, which four of gopal's policies
define; every other policy fails it and the caller then reads the decision
rules instead. The #78 fallback made that correct and left the log level
behind.

The highlighter counted lines where it should have counted screen rows. Two
130-character warnings wrapped, and `BELOW THRESHOLD` was drawn across an
unrelated score three rows below itself. Verified now by replaying the cast
through a VT emulator and checking every highlighted row against what the
command actually printed.

Pins gopal 0936496, which is the version the numbers on screen come from.
@github-actions github-actions Bot added 📝 documentation Improvements or additions to documentation 🚀 enhancement New feature or request 🤝 good first issue Good for newcomers 🛡️ security Security vulnerabilities labels Aug 30, 2026
@kmadan
kmadan merged commit 117be17 into main Aug 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

📝 documentation Improvements or additions to documentation 🚀 enhancement New feature or request 🤝 good first issue Good for newcomers 🛡️ security Security vulnerabilities

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant