An extended four-act demo, and the noise it exposed - #92
Merged
Conversation
GOPAL's demo is one beat because GOPAL decides things: swap the model, the
build fails. AICertify gathers things, and the interesting thing about it is
how honest it is about the part it cannot gather. That needs four acts.
1. score-card bert-base-uncased a real card, scored, no setup
2. explain eu_ai_act 150 you declare, 14 an evaluator computes
3. init-contract + evaluate 29 policies, 0 green, because you have
asserted none of the 150
4. metric gap report 22 of 27 supplied, and the five that
are not, which is a contributor's list
Two artefacts from one recorder: a 48-second CSS-animated SVG for the README,
where JavaScript never runs, and the full 93-second cast for the website.
Commands are really run and their output captured verbatim. Only the timing is
synthesised, because the four take about three minutes between them and a cast
at real speed is unwatchable. The recorder refuses to write if bert stops
scoring 0.49, if explain stops reporting 150 and 14, if an empty contract stops
failing all 29 policies, or if the gap report stops saying 22 of 27.
Two fixes the recording forced out:
extraction.py logged an expected condition at error and dumped a pydantic
validation report per policy, so a successful evaluation looked like a crash.
The schema path only recognises report_output, which four of gopal's policies
define; every other policy fails it and the caller then reads the decision
rules instead. The #78 fallback made that correct and left the log level
behind.
The highlighter counted lines where it should have counted screen rows. Two
130-character warnings wrapped, and `BELOW THRESHOLD` was drawn across an
unrelated score three rows below itself. Verified now by replaying the cast
through a VT emulator and checking every highlighted row against what the
command actually printed.
Pins gopal 0936496, which is the version the numbers on screen come from.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GOPAL's demo is one beat because GOPAL decides things: swap the model, the build fails. AICertify gathers things, and the interesting thing about it is how honest it is about the part it cannot gather. That needs four acts.
score-card bert-base-uncasedexplain eu_ai_actinit-contract+evaluateAct 3 ending on nothing passing is the point rather than an embarrassment. It is act 2's argument arriving as a verdict.
Act 4 is the one I would defend hardest: the tool ships a command that enumerates its own gaps, and every
GAPline is a contribution target with a name.Two artefacts, one recorder
A 48-second CSS-animated SVG for the README, where JavaScript never runs, and the full 93-second cast for the website page.
Commands are really run and their output captured verbatim. Only the timing is synthesised: the four take about three minutes between them, mostly downloading model weights, and a cast at real speed is unwatchable. The recorder refuses to write if bert stops scoring 0.49, if
explainstops reporting 150 and 14, if an empty contract stops failing all 29, or if the gap report stops saying 22 of 27.Two fixes the recording forced out
extraction.pymade a successful run look like a crash. It logged an expected condition aterrorand dumped a pydantic validation report per policy. The schema path only recognisesreport_output, which four of gopal's policies define; every other policy fails it and the caller then reads the decision rules instead. The #78 fallback made that correct and left the log level behind.The highlighter counted lines where it should have counted screen rows. Two 130-character warnings wrapped, and
BELOW THRESHOLDwas drawn across an unrelated score three rows below itself. Caught by replaying the cast through a VT emulator and checking every highlighted row against what the command actually printed; that check is how I would verify any future re-record.Submodule
Pins gopal
0936496, which is the version the numbers on screen come from. That bump is also why the gap report says 22 of 27 rather than 21 of 26: gopal#95 declaredmetrics.toxicity.max_toxicityin a metadata block, andContentSafetyEvaluatoralready supplied it.216 tests pass.