Skip to content

Stop the examples spending money, and gate README flag drift - #94

Merged
kmadan merged 1 commit into
mainfrom
fix/examples-do-not-spend-money
Aug 31, 2026
Merged

Stop the examples spending money, and gate README flag drift#94
kmadan merged 1 commit into
mainfrom
fix/examples-do-not-spend-money

Conversation

@kmadan

@kmadan kmadan commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Both of these are things a first-time contributor hits before writing a line, found while checking that #51 is actually startable.

The examples bill you for running them

All three examples/*/run.py call OpenAI. DeepEval judges toxicity with an LLM and activates off the presence of OPENAI_API_KEY, so:

  • anyone with a key in their shell gets billed for running an example
  • anyone with an exhausted quota watches nine retries per interaction scroll past and reasonably concludes the example is broken

The bundled demo already guards against exactly this, popping the key unless --with-llm-metrics is passed. The examples never got the same treatment.

before:  ERROR:root:OpenAI Error: 429 ... Retrying: 9 time(s)...   (×many)
after:   0 OpenAI calls, same report, 54s

Opt in with AICERTIFY_WITH_LLM_METRICS=1, documented in each example README.

The guard must run before from aicertify import ..., because the evaluator stack reads the key at import time. That trips ruff's E402, so the import carries a noqa and a comment explaining that the ordering is load-bearing. Reordering to satisfy the linter would reintroduce the bug.

The README flag tables had drifted in both directions

Flag State
--evaluators removed in #91, still documented in all four translations
--params has always existed, documented in none of the five READMEs

Neither shows up in a test, a lint or a build. A reader following the Japanese README gets unrecognized arguments and concludes the tool is broken.

tests/test_readme_flags.py now asserts every flag any README documents actually exists, across all five languages, and that the English table covers every evaluate flag. It introspects the argparse tree rather than scraping --help, and has a guard-the-guard test so introspection breaking cannot make the others pass vacuously.

Mutation-tested before committing: removing --params from the English table fails; putting --evaluators back in the Japanese one fails. Restored, green.

The assertion is scoped to the evaluate subcommand rather than allow-listing exemptions. An allow-list of "documented elsewhere" flags is somewhere to quietly park anything inconvenient, and it grows until the check means nothing. It did immediately surface four more undocumented flags on demo and init-contract, which are described in prose.

The translations still lack a --params row. That needs a native speaker rather than a machine translation nobody can vouch for, so the test does not demand it and it stays a contribution task.

268 tests pass.

@github-actions github-actions Bot added 🐞 bug Something isn't working 📝 documentation Improvements or additions to documentation 🚀 enhancement New feature or request 🤝 good first issue Good for newcomers 🛡️ security Security vulnerabilities labels Aug 31, 2026
… drift

Two things a first-time contributor hits before writing a line.

The three example run.py scripts call OpenAI. DeepEval judges toxicity with an
LLM and activates off the presence of OPENAI_API_KEY, so anyone with a key in
their shell gets billed for running an example, and anyone with an exhausted
quota watches nine retries per interaction scroll past and concludes the
example is broken. The bundled demo already guards against this by popping the
key unless --with-llm-metrics is passed; the examples never got the same
treatment. They do now, opting in through AICERTIFY_WITH_LLM_METRICS=1, which
each example README documents.

customer-support-bot goes from a wall of 429s to zero OpenAI calls and still
produces its report.

The guard has to run before `from aicertify import ...`, because the evaluator
stack reads the key at import time. That puts the import below module-level
code, which ruff flags as E402; the ordering is load-bearing, so it carries a
noqa and a comment saying why rather than being reordered into correctness bug.

Separately, the README flag tables had drifted in both directions:

  --evaluators   removed in #91, still documented in all four translations
  --params       has always existed, documented in none of the five READMEs

Neither shows up in a test, a lint or a build. A reader following the Japanese
README gets `unrecognized arguments` and reasonably concludes the tool is
broken. tests/test_readme_flags.py now asserts every documented flag exists,
across all five languages, and that the English table covers every `evaluate`
flag. Mutation-tested in both directions before committing.

The translations are still missing a --params row. That needs a native speaker
rather than a machine translation nobody can vouch for, so the test does not
demand it and it stays a contribution task.
@kmadan
kmadan force-pushed the fix/examples-do-not-spend-money branch from d2cae28 to acb168f Compare August 31, 2026 06:59
@kmadan
kmadan merged commit a11b5fc into main Aug 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🐞 bug Something isn't working 📝 documentation Improvements or additions to documentation 🚀 enhancement New feature or request 🤝 good first issue Good for newcomers 🛡️ security Security vulnerabilities

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant