Stop the examples spending money, and gate README flag drift - #94
Merged
Conversation
… drift Two things a first-time contributor hits before writing a line. The three example run.py scripts call OpenAI. DeepEval judges toxicity with an LLM and activates off the presence of OPENAI_API_KEY, so anyone with a key in their shell gets billed for running an example, and anyone with an exhausted quota watches nine retries per interaction scroll past and concludes the example is broken. The bundled demo already guards against this by popping the key unless --with-llm-metrics is passed; the examples never got the same treatment. They do now, opting in through AICERTIFY_WITH_LLM_METRICS=1, which each example README documents. customer-support-bot goes from a wall of 429s to zero OpenAI calls and still produces its report. The guard has to run before `from aicertify import ...`, because the evaluator stack reads the key at import time. That puts the import below module-level code, which ruff flags as E402; the ordering is load-bearing, so it carries a noqa and a comment saying why rather than being reordered into correctness bug. Separately, the README flag tables had drifted in both directions: --evaluators removed in #91, still documented in all four translations --params has always existed, documented in none of the five READMEs Neither shows up in a test, a lint or a build. A reader following the Japanese README gets `unrecognized arguments` and reasonably concludes the tool is broken. tests/test_readme_flags.py now asserts every documented flag exists, across all five languages, and that the English table covers every `evaluate` flag. Mutation-tested in both directions before committing. The translations are still missing a --params row. That needs a native speaker rather than a machine translation nobody can vouch for, so the test does not demand it and it stays a contribution task.
kmadan
force-pushed
the
fix/examples-do-not-spend-money
branch
from
August 31, 2026 06:59
d2cae28 to
acb168f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both of these are things a first-time contributor hits before writing a line, found while checking that #51 is actually startable.
The examples bill you for running them
All three
examples/*/run.pycall OpenAI. DeepEval judges toxicity with an LLM and activates off the presence ofOPENAI_API_KEY, so:The bundled demo already guards against exactly this, popping the key unless
--with-llm-metricsis passed. The examples never got the same treatment.Opt in with
AICERTIFY_WITH_LLM_METRICS=1, documented in each example README.The guard must run before
from aicertify import ..., because the evaluator stack reads the key at import time. That trips ruff's E402, so the import carries a noqa and a comment explaining that the ordering is load-bearing. Reordering to satisfy the linter would reintroduce the bug.The README flag tables had drifted in both directions
--evaluators--paramsNeither shows up in a test, a lint or a build. A reader following the Japanese README gets
unrecognized argumentsand concludes the tool is broken.tests/test_readme_flags.pynow asserts every flag any README documents actually exists, across all five languages, and that the English table covers everyevaluateflag. It introspects the argparse tree rather than scraping--help, and has a guard-the-guard test so introspection breaking cannot make the others pass vacuously.Mutation-tested before committing: removing
--paramsfrom the English table fails; putting--evaluatorsback in the Japanese one fails. Restored, green.The assertion is scoped to the
evaluatesubcommand rather than allow-listing exemptions. An allow-list of "documented elsewhere" flags is somewhere to quietly park anything inconvenient, and it grows until the check means nothing. It did immediately surface four more undocumented flags ondemoandinit-contract, which are described in prose.The translations still lack a
--paramsrow. That needs a native speaker rather than a machine translation nobody can vouch for, so the test does not demand it and it stays a contribution task.268 tests pass.