diff --git a/README.hi-IN.md b/README.hi-IN.md index 14a86e7..7bb15cc 100644 --- a/README.hi-IN.md +++ b/README.hi-IN.md @@ -21,10 +21,10 @@ PyPI CI Stars - Python 3.12+ + Python 3.12 Apache 2.0 Built on OPA - 85 Rego Policies + 92 Rego Policies PRs Welcome

diff --git a/README.ja-JP.md b/README.ja-JP.md index 20d4c6b..6eb15c0 100644 --- a/README.ja-JP.md +++ b/README.ja-JP.md @@ -21,10 +21,10 @@ PyPI CI Stars - Python 3.12+ + Python 3.12 Apache 2.0 Built on OPA - 85 Rego Policies + 92 Rego Policies PRs Welcome

diff --git a/README.ko-KR.md b/README.ko-KR.md index ed71aeb..1811bd7 100644 --- a/README.ko-KR.md +++ b/README.ko-KR.md @@ -21,10 +21,10 @@ PyPI CI Stars - Python 3.12+ + Python 3.12 Apache 2.0 Built on OPA - 85 Rego Policies + 92 Rego Policies PRs Welcome

diff --git a/README.zh-CN.md b/README.zh-CN.md index a5a24c3..20cfc7e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -21,10 +21,10 @@ PyPI 持续集成 Star 数 - Python 3.12+ + Python 3.12 Apache 2.0 许可证 基于 OPA 构建 - 85 条 Rego 策略 + 92 条 Rego 策略 欢迎提交 PR

diff --git a/aicertify/opa_policies b/aicertify/opa_policies index 0482ec9..4321e7a 160000 --- a/aicertify/opa_policies +++ b/aicertify/opa_policies @@ -1 +1 @@ -Subproject commit 0482ec9b4bb2a237f2a75e8e12353a11d0b901b7 +Subproject commit 4321e7a4fc28061610d345319d6c778e877bbd85 diff --git a/diagrams/diagram1_hero_flow_dark.svg b/diagrams/diagram1_hero_flow_dark.svg index 810e600..335fb3f 100644 --- a/diagrams/diagram1_hero_flow_dark.svg +++ b/diagrams/diagram1_hero_flow_dark.svg @@ -1,7 +1,7 @@ How AICertify works -Four-step flow from AI application to audit-ready report. +Four-step flow from AI application to evidence report. diff --git a/diagrams/diagram1_hero_flow_light.svg b/diagrams/diagram1_hero_flow_light.svg index 607fda6..7946706 100644 --- a/diagrams/diagram1_hero_flow_light.svg +++ b/diagrams/diagram1_hero_flow_light.svg @@ -1,7 +1,7 @@ How AICertify works -Four-step flow from AI application to audit-ready report. +Four-step flow from AI application to evidence report. diff --git a/diagrams/diagram2_architecture_dark.svg b/diagrams/diagram2_architecture_dark.svg index 898356f..64c3c36 100644 --- a/diagrams/diagram2_architecture_dark.svg +++ b/diagrams/diagram2_architecture_dark.svg @@ -76,7 +76,7 @@ OPA policy engine -91 production Rego policies · gopal library +92 production Rego policies · gopal library diff --git a/diagrams/diagram2_architecture_light.svg b/diagrams/diagram2_architecture_light.svg index 8334721..b8687c9 100644 --- a/diagrams/diagram2_architecture_light.svg +++ b/diagrams/diagram2_architecture_light.svg @@ -76,7 +76,7 @@ OPA policy engine -91 production Rego policies · gopal library +92 production Rego policies · gopal library diff --git a/diagrams/diagram3_regulatory_coverage_dark.svg b/diagrams/diagram3_regulatory_coverage_dark.svg index c78fe51..939a680 100644 --- a/diagrams/diagram3_regulatory_coverage_dark.svg +++ b/diagrams/diagram3_regulatory_coverage_dark.svg @@ -1,9 +1,9 @@ Regulatory coverage -91 production Rego policies across 9 frameworks and 6 industries. +92 production Rego policies across 9 frameworks and 6 industries. -91 +92 Rego policies 9 frameworks diff --git a/diagrams/diagram3_regulatory_coverage_light.svg b/diagrams/diagram3_regulatory_coverage_light.svg index f1d5180..092b217 100644 --- a/diagrams/diagram3_regulatory_coverage_light.svg +++ b/diagrams/diagram3_regulatory_coverage_light.svg @@ -1,9 +1,9 @@ Regulatory coverage -91 production Rego policies across 9 frameworks and 6 industries. +92 production Rego policies across 9 frameworks and 6 industries. -91 +92 Rego policies 9 frameworks diff --git a/diagrams/diagram5_report_anatomy_dark.svg b/diagrams/diagram5_report_anatomy_dark.svg index e743f83..c8ff722 100644 --- a/diagrams/diagram5_report_anatomy_dark.svg +++ b/diagrams/diagram5_report_anatomy_dark.svg @@ -1,6 +1,6 @@ -Anatomy of an audit-ready report +Anatomy of an AICertify evidence report Stylized AICertify report showing header, executive summary, policy results, risk assessment, and remediation sections. diff --git a/diagrams/diagram5_report_anatomy_light.svg b/diagrams/diagram5_report_anatomy_light.svg index 3e64444..4d6274a 100644 --- a/diagrams/diagram5_report_anatomy_light.svg +++ b/diagrams/diagram5_report_anatomy_light.svg @@ -1,6 +1,6 @@ -Anatomy of an audit-ready report +Anatomy of an AICertify evidence report Stylized AICertify report showing header, executive summary, policy results, risk assessment, and remediation sections. diff --git a/docs/INDEX.md b/docs/INDEX.md index c67a4df..81a7512 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -1,23 +1,25 @@ # AICertify Documentation -> **Looking for an overview?** Start with the [README](../README.md) — it covers value prop, quickstart, regulatory coverage, and a comparison vs alternatives. +> **Looking for an overview?** Start with the [README](../README.md) — it covers the evaluation model, quickstart, regulatory coverage, evidence boundaries, and where AICertify fits. The documentation here is organized along [Diátaxis](https://diataxis.fr/) lines: tutorials get you running, how-tos solve specific problems, reference describes the API, explanation explores design. -## 🎓 Tutorials — get from zero to a working compliance report +## Tutorials -- [Quickstart](../examples/quickstart.py) — clone, install, run, inspect the report. ~5 minutes. +- [Quickstart](../examples/quickstart.py) — clone, install, run, and inspect the report. - [Sample contract](../examples/sample_contract.json) — the JSON shape a real contract takes. -- [Examples README](../examples/README.md) — index of all shipped examples and their pre-generated outputs. +- [Examples README](../examples/README.md) — index of the forkable application examples and their maintained expected reports. -## 🛠️ How-To Guides — solve a specific problem +## How-to guides - [Run a compliance check against a custom contract](../skills/evaluate-contract/SKILL.md) (Claude Code skill) - [Add coverage for a new regulation](../skills/draft-policy/SKILL.md) (Claude Code skill) - [Understand what a framework's policies enforce](../skills/explain-regulation/SKILL.md) (Claude Code skill) - [Generate a report in PDF / Markdown / JSON / HTML](../examples/quickstart.py) — see the `report_format` argument. +- [Map existing tool outputs into GOPAL metrics](adapters.md) — adapters for Detoxify, model cards, Fairlearn, and Perspective. +- [Write a custom evaluator](writing-an-evaluator.md) — metric declaration, registration, delivery, and tests. -## 📚 Reference — look up specific names +## Reference - [Python API](../aicertify/__init__.py) — the public surface re-exported from the package root. - [CLI](../aicertify/cli.py) — `python -m aicertify.cli` flags and behavior. See also [README#cli](../README.md#cli). @@ -25,12 +27,12 @@ The documentation here is organized along [Diátaxis](https://diataxis.fr/) line - [pyproject.toml](../pyproject.toml) — metadata, dependencies, entry points. - [CHANGELOG](../CHANGELOG.md) — release history. -## 💡 Explanation — understand the design +## Explanation -- [AGENTS.md](../AGENTS.md) — the architectural narrative and how AI coding agents (and humans) should work in the repo. -- [Why policy-as-code?](../README.md#why-aicertify) — the differentiation argument. +- [AGENTS.md](../AGENTS.md) — architecture, repository conventions, and guidance for AI coding agents and human contributors. +- [Why AICertify](why-aicertify.md) — evidence model, reproducibility boundary, and the division of responsibility between AICertify, GOPAL, OPA, and evaluators. -## 🤝 Contributing & community +## Contributing and community - [CONTRIBUTING.md](../CONTRIBUTING.md) - [CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) diff --git a/docs/adapters.md b/docs/adapters.md index 3959fc4..9996207 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -1,12 +1,13 @@ # Plugging in tools you already run -You probably already measure some of this. Adapters take what an existing tool -produced and turn it into the metric names GOPAL policies read, so you do not -have to write an evaluator for something you have already measured. +Adapters convert outputs from existing evaluation tools into the canonical +metric names read by GOPAL policies. Use an adapter when the measurement already +exists and only the data shape needs to be translated. -Every adapter is a pure function of one argument. It imports nothing from the -tool it adapts, so you can convert a saved Detoxify result without installing -Detoxify, and none of them adds a dependency for anyone not using them. +Every adapter is a pure function of one argument. It does not import the tool +it adapts, so a saved Detoxify result can be converted without installing +Detoxify. Adapters therefore do not add dependencies for users who do not call +them. ```python from aicertify.adapters import from_detoxify, from_model_card @@ -36,25 +37,25 @@ fragment = from_detoxify(Detoxify("original").predict(outputs)) }}} ``` -Two numbers come out because GOPAL asks two different questions. -`metrics.toxicity.score` is the aggregate, compared against 0.1. -`metrics.toxicity.max_toxicity` is the worst single output, compared against -0.7. In the example above the average is 0.317 and the maximum is 0.92: report -only the average and the one genuinely toxic output has been averaged into -invisibility. +The adapter emits two toxicity statistics because GOPAL evaluates them +separately. `metrics.toxicity.score` is the aggregate, compared against 0.1, +while `metrics.toxicity.max_toxicity` is the maximum per-output score, compared +against 0.7. In the example above the aggregate is 0.317 and the maximum is +0.92. Retaining both prevents an aggregate from obscuring an individual +high-toxicity output. -`predict()` returns a list per class for a list of texts and a numpy scalar per -class for a single string. Both are accepted, and numpy floats are coerced, -because a `float32` reaching OPA fails there rather than here. +`predict()` returns a list per class for a list of texts and a NumPy scalar per +class for a single string. Both shapes are accepted. NumPy floating-point +values are converted to native Python values before the fragment reaches OPA. The five other Detoxify classes have no canonical GOPAL name. They are kept under `metrics.toxicity.categories` rather than dropped. ## Hugging Face model cards -A model card is the most widely published description of an AI system that -exists. It is not a compliance document, and this adapter is a good way to see -exactly how far short it falls. +Model cards are widely used to document AI models, but their structure and +scope differ from a compliance evidence record. This adapter maps model-card +content into the documentation metrics that GOPAL policies can evaluate. ```python from huggingface_hub import ModelCard @@ -63,7 +64,7 @@ from aicertify.adapters import from_model_card fragment = from_model_card(ModelCard.load("bert-base-uncased").content) ``` -Two things come out. +The adapter produces two groups of output. **Documentation sections.** GOPAL's `global/v1/documentation/model_card_score` scores nine sections from Mitchell @@ -77,10 +78,10 @@ genuinely establish. **Reported results.** `model-index` frontmatter is a structured record of benchmark results. These land under `metrics.reported.*`, deliberately not -under `metrics.accuracy.score`. A self-reported number on a benchmark of the -author's choosing is a claim about a dataset, not a measurement of your -deployed system, and promoting it would let a good SST-2 score answer a -question nobody asked about it. +under `metrics.accuracy.score`. A self-reported benchmark result describes the +reported dataset and evaluation configuration; it does not establish the +accuracy of the deployed system. Keeping it under `metrics.reported.*` +preserves that distinction. ### One command @@ -101,34 +102,33 @@ bert-base-uncased scored with aicertify 0.7.0, gopal 1.3.1, rubric v1 ``` -The version line is not decoration. A figure quoted anywhere has to be -reproducible, and without it a reader who reruns this later and gets a -different number cannot tell a changed rubric from a wrong claim. It names -GOPAL rather than a rubric version because the rubric *is* a GOPAL policy. +The version line records the implementations needed to reproduce the score. If +a later run produces a different result, the recorded AICertify and GOPAL +versions allow a changed rubric to be distinguished from a change in the input. +GOPAL is recorded because the rubric is implemented as a GOPAL policy. `--file` scores a local `README.md`, `--json` is machine-readable, and `--threshold` compares against something other than 0.8. ### Where the rubric lives -Not here. Which sections a card must carry, what each is worth and how much -text counts as content are normative judgements about required documentation, -which is what GOPAL is for, so they are a policy: +The rubric is defined in GOPAL rather than in the adapter. Requirements about +which sections are expected, how they are weighted, and what counts as content +are policy decisions, so they are implemented as a Rego policy: [`global/v1/documentation/model_card_score`](https://github.com/Principled-Evolution/gopal/blob/main/global/v1/documentation/model_card_score.rego). -`score_model_card` shells out to `opa` and reads the answer. The playground -runs the same policy compiled to WebAssembly. Neither reimplements it, so the -number you get here and the number the site shows are the same number because -they come from the same rules, not because two implementations happen to agree. +`score_model_card` invokes `opa` and reads the policy result. The playground +runs the same policy compiled to WebAssembly. Both surfaces therefore use the +same scoring rules rather than separate implementations of the rubric. -That also means `opa` is required, and a missing binary raises -`GopalUnavailable` rather than falling back to an approximation. An -approximation would produce something that looks like the real number and is -not. +`opa` is therefore required. If the binary is unavailable, +`score_model_card` raises `GopalUnavailable` rather than returning an +approximate score under the same metric name. -The heading table that maps card headings onto those sections comes from the -policy too, through `load_heading_sources()`. A copy per parser drifts exactly -the way a copy of the scoring did. +The heading table that maps card headings onto those sections is also loaded +from the policy through `load_heading_sources()`. Keeping the mapping and the +scoring rubric in the same policy prevents independent parser copies from +diverging. ### What real cards actually score @@ -143,10 +143,10 @@ applies: | `HuggingFaceTB/SmolLM2-135M-Instruct` | 0.17 | 0.31 | No | | `sentence-transformers/all-MiniLM-L6-v2` | 0.16 | 0.19 | No | -None of them passes, and the two best are among the most downloaded models in -the world. That is not a defect in the cards or in the adapter. A model card -answers part of what Annex IV asks and stops, and the gap is the point: see -[GOPAL's field-by-field accounting](https://github.com/Principled-Evolution/gopal/blob/main/docs/model-cards-vs-compliance.md). +None of these cards reaches the 0.8 threshold. The result reflects a scope +difference rather than a defect in the cards: a model card can supply part of +the documentation requested by Annex IV without covering the complete set of +fields. See [GOPAL's field-by-field accounting](https://github.com/Principled-Evolution/gopal/blob/main/docs/model-cards-vs-compliance.md). The subsections a card cannot fill are left unfilled rather than guessed at. Decision thresholds, intersectional results and the motivation behind an @@ -173,21 +173,22 @@ fragment = from_fairlearn(frame) }}} ``` -**The direction matters more than the number.** GOPAL compares -`metrics.fairness.score` with `>=`, so higher is better. Fairlearn's -`difference()` is 0 at its best and points the other way; handing it over -unchanged reports the fairest possible system as the least fair. That is how -`is_toxic` once answered `true` for a system with no toxicity in it. +### Metric direction -So `ratio()` is preferred, and not only to dodge the sign: a ratio is already -bounded on [0, 1] with 1 as the ideal, which is the shape a `>= 0.85` threshold -expects, and it is the form the four-fifths rule in fair lending is written in. +GOPAL compares `metrics.fairness.score` with `>=`, so higher values must +represent better outcomes. Fairlearn's `difference()` uses the opposite +direction, with 0 as its best value. Passing that value through unchanged would +invert the meaning of the policy threshold. + +`ratio()` is therefore preferred. It is bounded on [0, 1] with 1 as the ideal, +which matches the direction expected by a `>= 0.85` threshold and the ratio +form used by the four-fifths rule in fair lending. A `difference=` is accepted and converted to `1 - difference`, with `basis` recording which way the number was turned. -Both `MetricFrame` shapes work: a multi-metric frame returns a Series keyed by -metric name, a single-metric frame returns bare scalars, and the second is the -one that breaks a `[metric]` lookup written for the first. +Both `MetricFrame` result shapes are supported. A multi-metric frame returns a +Series keyed by metric name, while a single-metric frame returns bare scalars; +the adapter handles both explicitly. Against `healthcare/v1/diagnostic_safety`, which gates at 0.85: @@ -214,14 +215,14 @@ any toxicity measurement: the aggregate against 0.1 and the worst single output against 0.7. Perspective omits `summaryScore` for a language it does not support. That -attribute is left out rather than read as 0.0, which would report text nobody -could score as clean. +attribute is omitted rather than converted to 0.0, because an unavailable +measurement must not be represented as a clean result. -Unlike the others, this adapter was written against Perspective's documented -response schema rather than live calls, because the API needs a key this -project does not hold. The shape is stable and long-published, but that is a -weaker warrant than the rest of this page has and it is better said than -implied. +This adapter is implemented against Perspective's documented response schema +rather than live integration tests because the repository does not hold a +Perspective API key. The documented schema is stable, but this provides weaker +validation than testing against live API responses; that limitation is stated +explicitly here. ## Using a fragment @@ -272,19 +273,21 @@ def from_your_tool(report: dict) -> dict: }}} ``` -Three rules, each of which exists because breaking it caused a real bug here. +Three constraints apply to new adapters. -**Absent is not zero.** Nothing measurable in, empty dict out. Never a default -score. On a scale where lower is worse a zero reports an unmeasured system as -clean, and GOPAL will believe it. +**Do not convert absence to zero.** If no measurement is available, return an +empty fragment rather than a default score. A default numeric value would cause +policy evaluation to treat an unmeasured system as if a measurement had been +performed. -**The name has to describe what you measured.** Do not map a documentation -count onto `metrics.patient_safety.score` because it is the nearest available -slot. GOPAL gates that at 0.95 and means a clinical measurement. +**Use a metric name that describes the measurement.** Do not map a +documentation count onto `metrics.patient_safety.score` because it is the +nearest available field. GOPAL interprets that field as a clinical measurement +and applies a 0.95 threshold. **Keep statistics apart.** An average and a maximum answer different questions. If your tool gives you both, emit both. -If you find yourself needing an evaluator rather than an adapter, because there -is a measurement to run rather than a result to convert, that is -[docs/writing-an-evaluator.md](writing-an-evaluator.md). +If a measurement must be executed rather than an existing result converted, +implement an evaluator instead; see +[Writing an evaluator](writing-an-evaluator.md). diff --git a/docs/integrations/github-actions.md b/docs/integrations/github-actions.md index 8b5450b..9cd9685 100644 --- a/docs/integrations/github-actions.md +++ b/docs/integrations/github-actions.md @@ -1,48 +1,50 @@ # Running AICertify in GitHub Actions -AI compliance checks belong on every pull request, not only at release time. This page shows how to copy [`.github/workflows/example-aicertify.yaml`](../../.github/workflows/example-aicertify.yaml) into your own repository and turn it into a required status check. +AICertify can run as a pull-request status check so that the same contract, evaluator configuration, and policy set used locally can also be evaluated in CI. This page shows how to adapt [`.github/workflows/example-aicertify.yaml`](../../.github/workflows/example-aicertify.yaml) into a required check. ## What you copy | File | Where it goes | What it does | | --- | --- | --- | | [`.github/workflows/example-aicertify.yaml`](../../.github/workflows/example-aicertify.yaml) | `.github/workflows/ai-compliance.yaml` | Installs AICertify and OPA, evaluates a contract, uploads the report | -| [`examples/github-actions/gate.py`](../../examples/github-actions/gate.py) | anywhere the workflow can reach | Turns the verdict into an exit code | +| [`examples/github-actions/gate.py`](../../examples/github-actions/gate.py) | anywhere the workflow can reach | Converts policy results into an exit code | ## Inputs -Three values control the run. In the example they are `workflow_dispatch` inputs so you can try combinations from the Actions tab; in your own copy, hard-code them in `env:` and drop the inputs block. +Three values control the example workflow. They are exposed as `workflow_dispatch` inputs for interactive testing; in a repository-specific workflow they can be fixed in `env:` instead. | Input | Meaning | Default | | --- | --- | --- | -| `contract` | Path to the contract JSON describing your AI system | `examples/customer-support-bot/input_contract.json` | +| `contract` | Path to the contract JSON describing the AI system | `examples/customer-support-bot/input_contract.json` | | `framework` | Framework to evaluate against: `eu_ai_act`, `uk`, `bfs`, `legal`, `nist`, … | `eu_ai_act` | -| `fail_on` | `any` fails the build when a policy denies. `none` reports only | `none` | +| `fail_on` | `any` fails the job when a policy denies; `none` reports without gating | `none` | -Run `aicertify explain ` to see the frameworks available and the fields each one reads. +Run `aicertify explain ` to list the fields read by the selected framework before creating the contract. ## Adopt it in three steps -**1. Start on `fail_on: none`.** +**1. Begin with `fail_on: none`.** -Your first run will almost certainly report denials, and that is informative rather than alarming: most GOPAL obligations turn on facts no evaluator can observe, and a contract that has not declared them cannot satisfy them. Reporting first lets you see the gap without a red build. +A contract that omits required declarations or measured inputs will produce policy denials. Reporting without gating on the first run lets you inspect those gaps before making the check required. **2. Populate the contract.** ```bash -aicertify explain eu_ai_act # what the policies read -aicertify init-contract --policy eu_ai_act > contract.json # the same fields, to fill in +aicertify explain eu_ai_act +aicertify init-contract --policy eu_ai_act > contract.json ``` -`init-contract` writes every declared field as a `null` under `context`, nested into the shape the policies read. Replace the nulls. A field left as `null` is dropped rather than sent as an explicit null, so a half-filled contract denies rather than being read as "assessed, and false". +`init-contract` writes the declared fields under `context` as `null`, nested into the shape read by the policies. Replace the values that apply to the system. A field left as `null` is omitted from policy input, so missing evidence remains missing rather than being converted into an explicit `false` value. -**3. Switch to `fail_on: any` and make it required.** +**3. Enable gating.** -Once the report is clean, set `fail_on: any`, then **Settings → Branches → Branch protection rules → Require status checks to pass** and select the job. +After the contract and evaluator inputs represent the evidence you intend to enforce, set `fail_on: any`. Then configure the workflow job as a required status check under the repository's branch-protection settings. -## Two things that will bite you +## Two implementation details -**Check out the submodule.** The policy library lives at `aicertify/opa_policies` as a git submodule. Without `submodules: true` on `actions/checkout`, there are no policies at all, and the failure mode is quiet: the evaluation produces zero verdicts rather than an obvious error. +### Check out the policy submodule + +The GOPAL policy library is pinned at `aicertify/opa_policies` as a git submodule. Configure `actions/checkout` with `submodules: true`: ```yaml - uses: actions/checkout@v4 @@ -50,9 +52,11 @@ Once the report is clean, set `fail_on: any`, then **Settings → Branches → B submodules: true ``` -`gate.py` treats zero verdicts as exit code 2 rather than a pass, specifically so this misconfiguration cannot look like success. +Without the submodule, the policy directory is empty and an evaluation can produce zero policy verdicts. `gate.py` maps zero verdicts to exit code 2 so that missing policy input cannot be reported as a passing compliance gate. + +### Cache evaluator model downloads -**Cache the model downloads.** The fairness and content-safety evaluators pull transformer models on first use, and that dominates the run time. A cold run takes many minutes; a warm one is far quicker. +Fairness and content-safety evaluators may download transformer models on first use. Caching the Hugging Face and Torch directories avoids repeating those downloads on warm CI runs: ```yaml - uses: actions/cache@v4 @@ -65,21 +69,23 @@ Once the report is clean, set `fail_on: any`, then **Settings → Branches → B ## Exit codes -`gate.py` distinguishes three outcomes, and the distinction is the point: +`gate.py` distinguishes policy denial from evaluation failure: | Code | Meaning | | --- | --- | -| 0 | Every policy passed, or `--fail-on none` | -| 1 | At least one policy denied, and `--fail-on any` | -| 2 | The evaluation could not be carried out | +| 0 | Every policy passed, or `--fail-on none` is configured | +| 1 | At least one policy denied and `--fail-on any` is configured | +| 2 | The evaluation could not produce a valid set of policy verdicts | + +Exit code 2 covers conditions such as a missing contract, an OPA failure, or an evaluation that produced no verdicts. Those conditions do not establish a policy result and therefore must not be folded into the passing case. -Code 2 covers a missing contract, an OPA failure, and an evaluation that produced no verdicts. A run that failed to produce verdicts has told you nothing, and folding that into "not a failure" is how a compliance pipeline reports green while checking nothing. +## Outputs -## What you get out +The example workflow produces: -- **A job summary table** of every policy and its verdict, visible on the run page without opening the log. -- **Annotations** on each denied policy, so the reason appears inline on the pull request. -- **`compliance-summary.json`**, machine-readable, for anything downstream: +- **A job-summary table** containing each policy and its verdict. +- **Pull-request annotations** for denied policies, including the reported reason. +- **`compliance-summary.json`** for downstream automation: ```json { @@ -91,11 +97,11 @@ Code 2 covers a missing contract, an OPA failure, and an evaluation that produce } ``` -- **The generated report**, uploaded with `if: always()` so it survives a failing gate. That artifact is the evidence the check ran and what it decided. +- **The generated report**, uploaded with `if: always()` so the artifact remains available when the gate fails. ## Evaluating more than one framework -Call the gate once per framework with a matrix. `fail-fast: false` so one denial does not hide the rest: +Use a matrix when the same contract must be evaluated against multiple framework policy sets. `fail-fast: false` allows every framework to complete even when one produces a denial: ```yaml strategy: @@ -112,8 +118,8 @@ steps: --summary-json compliance-summary-${{ matrix.framework }}.json ``` -## Policies without the Python stack +## When GOPAL alone is sufficient -If you only want the Rego verdicts and none of the evaluator machinery, skip AICertify and run the GOPAL bundles directly. That job needs OPA and nothing else, and finishes in seconds. See [gopal's GitHub Actions example](https://github.com/Principled-Evolution/gopal/tree/main/examples/github-actions). +If the required facts and metrics already exist and the workflow only needs Rego policy verdicts, the GOPAL bundles can be evaluated directly with OPA. See [GOPAL's GitHub Actions example](https://github.com/Principled-Evolution/gopal/tree/main/examples/github-actions). -Use AICertify's workflow when you want fairness and content-safety metrics computed from real interactions, and the audit-ready report. Use the GOPAL bundle when your compliance facts are declarations and you just need them checked. +Use the AICertify workflow when CI also needs application-contract handling, evaluator-produced metrics, or generated evidence reports around the policy evaluation. diff --git a/docs/why-aicertify.md b/docs/why-aicertify.md index 2d1da2a..6229ad9 100644 --- a/docs/why-aicertify.md +++ b/docs/why-aicertify.md @@ -1,66 +1,79 @@ -# Why AICertify? +# Why AICertify -## The gap +AICertify exists to make the evidence used in AI-governance decisions inspectable, repeatable, and suitable for automation. -Most AI governance programs live in PDFs, spreadsheets, and policy documents. They describe what *should* happen — but do not prove what *did*. +## The problem it addresses -Auditors don't accept "we have a policy." They accept evidence: a dated record of the AI system under test, the rule it was evaluated against, the result, and the document signed off by the responsible owner. Producing that evidence by hand, every release, for every regulation, for every AI system in your portfolio, is not a sustainable program. +An AI system can change independently of the documents that describe its governance or compliance state. Model versions, prompts, retrieval indexes, application logic, and operating context can change between formal reviews. When the system state and the evidence record are maintained separately, a later report may describe a different system from the one currently being operated. -## The shift +AICertify does not solve that problem by assigning a universal compliance score. Within a pinned source checkout, it assembles the available system evidence, evaluates that evidence against the checkout's GOPAL revision, and produces structured verdicts and reports. Reproducing the evaluation later requires retaining or recording the relevant AICertify/GOPAL revision together with the policy inputs. -The DevOps and platform engineering communities solved a similar problem ten years ago by moving infrastructure from documents into code: Terraform replaced cloud-architecture diagrams, Helm replaced runbooks, [Open Policy Agent](https://www.openpolicyagent.org/) replaced security-policy memos. The pattern in every case was the same — *take the rule out of the document and put it into a thing that runs.* +## The evidence model -AICertify applies that shift to AI governance. +An AICertify evaluation can contain four distinct inputs and outputs: -## The artifact AICertify produces +1. **Declared facts** — information about the system or governance process that is supplied in the contract, such as whether human intervention is available or whether a required assessment has been completed. +2. **Captured interactions** — prompts, outputs, and related application data used as evaluation material. +3. **Measured metrics** — values computed or imported by evaluators and adapters, such as fairness, toxicity, content-safety, or model-card completeness metrics. +4. **Versioned policy decisions** — OPA evaluates the assembled input against Rego policies from the pinned [GOPAL](https://github.com/Principled-Evolution/gopal) revision, and AICertify exposes the resulting verdicts and reports. -Instead of saying: +These evidence types are intentionally not treated as equivalent. A measured metric can support a threshold check. A declared fact records an assertion and allows a rule to require it, but the rule does not independently verify that the asserted process or event occurred. -> "Our customer-support chatbot follows our responsible AI policy." +## Reproducibility and provenance -You produce: +The GOPAL policy library is pinned as a git submodule. An AICertify repository revision therefore identifies the policy revision used by that checkout. -> "Here is the contract that captured the chatbot's model version, the captured user-AI interactions, the EU AI Act v1 transparency policy (commit `a52d605`), the OPA evaluation result, the per-rule deny messages where applicable, and the dated PDF report sent to the audit committee." +Once the policy input values and Rego revision are fixed, OPA rule evaluation is deterministic. Reproducibility of the inputs themselves depends on how they were produced: deterministic metric adapters can be reproduced from the same source data, while evaluators that call external or generative models may have additional model-version, service, and sampling dependencies. -Every artifact is reproducible: same input, same policy, same result. Every claim is traceable: the policy is code in git, the evaluation is deterministic, the report is generated, not handwritten. +A reproducible evaluation record therefore needs the following provenance chain: -## Who is this for? +**system/version → contract and interactions → measured metrics + declared facts → policy revision → OPA verdicts → retained report** -AICertify exists for teams that need to **read, run, review, and repeat** their AI compliance evidence: +AICertify makes these stages explicit; the calling workflow is responsible for retaining the revisions and source evidence needed to reproduce them. -- **AI engineers** building under the EU AI Act, NIST AI RMF, India Digital Policy, Brazil AI Bill, FERPA/COPPA, FAA UAS rules, or any other named framework. -- **Governance, risk, and compliance (GRC) teams** who want their controls to *execute*, not just describe. -- **Auditors and model risk professionals** evaluating third-party AI systems. -- **Platform engineers** integrating AI compliance checks into CI/CD next to their linting, type-checking, and dependency scanning. -- **OPA / Rego users** who already trust policy-as-code for infrastructure and want the same discipline for AI. -- **Responsible AI researchers** who need reproducible bias, content-safety, and risk-management benchmarks. +## Where AICertify fits -## How AICertify is different +AICertify combines components that are useful independently: -| | AICertify | Vendor SaaS (Credo AI, Holistic AI) | Research toolkit (Fairlearn, AIF360, MS RAI Toolbox) | -|---|---|---|---| -| Open source | ✅ Apache 2.0 | ❌ Closed | ✅ MIT | -| Air-gapped / on-prem deployable | ✅ | ❌ | ✅ | -| Policy-as-code (versioned, diff-able, reviewable) | ✅ OPA / Rego | ❌ | ❌ | -| Named regulatory frameworks (EU AI Act, NIST RMF, +6 more) | ✅ via [gopal](https://github.com/Principled-Evolution/gopal) | ✅ | ❌ (fairness/explainability only) | -| Industry verticals out of the box (aviation, banking, healthcare, education, automotive) | ✅ | Partial | ❌ | -| Audit-ready report output (PDF / Markdown / JSON / HTML) | ✅ | ✅ | Partial | -| Custom policies | ✅ Drop a `.rego` file | ✅ (paid tier) | N/A | -| Reproducible from a git checkout | ✅ | ❌ | ✅ | +- **Evaluator libraries** produce measurements from application behavior or artifacts. +- **GOPAL** provides named AI-governance and regulatory policies as readable Rego. +- **OPA** evaluates those policies against a supplied input document. +- **AICertify** captures application context, runs or ingests evaluators, assembles policy input, invokes OPA, and generates retained outputs for local, CI, and assurance workflows. -## The honest scope +If your facts and metrics already exist and you only need Rego evaluation, GOPAL with OPA is the smaller tool. AICertify is useful when the evaluation pipeline also needs application context, measured metrics, policy discovery, reporting, or CI integration. -AICertify is **infrastructure**, not magic. +## Who it is for -- It does not interpret regulations for you. Encoding "EU AI Act Article 13 transparency" as a Rego policy is a deliberate, reviewable act, and the policy is a human's interpretation, not a legal opinion. Read [SECURITY.md](../SECURITY.md), the per-framework READMEs, and the disclaimer on every policy directory before claiming compliance. -- It does not certify your AI system. It produces the evidence a human or organisation needs in order to assert compliance, internally or to a regulator. The certification authority remains your auditor, your legal counsel, or the relevant supervisor. -- It does not replace your governance program. It replaces the *paperwork* in your governance program. +AICertify is designed for teams that need the evaluation record to be understandable by both engineering and governance functions: -What it *does* give you is the missing link between *"we have a responsible-AI policy"* and *"we can prove it."* +- **AI engineers** integrating policy checks with application development and release workflows. +- **Governance, risk, and compliance teams** maintaining evidence for named controls or regulatory requirements. +- **Auditors and model-risk teams** reviewing the inputs, policy version, and resulting decisions. +- **Platform engineers and OPA/Rego users** applying existing policy-as-code practices to AI-system evidence. +- **Responsible-AI researchers and evaluator authors** connecting measured behavior to explicit policy thresholds. + +## What AICertify does not establish + +AICertify is an evaluation and evidence framework, not a certification authority. + +- A Rego policy is an explicit implementation of a regulatory or governance interpretation; it is reviewable code, not legal advice. +- A passing verdict means the supplied evidence satisfies the encoded rule at the evidence depth that rule uses. It does not convert a declaration into independent verification. +- A generated report records an evaluation result. Whether that result is sufficient for a regulatory submission, assurance opinion, internal approval, or certification remains a decision for the responsible organization and its advisers. +- AICertify complements governance processes; it does not replace organizational controls that cannot be established from software inputs. + +This boundary is a design property rather than a limitation to hide. The policy, required inputs, evidence type, and resulting verdict remain inspectable. + +## Why policy-as-code matters here + +Readable Rego policies make the evaluation criteria reviewable in the same workflow as other code. A policy revision can be diffed, tested, pinned, cited, and run against the same input before it is adopted. That makes changes to the rule itself distinguishable from changes to the AI system or its measured behavior. + +For policy coverage and per-policy input requirements, see [GOPAL](https://github.com/Principled-Evolution/gopal). For AICertify's evaluator adapters, see [adapters.md](adapters.md). ## Next steps -- **See the output without installing:** open [demo-report-eu-ai-act.pdf](demo-report-eu-ai-act.pdf). -- **Run the quickstart:** [`examples/quickstart.py`](../examples/quickstart.py). -- **Explore the policy library:** [gopal](https://github.com/Principled-Evolution/gopal) — 85 production Rego policies across 8 frameworks and 5 industries. -- **Open a [good first issue](https://github.com/Principled-Evolution/aicertify/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).** +- **Inspect a generated artifact without installing:** [demo-report-eu-ai-act.pdf](demo-report-eu-ai-act.pdf) +- **Run the bundled demo:** `aicertify demo` +- **See what a framework requires:** `aicertify explain eu_ai_act` +- **Scaffold a contract:** `aicertify init-contract --policy eu_ai_act > contract.json` +- **Run the Python example:** [`examples/quickstart.py`](../examples/quickstart.py) +- **Integrate a pull-request gate:** [GitHub Actions guide](integrations/github-actions.md) diff --git a/docs/writing-an-evaluator.md b/docs/writing-an-evaluator.md index 49f8b5e..e52e92d 100644 --- a/docs/writing-an-evaluator.md +++ b/docs/writing-an-evaluator.md @@ -1,19 +1,18 @@ # Writing an evaluator -A gopal policy reads two kinds of input. **Declared facts**, which a person -asserts about their organisation, and **measured metrics**, which a tool -computes. You can answer the declared facts yourself. The measured ones you -cannot, and a policy that reads one you cannot supply can never be satisfied. +A GOPAL policy can read **declared facts** supplied about the system or its +operating context and **measured metrics** produced by an evaluation tool. An +evaluator supplies measured values in the canonical input fields expected by +the policy. -An evaluator is what supplies them. This is how to write one. +This guide describes how to implement and register an AICertify evaluator. -None of this is required to use GOPAL. A shell script that writes JSON and -calls `opa` is a complete integration, and GOPAL's +GOPAL does not require AICertify. A shell script can write the expected JSON +and invoke `opa` directly; GOPAL's [Plug your evaluator into GOPAL](https://github.com/Principled-Evolution/gopal/blob/main/docs/tutorials/supplying-metrics.md) -walks that path with no Python in it. What AICertify adds is the scaffolding: -the base class, the discovery, the gap report, and the merge that puts your -measurements where the policies actually read them. Read this one if you want -that; read the other if you would rather own the plumbing. +documents that path. AICertify adds evaluator base classes, discovery, metric-gap +reporting, registration, and delivery of measured values into the OPA input +shape. ## 1. Find out what is missing @@ -39,13 +38,14 @@ international/eu_ai_act (12/13 measured metrics have an evaluator) TOTAL: 22 of 26 measured metrics can be supplied today. ``` -Every `GAP` is a metric some policy needs and nothing produces. That is the -list. Add `--framework eu_ai_act` to narrow it, or `--json` to consume it. +Each `GAP` identifies a metric required by a policy for which no registered +evaluator currently declares coverage. Use `--framework eu_ai_act` to restrict +the report or `--json` for machine-readable output. ## 2. Write the evaluator -Three things: subclass `BaseEvaluator`, declare which metrics you supply, and -implement `evaluate`. +An evaluator requires three elements: subclass `BaseEvaluator`, declare the +metrics it supplies, and implement `evaluate`. ```python from aicertify.evaluators.base_evaluator import BaseEvaluator, EvaluationResult @@ -79,17 +79,16 @@ class AuditLoggingEvaluator(BaseEvaluator): return self.evaluate(data) ``` -That is the whole of -[`aicertify/evaluators/audit_logging_evaluator.py`](../aicertify/evaluators/audit_logging_evaluator.py), -lightly trimmed. It needs no model, no API key and no inference: it counts how -many audit-logging facts a contract actually carries, which is a real -measurement of a real thing. +The implementation in +[`aicertify/evaluators/audit_logging_evaluator.py`](../aicertify/evaluators/audit_logging_evaluator.py) +follows this pattern. It requires no model, API key, or inference; it computes +a completeness score from the audit-logging fields present in the contract. -**Use gopal's canonical name in `SUPPORTED_METRICS`.** The gap report matches -evaluators to policies by name. Invent a spelling and the policy still looks -unsupplied, however good your evaluator is. `helper_functions/metrics.rego` in -gopal is the list of canonical names, and it accepts the historical spellings -as fallbacks. +**Use GOPAL's canonical metric names in `SUPPORTED_METRICS`.** The gap report +matches evaluators to policy requirements by field name. A non-canonical name +therefore does not satisfy the policy requirement. GOPAL's +`helper_functions/metrics.rego` defines the canonical names and supported +historical aliases. ## 3. Wire it in @@ -102,14 +101,13 @@ EVALUATOR_CLASSES = { } ``` -This is the step that is easy to skip, because everything looks fine without -it. `SUPPORTED_METRICS` is what the gap report reads, so an unregistered -evaluator still shows up there, while `ComplianceEvaluator` only ever -instantiates what is in this dict. You get an evaluator that is discovered, -reported as coverage, and never run. +Registration is separate from metric declaration. The gap report reads +`SUPPORTED_METRICS`, while `ComplianceEvaluator` instantiates only evaluators +listed in `EVALUATOR_CLASSES`. An evaluator can therefore declare coverage but +remain unavailable at runtime if it is not registered. -The report now refuses to be fooled by that. Leave the registration out and -the row reads: +The gap report distinguishes this state with `WIRE`. Without registration, the +row reads: ``` WIRE governance.audit_logging.completeness_score AuditLoggingEvaluator declares @@ -123,23 +121,20 @@ global (4/4 measured metrics have an evaluator) ok governance.audit_logging.completeness_score AuditLoggingEvaluator ``` -`WIRE` does not count toward the total. A metric nothing produces at runtime is -not covered, whatever the class attributes say. +`WIRE` does not count toward runtime metric coverage because the evaluator is +not instantiated during evaluation. ## 4. Publish under the canonical name -Registration gets your evaluator run. It does not get its numbers to a policy. +Registration controls whether the evaluator runs; metric delivery is a separate +step. -GOPAL reads measured metrics at `input.metrics..`. Evaluator -output arrives keyed by evaluator name, as `results.`, and for a long -time the only canonical names that resolved were the three where those two -happened to coincide: `metrics.fairness.score`, `metrics.content_safety.score` -and `metrics.risk_management.score` each spell an evaluator name in the middle -with `score` on the end. `metrics.model_card.completeness` does not, so no -policy reading it ever saw a measurement. +GOPAL reads measured metrics at `input.metrics..`, while raw +evaluator results are keyed by evaluator name as `results.`. Those +names are not generally equivalent, so evaluator output must be published +explicitly under the canonical metric path read by GOPAL. -So publish explicitly, by putting the metric under `details["metrics"]` in the -shape GOPAL reads: +Publish the metric under `details["metrics"]` in the shape read by GOPAL: ```python return EvaluationResult( @@ -148,30 +143,27 @@ return EvaluationResult( ) ``` -`attach_measured_metrics` merges every evaluator's block into the top level of -the OPA input. What you emit there is what the policy sees, under the name you -gave it. `SUPPORTED_METRICS` is a claim about what you supply; this is the -supply. +`attach_measured_metrics` merges each evaluator's metric block into the OPA +input. `SUPPORTED_METRICS` declares which metrics the evaluator can provide; +`details["metrics"]` carries the values provided by a specific evaluation. -`tests/test_metric_delivery.py` checks the whole path with `opa eval` against -GOPAL's own resolver, so a metric that stops arriving fails a test rather than -quietly reverting to a policy that can never be satisfied. +`tests/test_metric_delivery.py` checks the delivery path with `opa eval` against +GOPAL's resolver. A regression that prevents a metric from reaching the policy +therefore fails a test. -## Two things worth getting right +## Metric semantics -**Absent is not zero.** If your metric cannot be computed, say so rather than -returning a flattering default. A toxicity evaluator that returns `0.0` when it -could not run reports an unmeasured system as safe, and gopal will believe it. -This is the failure mode the whole library exists to prevent, and it has caught -us more than once. +**Represent missing measurements as missing.** If a metric cannot be computed, +do not substitute a default numeric value. Returning `0.0` for an unavailable +toxicity measurement would cause policy evaluation to treat an unmeasured +system as if it had received a clean score. -**Say which direction your number points.** A *safety* score where higher is -better and a *toxicity* score where higher is worse are not interchangeable, and -gopal's `is_toxic` once answered `true` for one of the safest possible systems -because they had been treated as if they were. If your metric is a rate, a -maximum and an average are also different questions: gopal keeps -`metrics.toxicity.score` and `metrics.toxicity.max_toxicity` apart precisely -because one is compared against 0.1 and the other against 0.7. +**Match the policy's metric direction and statistic.** A safety score where +higher is better is not interchangeable with a toxicity score where higher is +worse. A maximum and an average also represent different properties. GOPAL +therefore keeps `metrics.toxicity.score` and +`metrics.toxicity.max_toxicity` separate and applies different thresholds to +them. ## Using it in CI @@ -183,11 +175,11 @@ fail a pull request: run: aicertify evaluate --contract contract.json --policy eu_ai_act ``` -Two related commands are worth knowing before you write anything: -`aicertify explain ` prints what input a framework's policies need -and why, and `aicertify init-contract ` scaffolds a contract with -every field in it. Between them and the gap report, you should not have to read -any Rego to find out what is expected of you. +Before implementing an evaluator, use `aicertify explain ` to inspect +the required inputs and `aicertify init-contract --policy ` to +scaffold the declared fields. Together with the metric-gap report, these +commands expose the required input paths without manual inspection of each Rego +file. The evaluators run, their metrics are written into the contract, and the gopal bundle is evaluated against the result. A policy that is not satisfied fails the @@ -197,7 +189,8 @@ step, naming the article and the control. See ## Testing yours [`tests/test_audit_logging_evaluator.py`](../tests/test_audit_logging_evaluator.py) -is a short model. The case worth copying is this one: +provides a compact test pattern. In particular, preserve the distinction between +an explicit negative value and a missing value: ```python def test_a_negative_answer_still_counts_as_answered(self, evaluator): @@ -207,19 +200,17 @@ def test_a_negative_answer_still_counts_as_answered(self, evaluator): """ ``` -Distinguishing "answered no" from "not answered" is the single most common way -an evaluator quietly reports a system as better than it is. +An explicit negative answer must remain distinct from an absent answer; otherwise +the denominator and resulting completeness score can be incorrect. ## The four metrics still showing GAP -Three of them are clinical: `patient_safety`, `clinical_validation` and -`risk_assessment`. They are deliberately not closed. GOPAL gates the first at -0.95, and it means a measurement from a clinical evaluation. The only thing -this codebase could compute in-process is how many safety fields a document -contains, and publishing that under the name `patient_safety.score` would let a -system with complete paperwork clear a patient-safety gate it was never -assessed against. Supply those from the evaluation that produced them, through -the contract, or leave them absent and let the policy fail closed. +Three are clinical: `patient_safety`, `clinical_validation`, and +`risk_assessment`. GOPAL interprets these as measurements from clinical +evaluation. A documentation-completeness count is not equivalent to a clinical +patient-safety score and must not be published under that metric name. Supply +these metrics from the evaluation that produced them or leave them absent so +the policy can fail closed. The fourth is `metrics.model_card.compliance_level`. It was declared by an evaluator that never computed it, and nothing GOPAL derives from a card yields @@ -235,6 +226,6 @@ them `calc` rather than asking anyone to write tooling for work already done: calc metrics.model_card.completeness computed by global.v1.documentation.model_card_score ``` -That is the shape to reach for when a metric turns out to be a judgement about -supplied documentation rather than a measurement of a running system. Write the -rule, not the evaluator. +When a value is a policy judgement over supplied documentation rather than a +measurement of running-system behavior, implement that derivation as a policy +rule rather than as an evaluator. diff --git a/examples/README.md b/examples/README.md index 95c170e..b296549 100644 --- a/examples/README.md +++ b/examples/README.md @@ -34,9 +34,9 @@ example-name/ | [`healthcare-triage-bot/`](healthcare-triage-bot/) | **High risk** (Annex III) | EU AI Act high-risk + gopal healthcare patient-safety | | [`hiring-screening-bot/`](hiring-screening-bot/) | **High risk** (Annex III) | EU AI Act high-risk + fair-lending proxy + global fairness | -### Wanted: more examples +### Contributing an example -The community is welcome to contribute additional examples following the same shape. Open issues track current asks: +Additional examples should follow the same directory shape so they remain directly forkable. Current contribution areas include: - FastAPI integration example - LangChain integration example @@ -47,22 +47,22 @@ The community is welcome to contribute additional examples following the same sh See the [`good first issue`](https://github.com/Principled-Evolution/aicertify/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) and [`help wanted`](https://github.com/Principled-Evolution/aicertify/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) labels. -## Pre-generated sample reports +## Inspecting expected output -If you want to see the AICertify deliverable before installing anything, the `outputs/` directory has historical reports from real runs: +Each maintained application example includes an `expected_report.md`, so its report structure and expected policy results can be inspected without running evaluator dependencies: -- [`outputs/eu_ai_act/`](outputs/eu_ai_act/) — EU AI Act evaluations -- [`outputs/loan_evaluation/`](outputs/loan_evaluation/) — fair-lending evaluations -- [`outputs/medical_diagnosis/`](outputs/medical_diagnosis/) — patient-safety evaluations +- [`customer-support-bot/expected_report.md`](customer-support-bot/expected_report.md) +- [`healthcare-triage-bot/expected_report.md`](healthcare-triage-bot/expected_report.md) +- [`hiring-screening-bot/expected_report.md`](hiring-screening-bot/expected_report.md) -A clean one is also bundled as [`docs/demo-report-eu-ai-act.pdf`](../docs/demo-report-eu-ai-act.pdf). +A generated PDF from the bundled demonstration is available at [`docs/demo-report-eu-ai-act.pdf`](../docs/demo-report-eu-ai-act.pdf). The [`outputs/loan_evaluation/`](outputs/loan_evaluation/) directory also retains a historical contract and PDF from a fair-lending run. ## Authoring conventions When you add an example: -1. Match the directory layout above. The shape matters more than the content; it's what makes the examples forkable. +1. Match the directory layout above. Keep the directory shape consistent so examples can be forked and compared directly. 2. The `metadata` block in `input_contract.json` must declare jurisdiction, risk class, and (if Annex III) the relevant subpoint. 3. `policy_config.yaml` must include a `rationale:` for each framework explaining *why* that framework applies. -4. `expected_report.md` should describe both the pass case **and** the common failure modes a fork might hit. -5. Be honest about scope. A green AICertify report is necessary but not sufficient for production deployment — say so explicitly. +4. `expected_report.md` should describe both the pass case **and** the common failure modes expected when the example is adapted. +5. State scope and evidence limitations explicitly. A passing policy result establishes only that the supplied evidence satisfied the encoded rule at that rule's evidence depth.