diff --git a/.github/workflows/opa-ci.yaml b/.github/workflows/opa-ci.yaml index 68829dd..d09a9d4 100644 --- a/.github/workflows/opa-ci.yaml +++ b/.github/workflows/opa-ci.yaml @@ -89,6 +89,15 @@ jobs: # social-scoring gate and the suite still reported 604/604. - name: Check every policy has a test and an empty-input test run: scripts/check-test-coverage.sh + + # opa check is static and opa test queries individual rules, so neither + # sees a conflict that only arises when a package is evaluated as a + # whole. Two education policies each defined a complete rule named + # `thresholds`, which made the entire package fail at eval time while + # both gates stayed green. A consumer queries data., so this + # asks each package the same question a consumer asks. + - name: Check every package evaluates without conflict + run: scripts/check-eval-conflicts.sh working-directory: ${{ github.workspace }} # The version appears in seven hand-maintained places, including a README diff --git a/AGENTS.md b/AGENTS.md index c0c5a4b..5de354a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,7 +37,7 @@ gopal/ └── .github/workflows/ OPA + Regal CI ``` -**85 production policies. 124 Rego files including tests.** +**92 production policies. 198 Rego files including tests.** ## Useful commands diff --git a/CLAUDE.md b/CLAUDE.md index daa36c3..140a22a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ ## What you're working on -**GOPAL** is an 85-policy Rego library for AI compliance. The user evaluates inputs (model cards, AI-system metadata, eval results) against named regulations (EU AI Act, NIST RMF, aviation standards, FERPA/COPPA, etc.) and gets back structured compliance verdicts. +**GOPAL** is a 92-policy Rego library for AI compliance. The user evaluates inputs (model cards, AI-system metadata, eval results) against named regulations (EU AI Act, NIST RMF, aviation standards, FERPA/COPPA, etc.) and gets back structured compliance verdicts. ## Fast orientation diff --git a/README.md b/README.md index f9c285a..5e9784d 100644 --- a/README.md +++ b/README.md @@ -277,7 +277,7 @@ gopal/ └── custom/ Your private policies (git-ignored, CI-skipped) ``` -**92 policies that reach a verdict, plus 7 shared libraries they import. 196 Rego files including tests.** These figures are generated from the tree by [`scripts/generate-coverage.sh`](scripts/generate-coverage.sh) and checked in CI, so they cannot drift from the code. Run `jq .totals docs/coverage/coverage.json` for the current numbers. +**92 policies that reach a verdict, plus 7 shared libraries they import. 198 Rego files including tests.** These figures are generated from the tree by [`scripts/generate-coverage.sh`](scripts/generate-coverage.sh) and checked in CI, so they cannot drift from the code. Run `jq .totals docs/coverage/coverage.json` for the current numbers. --- diff --git a/docs/coverage/coverage.json b/docs/coverage/coverage.json index a46152e..90f5ec0 100644 --- a/docs/coverage/coverage.json +++ b/docs/coverage/coverage.json @@ -8,10 +8,11 @@ "policies": 92, "library_files": 7, "rego_files": 99, + "rego_files_including_tests": 198, "policies_with_tests": 92, "policies_with_empty_input_test": 92, "decision_rules": 327, - "tests": 842 + "tests": 843 }, "frameworks": [ { diff --git a/docs/i18n/README.hi-IN.md b/docs/i18n/README.hi-IN.md index 9775d91..a0b1834 100644 --- a/docs/i18n/README.hi-IN.md +++ b/docs/i18n/README.hi-IN.md @@ -42,7 +42,7 @@ Rego में लिखी गई [OPA](https://www.openpolicyagent.org/) प

- GOPAL कवरेज: 91 पॉलिसीज़, इंटरनेशनल स्टैंडर्ड्स, एविएशन, इंडस्ट्री वर्टिकल्स, और क्रॉस-कटिंग सिद्धांत + GOPAL कवरेज: 92 पॉलिसीज़, इंटरनेशनल स्टैंडर्ड्स, एविएशन, इंडस्ट्री वर्टिकल्स, और क्रॉस-कटिंग सिद्धांत

@@ -201,7 +201,7 @@ gopal/ └── custom/ Your private policies (git-ignored, CI-skipped) ``` -**91 पॉलिसीज़ जो निर्णय देती हैं, साथ ही वे 7 साझा लाइब्रेरीज़ जिन्हें वे इम्पोर्ट करती हैं। टेस्ट्स सहित कुल 196 Rego फ़ाइलें।** ये आंकड़े [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) द्वारा ट्री से जनरेट होते हैं और CI में जाँचे जाते हैं, इसलिए ये कोड से अलग नहीं हो सकते। वर्तमान संख्याओं के लिए `jq .totals docs/coverage/coverage.json` चलाएँ। +**92 पॉलिसीज़ जो निर्णय देती हैं, साथ ही वे 7 साझा लाइब्रेरीज़ जिन्हें वे इम्पोर्ट करती हैं। टेस्ट्स सहित कुल 198 Rego फ़ाइलें।** ये आंकड़े [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) द्वारा ट्री से जनरेट होते हैं और CI में जाँचे जाते हैं, इसलिए ये कोड से अलग नहीं हो सकते। वर्तमान संख्याओं के लिए `jq .totals docs/coverage/coverage.json` चलाएँ। --- diff --git a/docs/i18n/README.ja-JP.md b/docs/i18n/README.ja-JP.md index 76baecc..973474d 100644 --- a/docs/i18n/README.ja-JP.md +++ b/docs/i18n/README.ja-JP.md @@ -42,7 +42,7 @@ AI システムのメタデータ、モデルカード、評価結果に対し

- GOPAL のカバレッジ: 91 ポリシー、国際規格、航空、業種別、共通原則の各カテゴリ + GOPAL のカバレッジ: 92 ポリシー、国際規格、航空、業種別、共通原則の各カテゴリ

@@ -199,7 +199,7 @@ gopal/ └── custom/ Your private policies (git-ignored, CI-skipped) ``` -**判定を返すポリシー 91 個と、それらが読み込む共有ライブラリ 7 個。テストを含む Rego ファイル 196 個。** これらの数値は [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) がツリーから生成し、CI で検証しているため、コードとずれることはありません。現在の値は `jq .totals docs/coverage/coverage.json` で確認できます。 +**判定を返すポリシー 92 個と、それらが読み込む共有ライブラリ 7 個。テストを含む Rego ファイル 198 個。** これらの数値は [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) がツリーから生成し、CI で検証しているため、コードとずれることはありません。現在の値は `jq .totals docs/coverage/coverage.json` で確認できます。 --- diff --git a/docs/i18n/README.ko-KR.md b/docs/i18n/README.ko-KR.md index f5ca423..47ec82f 100644 --- a/docs/i18n/README.ko-KR.md +++ b/docs/i18n/README.ko-KR.md @@ -42,7 +42,7 @@ Rego로 작성된 [OPA](https://www.openpolicyagent.org/) 정책을 엄선한

- GOPAL 커버리지: 91개 정책, 국제 표준·항공·산업 수직 영역·공통 원칙 + GOPAL 커버리지: 92개 정책, 국제 표준·항공·산업 수직 영역·공통 원칙

@@ -199,7 +199,7 @@ gopal/ └── custom/ 비공개 정책 (git-ignored, CI 제외) ``` -**판정을 내리는 정책 91개와 이들이 가져다 쓰는 공유 라이브러리 7개. 테스트를 포함해 총 196개의 Rego 파일.** 이 수치는 [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh)가 트리에서 생성하고 CI에서 검증하므로 코드와 어긋날 수 없습니다. 현재 값은 `jq .totals docs/coverage/coverage.json`으로 확인하세요. +**판정을 내리는 정책 92개와 이들이 가져다 쓰는 공유 라이브러리 7개. 테스트를 포함해 총 198개의 Rego 파일.** 이 수치는 [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh)가 트리에서 생성하고 CI에서 검증하므로 코드와 어긋날 수 없습니다. 현재 값은 `jq .totals docs/coverage/coverage.json`으로 확인하세요. --- diff --git a/docs/i18n/README.zh-CN.md b/docs/i18n/README.zh-CN.md index ace6f45..23bbe73 100644 --- a/docs/i18n/README.zh-CN.md +++ b/docs/i18n/README.zh-CN.md @@ -42,7 +42,7 @@

- GOPAL 覆盖范围:91 条策略,涵盖国际标准、航空、行业垂直领域与跨领域原则 + GOPAL 覆盖范围:92 条策略,涵盖国际标准、航空、行业垂直领域与跨领域原则

@@ -199,7 +199,7 @@ gopal/ └── custom/ 您的私有策略(已 git-ignore,CI 跳过) ``` -**91 条能给出判定的策略,以及它们所引用的 7 个共享库。包括测试在内共 196 个 Rego 文件。** 这些数字由 [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) 从目录树生成,并在 CI 中校验,因此不会与代码脱节。运行 `jq .totals docs/coverage/coverage.json` 可查看当前数值。 +**92 条能给出判定的策略,以及它们所引用的 7 个共享库。包括测试在内共 198 个 Rego 文件。** 这些数字由 [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) 从目录树生成,并在 CI 中校验,因此不会与代码脱节。运行 `jq .totals docs/coverage/coverage.json` 可查看当前数值。 --- diff --git a/industry_specific/education/v1/fairness_and_equity/equitable_admissions_systems.rego b/industry_specific/education/v1/fairness_and_equity/equitable_admissions_systems.rego index 85e02d0..f799cb7 100644 --- a/industry_specific/education/v1/fairness_and_equity/equitable_admissions_systems.rego +++ b/industry_specific/education/v1/fairness_and_equity/equitable_admissions_systems.rego @@ -16,7 +16,7 @@ equitable_admissions_systems_compliant if { not uses_prohibited_features(declarations.resolve(input, ["admissions_model", "features"]), prohibited_features) every group in declarations.resolve(input, ["bias_report", "demographic_groups"]) { every metric in group.fairness_metrics { - is_within_threshold(metric, thresholds) + admissions_metric_within_threshold(metric, admissions_thresholds) } } } @@ -35,7 +35,7 @@ deny contains msg if { failing_metrics := {metric | some group in declarations.resolve(input, ["bias_report", "demographic_groups"]) some metric in group.fairness_metrics - not is_within_threshold(metric, thresholds) + not admissions_metric_within_threshold(metric, admissions_thresholds) } msg := sprintf("Admissions model is not compliant. Fairness metrics are not met: %v", [failing_metrics]) } @@ -55,11 +55,11 @@ uses_prohibited_features(features, prohibited) if { } # Defines acceptable thresholds for different fairness metrics. -thresholds := { +admissions_thresholds := { "demographic_parity": 0.1, "equalized_odds": 0.1, } -is_within_threshold(metric, thresholds) if { - abs(metric.value) < thresholds[metric.name] +admissions_metric_within_threshold(metric, admissions_thresholds) if { + abs(metric.value) < admissions_thresholds[metric.name] } diff --git a/industry_specific/education/v1/fairness_and_equity/unbiased_automated_grading.rego b/industry_specific/education/v1/fairness_and_equity/unbiased_automated_grading.rego index 5c6f879..8d845b2 100644 --- a/industry_specific/education/v1/fairness_and_equity/unbiased_automated_grading.rego +++ b/industry_specific/education/v1/fairness_and_equity/unbiased_automated_grading.rego @@ -15,7 +15,7 @@ default unbiased_automated_grading_compliant := false unbiased_automated_grading_compliant if { every group in declarations.resolve(input, ["bias_report", "demographic_groups"]) { every metric in group.fairness_metrics { - is_within_threshold(metric, thresholds) + grading_metric_within_threshold(metric, grading_thresholds) } } } @@ -27,7 +27,7 @@ deny contains msg if { failing_metrics := {metric | some group in declarations.resolve(input, ["bias_report", "demographic_groups"]) some metric in group.fairness_metrics - not is_within_threshold(metric, thresholds) + not grading_metric_within_threshold(metric, grading_thresholds) } msg := sprintf("Automated grading model is not compliant. Fairness metrics are not met: %v", [failing_metrics]) } @@ -35,19 +35,19 @@ deny contains msg if { # --- Helper Functions --- # Defines acceptable thresholds for different fairness metrics. -thresholds := { +grading_thresholds := { "equal_opportunity_difference": 0.05, "average_odds_difference": 0.05, "disparate_impact": 0.8, # Should be above this value } # Checks if a given metric is within its acceptable threshold. -is_within_threshold(metric, thresholds) if { +grading_metric_within_threshold(metric, grading_thresholds) if { metric.name == "disparate_impact" - metric.value >= thresholds[metric.name] + metric.value >= grading_thresholds[metric.name] } -is_within_threshold(metric, thresholds) if { +grading_metric_within_threshold(metric, grading_thresholds) if { metric.name != "disparate_impact" - abs(metric.value) < thresholds[metric.name] + abs(metric.value) < grading_thresholds[metric.name] } diff --git a/industry_specific/education/v1/fairness_and_equity/unbiased_automated_grading_test.rego b/industry_specific/education/v1/fairness_and_equity/unbiased_automated_grading_test.rego index 67c0333..f5f063a 100644 --- a/industry_specific/education/v1/fairness_and_equity/unbiased_automated_grading_test.rego +++ b/industry_specific/education/v1/fairness_and_equity/unbiased_automated_grading_test.rego @@ -10,3 +10,19 @@ import rego.v1 test_unbiased_automated_grading_compliant_denies_on_empty_input if { not policy.unbiased_automated_grading_compliant with input as {} } + +# Three policies share this package, and two of them defined a complete rule +# named `thresholds` with different values. OPA raises eval_conflict_error when +# a complete rule produces multiple outputs, which failed the whole package: a +# consumer querying data.industry_specific.education.v1.fairness_and_equity +# received nothing, so all three policies delivered no verdict while opa check +# and opa test both passed. +# +# Referencing both threshold sets in one expression forces them to be evaluated +# together, which is what the package query does and what the individual rule +# tests did not. +test_grading_and_admissions_thresholds_do_not_collide if { + policy.grading_thresholds.disparate_impact == 0.8 + policy.admissions_thresholds.demographic_parity == 0.1 + policy.grading_thresholds != policy.admissions_thresholds +} diff --git a/scripts/check-eval-conflicts.sh b/scripts/check-eval-conflicts.sh new file mode 100755 index 0000000..2a06bba --- /dev/null +++ b/scripts/check-eval-conflicts.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# +# Evaluate every policy package as a whole and fail on an evaluation-time error. +# +# opa check and opa test both passed while three education policies delivered no +# verdict at all. equitable_admissions_systems.rego and +# unbiased_automated_grading.rego share the package +# industry_specific.education.v1.fairness_and_equity, and each defined a complete +# rule named `thresholds` with a different value. OPA raises +# +# eval_conflict_error: complete rules must not produce multiple outputs +# +# which fails the whole package, so all three policies in it returned nothing. +# +# Neither existing gate could see it. `opa check` is static and the conflict is +# an evaluation-time condition. `opa test` queries individual rules, and the +# conflict only arises when both definitions are evaluated together, which is +# what happens when the package itself is queried. A consumer queries +# `data.`, so the gap between how this library tests itself and how it +# is actually used is where the defect lived. +# +# This closes that gap: it asks each package the same question a consumer asks. +# +# Usage: scripts/check-eval-conflicts.sh + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "${REPO_ROOT}" + +COVERAGE="docs/coverage/coverage.json" +[ -f "${COVERAGE}" ] || { echo "error: ${COVERAGE} not found. Run scripts/generate-coverage.sh." >&2; exit 1; } + +EMPTY="$(mktemp)" +trap 'rm -f "${EMPTY}"' EXIT +echo '{}' >"${EMPTY}" + +# Empty input, deliberately. A conflict between two complete rules is a property +# of the rules, not of the input, and empty input keeps this fast and stable. +# It is also the input under which a policy must still reach a defined default. +DATA_DIRS=(-d global -d international -d industry_specific -d operational -d helper_functions) + +packages="$(jq -r '.frameworks[].policies[]? | select(.is_library | not) | .package' "${COVERAGE}" | sort -u)" +total="$(printf '%s\n' "${packages}" | wc -l | tr -d ' ')" +echo "Evaluating ${total} policy packages for evaluation-time errors." + +failures=0 +while IFS= read -r pkg; do + [ -n "${pkg}" ] || continue + if ! out="$(opa eval -f pretty "${DATA_DIRS[@]}" -i "${EMPTY}" "data.${pkg}" 2>&1)"; then + echo " FAIL ${pkg}" >&2 + printf '%s\n' "${out}" | head -3 | sed 's/^/ /' >&2 + failures=$((failures + 1)) + continue + fi + # opa eval reports some evaluation errors on stdout with a zero exit status. + if printf '%s' "${out}" | grep -qE 'eval_conflict_error|eval_type_error|rego_type_error'; then + echo " FAIL ${pkg}" >&2 + printf '%s\n' "${out}" | grep -E 'eval_conflict_error|eval_type_error|rego_type_error' | head -2 | sed 's/^/ /' >&2 + failures=$((failures + 1)) + fi +done <<<"${packages}" + +if [ "${failures}" -gt 0 ]; then + echo "" >&2 + echo "${failures} package(s) raise an error when the package is queried as a whole." >&2 + echo "A consumer querying data. receives nothing for these, and a tool that" >&2 + echo "treats a missing package as 'no verdict' reports success while delivering none." >&2 + echo "Usually two files in one package define the same complete rule with different" >&2 + echo "values; give each a name of its own." >&2 + exit 1 +fi + +echo "All ${total} packages evaluate without error when queried as a whole." diff --git a/scripts/check-version-refs.sh b/scripts/check-version-refs.sh index 5018d33..dbfa050 100755 --- a/scripts/check-version-refs.sh +++ b/scripts/check-version-refs.sh @@ -114,6 +114,7 @@ fi POLICIES="$(jq -r '.totals.policies' "${COVERAGE}")" FRAMEWORKS="$(jq -r '[.frameworks[] | select(.id | startswith("international"))] | length' "${COVERAGE}")" INDUSTRIES="$(jq -r '[.frameworks[] | select(.id | startswith("industry_specific"))] | length' "${COVERAGE}")" +TOTAL_FILES="$(jq -r '.totals.rego_files_including_tests' "${COVERAGE}")" echo "" echo "Coverage data says ${POLICIES} policies, ${FRAMEWORKS} frameworks, ${INDUSTRIES} industries. Checking every reference agrees." @@ -123,6 +124,7 @@ check_count() { local expected="${template//%P%/${POLICIES}}" expected="${expected//%F%/${FRAMEWORKS}}" expected="${expected//%I%/${INDUSTRIES}}" + expected="${expected//%T%/${TOTAL_FILES}}" if [ ! -f "${file}" ]; then echo " MISSING FILE ${file}" >&2 failures=$((failures + 1)) @@ -153,11 +155,35 @@ check_count docs/diagrams/diagram1_hero_numbers_dark.svg '%P% production R # on the page still read 91. Checking only the accessible text would have # passed that. check_count docs/diagrams/diagram1_hero_numbers_light.svg 'text-anchor="middle">%P%' "numbers diagram figure, light" + +# The count of Rego files including tests was hand-maintained and read 196 +# while the tree held 198, inside the very sentence that says these figures +# are generated and cannot drift. It is now in coverage.json, so it can be +# checked like the rest. +check_count README.md '198 Rego files including tests.' "README total file count" + +# The translated front pages carried the previous count long after the English +# one moved: four of them said 91 policies, including in the alt text that a +# screen reader announces for a diagram rendering 92. A reader arriving in +# Chinese, Japanese, Korean or Hindi saw a different number than a reader +# arriving in English, on the same repository. +check_count docs/i18n/README.zh-CN.md '%P% 条策略' "zh-CN hero alt text" +check_count docs/i18n/README.ja-JP.md '%P% ポリシー' "ja-JP hero alt text" +check_count docs/i18n/README.ko-KR.md '%P%개' "ko-KR hero alt text" +check_count docs/i18n/README.hi-IN.md '%P% पॉलिसीज़' "hi-IN hero alt text" + +# The agent-facing files are read by Claude Code and Codex before they touch +# the tree, so a stale count there is repeated into new work. AGENTS.md said +# 85 policies and 124 files against an actual 92 and 198. +check_count AGENTS.md '**%P% production policies. %T% Rego files including tests.**' "AGENTS.md counts" +check_count CLAUDE.md 'a %P%-policy Rego library' "CLAUDE.md policy count" + check_count docs/diagrams/diagram1_hero_numbers_dark.svg 'text-anchor="middle">%P%' "numbers diagram figure, dark" if [ "${failures}" -gt 0 ]; then - echo "${failures} version reference(s) disagree with VERSION (${VERSION})." >&2 - echo "Bump them, or update scripts/check-version-refs.sh if a reference moved." >&2 + echo "${failures} reference(s) disagree with the generated data:" >&2 + echo " VERSION is ${VERSION}; coverage.json says ${POLICIES} policies, ${TOTAL_FILES} Rego files." >&2 + echo "Bump the stale reference, or update scripts/check-version-refs.sh if one moved." >&2 exit 1 fi echo "All enforced version references agree with ${VERSION}, and all published counts agree with the coverage data." diff --git a/scripts/generate-coverage.sh b/scripts/generate-coverage.sh index ed0b845..fcd4baf 100755 --- a/scripts/generate-coverage.sh +++ b/scripts/generate-coverage.sh @@ -61,6 +61,19 @@ policy_files() { sed 's|^\./||' | LC_ALL=C sort } +# The sibling test files, found the same way. The published figure for "Rego +# files including tests" was hand-maintained and read 196 while the tree held +# 198, in the same sentence that claims these numbers cannot drift from the +# code. Counting them here makes that sentence true. + +test_files() { + find . -type f -name '*_test.rego' \ + -not -path './custom/*' \ + -not -path './.venv-diagrams/*' \ + -not -path './.git/*' | + sed 's|^\./||' | LC_ALL=C sort +} + # Pull an indented "# - value" list that follows a "# Header:" line. comment_list() { local file="$1" header="$2" @@ -263,11 +276,14 @@ test_total="$(opa test "${OPA_IGNORES[@]}" . 2>/dev/null | # --------------------------------------------------------------------------- # 5. Assemble. # --------------------------------------------------------------------------- +test_file_total="$(test_files | wc -l | tr -d " ")" + jq -n >"${TMP}/coverage.json" \ --slurpfile policies <(jq -s '.' "${TMP}/policies.ndjson") \ --slurpfile frameworks <(jq -s '.' "${TMP}/frameworks.ndjson") \ --slurpfile metadata "${TMP}/metadata_map.json" \ --argjson test_total "${test_total}" \ + --argjson test_file_total "${test_file_total}" \ ' ($policies[0]) as $pols | ($frameworks[0]) as $fws @@ -339,6 +355,7 @@ jq -n >"${TMP}/coverage.json" \ policies: ($enriched | map(select(.is_library | not)) | length), library_files: ($enriched | map(select(.is_library)) | length), rego_files: ($enriched | length), + rego_files_including_tests: (($enriched | length) + $test_file_total), policies_with_tests: ($enriched | map(select((.is_library | not) and .has_test)) | length), policies_with_empty_input_test: ($enriched | map(select((.is_library | not) and .has_empty_input_test)) | length),