Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/opa-ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,15 @@ jobs:
# social-scoring gate and the suite still reported 604/604.
- name: Check every policy has a test and an empty-input test
run: scripts/check-test-coverage.sh

# opa check is static and opa test queries individual rules, so neither
# sees a conflict that only arises when a package is evaluated as a
# whole. Two education policies each defined a complete rule named
# `thresholds`, which made the entire package fail at eval time while
# both gates stayed green. A consumer queries data.<package>, so this
# asks each package the same question a consumer asks.
- name: Check every package evaluates without conflict
run: scripts/check-eval-conflicts.sh
working-directory: ${{ github.workspace }}

# The version appears in seven hand-maintained places, including a README
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ gopal/
└── .github/workflows/ OPA + Regal CI
```

**85 production policies. 124 Rego files including tests.**
**92 production policies. 198 Rego files including tests.**

## Useful commands

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

## What you're working on

**GOPAL** is an 85-policy Rego library for AI compliance. The user evaluates inputs (model cards, AI-system metadata, eval results) against named regulations (EU AI Act, NIST RMF, aviation standards, FERPA/COPPA, etc.) and gets back structured compliance verdicts.
**GOPAL** is a 92-policy Rego library for AI compliance. The user evaluates inputs (model cards, AI-system metadata, eval results) against named regulations (EU AI Act, NIST RMF, aviation standards, FERPA/COPPA, etc.) and gets back structured compliance verdicts.

## Fast orientation

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ gopal/
└── custom/ Your private policies (git-ignored, CI-skipped)
```

**92 policies that reach a verdict, plus 7 shared libraries they import. 196 Rego files including tests.** These figures are generated from the tree by [`scripts/generate-coverage.sh`](scripts/generate-coverage.sh) and checked in CI, so they cannot drift from the code. Run `jq .totals docs/coverage/coverage.json` for the current numbers.
**92 policies that reach a verdict, plus 7 shared libraries they import. 198 Rego files including tests.** These figures are generated from the tree by [`scripts/generate-coverage.sh`](scripts/generate-coverage.sh) and checked in CI, so they cannot drift from the code. Run `jq .totals docs/coverage/coverage.json` for the current numbers.

---

Expand Down
3 changes: 2 additions & 1 deletion docs/coverage/coverage.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,11 @@
"policies": 92,
"library_files": 7,
"rego_files": 99,
"rego_files_including_tests": 198,
"policies_with_tests": 92,
"policies_with_empty_input_test": 92,
"decision_rules": 327,
"tests": 842
"tests": 843
},
"frameworks": [
{
Expand Down
4 changes: 2 additions & 2 deletions docs/i18n/README.hi-IN.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Rego में लिखी गई [OPA](https://www.openpolicyagent.org/) प
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="../diagrams/diagram1_hero_numbers_dark.svg">
<img src="../diagrams/diagram1_hero_numbers_light.svg" alt="GOPAL कवरेज: 91 पॉलिसीज़, इंटरनेशनल स्टैंडर्ड्स, एविएशन, इंडस्ट्री वर्टिकल्स, और क्रॉस-कटिंग सिद्धांत" width="85%" />
<img src="../diagrams/diagram1_hero_numbers_light.svg" alt="GOPAL कवरेज: 92 पॉलिसीज़, इंटरनेशनल स्टैंडर्ड्स, एविएशन, इंडस्ट्री वर्टिकल्स, और क्रॉस-कटिंग सिद्धांत" width="85%" />
</picture>
</p>

Expand Down Expand Up @@ -201,7 +201,7 @@ gopal/
└── custom/ Your private policies (git-ignored, CI-skipped)
```

**91 पॉलिसीज़ जो निर्णय देती हैं, साथ ही वे 7 साझा लाइब्रेरीज़ जिन्हें वे इम्पोर्ट करती हैं। टेस्ट्स सहित कुल 196 Rego फ़ाइलें।** ये आंकड़े [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) द्वारा ट्री से जनरेट होते हैं और CI में जाँचे जाते हैं, इसलिए ये कोड से अलग नहीं हो सकते। वर्तमान संख्याओं के लिए `jq .totals docs/coverage/coverage.json` चलाएँ।
**92 पॉलिसीज़ जो निर्णय देती हैं, साथ ही वे 7 साझा लाइब्रेरीज़ जिन्हें वे इम्पोर्ट करती हैं। टेस्ट्स सहित कुल 198 Rego फ़ाइलें।** ये आंकड़े [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) द्वारा ट्री से जनरेट होते हैं और CI में जाँचे जाते हैं, इसलिए ये कोड से अलग नहीं हो सकते। वर्तमान संख्याओं के लिए `jq .totals docs/coverage/coverage.json` चलाएँ।

---

Expand Down
4 changes: 2 additions & 2 deletions docs/i18n/README.ja-JP.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ AI システムのメタデータ、モデルカード、評価結果に対し
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="../diagrams/diagram1_hero_numbers_dark.svg">
<img src="../diagrams/diagram1_hero_numbers_light.svg" alt="GOPAL のカバレッジ: 91 ポリシー、国際規格、航空、業種別、共通原則の各カテゴリ" width="85%" />
<img src="../diagrams/diagram1_hero_numbers_light.svg" alt="GOPAL のカバレッジ: 92 ポリシー、国際規格、航空、業種別、共通原則の各カテゴリ" width="85%" />
</picture>
</p>

Expand Down Expand Up @@ -199,7 +199,7 @@ gopal/
└── custom/ Your private policies (git-ignored, CI-skipped)
```

**判定を返すポリシー 91 個と、それらが読み込む共有ライブラリ 7 個。テストを含む Rego ファイル 196 個。** これらの数値は [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) がツリーから生成し、CI で検証しているため、コードとずれることはありません。現在の値は `jq .totals docs/coverage/coverage.json` で確認できます。
**判定を返すポリシー 92 個と、それらが読み込む共有ライブラリ 7 個。テストを含む Rego ファイル 198 個。** これらの数値は [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) がツリーから生成し、CI で検証しているため、コードとずれることはありません。現在の値は `jq .totals docs/coverage/coverage.json` で確認できます。

---

Expand Down
4 changes: 2 additions & 2 deletions docs/i18n/README.ko-KR.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Rego로 작성된 [OPA](https://www.openpolicyagent.org/) 정책을 엄선한
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="../diagrams/diagram1_hero_numbers_dark.svg">
<img src="../diagrams/diagram1_hero_numbers_light.svg" alt="GOPAL 커버리지: 91개 정책, 국제 표준·항공·산업 수직 영역·공통 원칙" width="85%" />
<img src="../diagrams/diagram1_hero_numbers_light.svg" alt="GOPAL 커버리지: 92개 정책, 국제 표준·항공·산업 수직 영역·공통 원칙" width="85%" />
</picture>
</p>

Expand Down Expand Up @@ -199,7 +199,7 @@ gopal/
└── custom/ 비공개 정책 (git-ignored, CI 제외)
```

**판정을 내리는 정책 91개와 이들이 가져다 쓰는 공유 라이브러리 7개. 테스트를 포함해 총 196개의 Rego 파일.** 이 수치는 [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh)가 트리에서 생성하고 CI에서 검증하므로 코드와 어긋날 수 없습니다. 현재 값은 `jq .totals docs/coverage/coverage.json`으로 확인하세요.
**판정을 내리는 정책 92개와 이들이 가져다 쓰는 공유 라이브러리 7개. 테스트를 포함해 총 198개의 Rego 파일.** 이 수치는 [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh)가 트리에서 생성하고 CI에서 검증하므로 코드와 어긋날 수 없습니다. 현재 값은 `jq .totals docs/coverage/coverage.json`으로 확인하세요.

---

Expand Down
4 changes: 2 additions & 2 deletions docs/i18n/README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="../diagrams/diagram1_hero_numbers_dark.svg">
<img src="../diagrams/diagram1_hero_numbers_light.svg" alt="GOPAL 覆盖范围:91 条策略,涵盖国际标准、航空、行业垂直领域与跨领域原则" width="85%" />
<img src="../diagrams/diagram1_hero_numbers_light.svg" alt="GOPAL 覆盖范围:92 条策略,涵盖国际标准、航空、行业垂直领域与跨领域原则" width="85%" />
</picture>
</p>

Expand Down Expand Up @@ -199,7 +199,7 @@ gopal/
└── custom/ 您的私有策略(已 git-ignore,CI 跳过)
```

**91 条能给出判定的策略,以及它们所引用的 7 个共享库。包括测试在内共 196 个 Rego 文件。** 这些数字由 [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) 从目录树生成,并在 CI 中校验,因此不会与代码脱节。运行 `jq .totals docs/coverage/coverage.json` 可查看当前数值。
**92 条能给出判定的策略,以及它们所引用的 7 个共享库。包括测试在内共 198 个 Rego 文件。** 这些数字由 [`scripts/generate-coverage.sh`](../../scripts/generate-coverage.sh) 从目录树生成,并在 CI 中校验,因此不会与代码脱节。运行 `jq .totals docs/coverage/coverage.json` 可查看当前数值。

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ equitable_admissions_systems_compliant if {
not uses_prohibited_features(declarations.resolve(input, ["admissions_model", "features"]), prohibited_features)
every group in declarations.resolve(input, ["bias_report", "demographic_groups"]) {
every metric in group.fairness_metrics {
is_within_threshold(metric, thresholds)
admissions_metric_within_threshold(metric, admissions_thresholds)
}
}
}
Expand All @@ -35,7 +35,7 @@ deny contains msg if {
failing_metrics := {metric |
some group in declarations.resolve(input, ["bias_report", "demographic_groups"])
some metric in group.fairness_metrics
not is_within_threshold(metric, thresholds)
not admissions_metric_within_threshold(metric, admissions_thresholds)
}
msg := sprintf("Admissions model is not compliant. Fairness metrics are not met: %v", [failing_metrics])
}
Expand All @@ -55,11 +55,11 @@ uses_prohibited_features(features, prohibited) if {
}

# Defines acceptable thresholds for different fairness metrics.
thresholds := {
admissions_thresholds := {
"demographic_parity": 0.1,
"equalized_odds": 0.1,
}

is_within_threshold(metric, thresholds) if {
abs(metric.value) < thresholds[metric.name]
admissions_metric_within_threshold(metric, admissions_thresholds) if {
abs(metric.value) < admissions_thresholds[metric.name]
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ default unbiased_automated_grading_compliant := false
unbiased_automated_grading_compliant if {
every group in declarations.resolve(input, ["bias_report", "demographic_groups"]) {
every metric in group.fairness_metrics {
is_within_threshold(metric, thresholds)
grading_metric_within_threshold(metric, grading_thresholds)
}
}
}
Expand All @@ -27,27 +27,27 @@ deny contains msg if {
failing_metrics := {metric |
some group in declarations.resolve(input, ["bias_report", "demographic_groups"])
some metric in group.fairness_metrics
not is_within_threshold(metric, thresholds)
not grading_metric_within_threshold(metric, grading_thresholds)
}
msg := sprintf("Automated grading model is not compliant. Fairness metrics are not met: %v", [failing_metrics])
}

# --- Helper Functions ---

# Defines acceptable thresholds for different fairness metrics.
thresholds := {
grading_thresholds := {
"equal_opportunity_difference": 0.05,
"average_odds_difference": 0.05,
"disparate_impact": 0.8, # Should be above this value
}

# Checks if a given metric is within its acceptable threshold.
is_within_threshold(metric, thresholds) if {
grading_metric_within_threshold(metric, grading_thresholds) if {
metric.name == "disparate_impact"
metric.value >= thresholds[metric.name]
metric.value >= grading_thresholds[metric.name]
}

is_within_threshold(metric, thresholds) if {
grading_metric_within_threshold(metric, grading_thresholds) if {
metric.name != "disparate_impact"
abs(metric.value) < thresholds[metric.name]
abs(metric.value) < grading_thresholds[metric.name]
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,19 @@ import rego.v1
test_unbiased_automated_grading_compliant_denies_on_empty_input if {
not policy.unbiased_automated_grading_compliant with input as {}
}

# Three policies share this package, and two of them defined a complete rule
# named `thresholds` with different values. OPA raises eval_conflict_error when
# a complete rule produces multiple outputs, which failed the whole package: a
# consumer querying data.industry_specific.education.v1.fairness_and_equity
# received nothing, so all three policies delivered no verdict while opa check
# and opa test both passed.
#
# Referencing both threshold sets in one expression forces them to be evaluated
# together, which is what the package query does and what the individual rule
# tests did not.
test_grading_and_admissions_thresholds_do_not_collide if {
policy.grading_thresholds.disparate_impact == 0.8
policy.admissions_thresholds.demographic_parity == 0.1
policy.grading_thresholds != policy.admissions_thresholds
}
74 changes: 74 additions & 0 deletions scripts/check-eval-conflicts.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#!/usr/bin/env bash
#
# Evaluate every policy package as a whole and fail on an evaluation-time error.
#
# opa check and opa test both passed while three education policies delivered no
# verdict at all. equitable_admissions_systems.rego and
# unbiased_automated_grading.rego share the package
# industry_specific.education.v1.fairness_and_equity, and each defined a complete
# rule named `thresholds` with a different value. OPA raises
#
# eval_conflict_error: complete rules must not produce multiple outputs
#
# which fails the whole package, so all three policies in it returned nothing.
#
# Neither existing gate could see it. `opa check` is static and the conflict is
# an evaluation-time condition. `opa test` queries individual rules, and the
# conflict only arises when both definitions are evaluated together, which is
# what happens when the package itself is queried. A consumer queries
# `data.<package>`, so the gap between how this library tests itself and how it
# is actually used is where the defect lived.
#
# This closes that gap: it asks each package the same question a consumer asks.
#
# Usage: scripts/check-eval-conflicts.sh

set -euo pipefail

REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "${REPO_ROOT}"

COVERAGE="docs/coverage/coverage.json"
[ -f "${COVERAGE}" ] || { echo "error: ${COVERAGE} not found. Run scripts/generate-coverage.sh." >&2; exit 1; }

EMPTY="$(mktemp)"
trap 'rm -f "${EMPTY}"' EXIT
echo '{}' >"${EMPTY}"

# Empty input, deliberately. A conflict between two complete rules is a property
# of the rules, not of the input, and empty input keeps this fast and stable.
# It is also the input under which a policy must still reach a defined default.
DATA_DIRS=(-d global -d international -d industry_specific -d operational -d helper_functions)

packages="$(jq -r '.frameworks[].policies[]? | select(.is_library | not) | .package' "${COVERAGE}" | sort -u)"
total="$(printf '%s\n' "${packages}" | wc -l | tr -d ' ')"
echo "Evaluating ${total} policy packages for evaluation-time errors."

failures=0
while IFS= read -r pkg; do
[ -n "${pkg}" ] || continue
if ! out="$(opa eval -f pretty "${DATA_DIRS[@]}" -i "${EMPTY}" "data.${pkg}" 2>&1)"; then
echo " FAIL ${pkg}" >&2
printf '%s\n' "${out}" | head -3 | sed 's/^/ /' >&2
failures=$((failures + 1))
continue
fi
# opa eval reports some evaluation errors on stdout with a zero exit status.
if printf '%s' "${out}" | grep -qE 'eval_conflict_error|eval_type_error|rego_type_error'; then
echo " FAIL ${pkg}" >&2
printf '%s\n' "${out}" | grep -E 'eval_conflict_error|eval_type_error|rego_type_error' | head -2 | sed 's/^/ /' >&2
failures=$((failures + 1))
fi
done <<<"${packages}"

if [ "${failures}" -gt 0 ]; then
echo "" >&2
echo "${failures} package(s) raise an error when the package is queried as a whole." >&2
echo "A consumer querying data.<package> receives nothing for these, and a tool that" >&2
echo "treats a missing package as 'no verdict' reports success while delivering none." >&2
echo "Usually two files in one package define the same complete rule with different" >&2
echo "values; give each a name of its own." >&2
exit 1
fi

echo "All ${total} packages evaluate without error when queried as a whole."
Loading
Loading