Where: faircode/cli.py's --fail-on-drift check (if args.fail_on_drift and result["flags"]:) vs. its own help text ("fail CI if any dimension drifted"); root cause in faircode/compare.py's _build_flags, which puts every kind of comparison flag - not just actual measured drift - into the single flags list --fail-on-drift checks.
The gap: flags also includes cases where a dimension simply couldn't be meaningfully compared at all (e.g. one side's values are banded ages and the other's are raw, so "drift comparison skipped"), which is not drift and involves no PSI or score-drop.
Repro:
$ python3 -c "
import pandas as pd
pd.DataFrame({'age': [25,30,35,40]*25}).to_csv('/tmp/age_num.csv', index=False)
pd.DataFrame({'age': ['1990-01-01','1985-05-05','1995-03-03','1980-07-07']*25}).to_csv('/tmp/age_dob.csv', index=False)
"
$ faircode compare /tmp/age_num.csv /tmp/age_dob.csv --json | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['dimensions'][0]['psi'], d['dimensions'][0]['drift_level'], d['flags'])"
0.0 none ['age: age values are banded (e.g. "18-30") in one dataset but left raw in the other - drift comparison skipped']
$ faircode compare /tmp/age_num.csv /tmp/age_dob.csv --fail-on-drift; echo "exit: $?"
error: representation drift detected (1 flag(s)) with --fail-on-drift set
exit: 1
PSI is 0.000 and drift_level is explicitly "none" - no drift was measured at all - yet --fail-on-drift still exits 1 with the message "representation drift detected."
Why it matters: a schema change between a training set and a production snapshot (a column reformatted, renamed, or dropped - a realistic real-world scenario) falsely triggers "drift detected" in a CI gate, when what actually happened is unrelated to demographic representation drift and contradicts what the flag's own help text and error message promise.
Suggested fix: either split _build_flags's output so --fail-on-drift only checks the overall-score-drop and drift_level != "none" entries specifically, or reword the help text/error message to clarify that a skipped/unmeasurable comparison counts as a failure too, so the behavior matches what's documented.
Where:
faircode/cli.py's--fail-on-driftcheck (if args.fail_on_drift and result["flags"]:) vs. its own help text ("fail CI if any dimension drifted"); root cause infaircode/compare.py's_build_flags, which puts every kind of comparison flag - not just actual measured drift - into the singleflagslist--fail-on-driftchecks.The gap:
flagsalso includes cases where a dimension simply couldn't be meaningfully compared at all (e.g. one side's values are banded ages and the other's are raw, so "drift comparison skipped"), which is not drift and involves no PSI or score-drop.Repro:
PSI is 0.000 and
drift_levelis explicitly "none" - no drift was measured at all - yet--fail-on-driftstill exits 1 with the message "representation drift detected."Why it matters: a schema change between a training set and a production snapshot (a column reformatted, renamed, or dropped - a realistic real-world scenario) falsely triggers "drift detected" in a CI gate, when what actually happened is unrelated to demographic representation drift and contradicts what the flag's own help text and error message promise.
Suggested fix: either split
_build_flags's output so--fail-on-driftonly checks the overall-score-drop anddrift_level != "none"entries specifically, or reword the help text/error message to clarify that a skipped/unmeasurable comparison counts as a failure too, so the behavior matches what's documented.