Skip to content

compare --fail-on-drift fires on a skipped/unmeasurable comparison, with PSI 0.0 and drift_level 'none' #472

Description

@yakew7

Where: faircode/cli.py's --fail-on-drift check (if args.fail_on_drift and result["flags"]:) vs. its own help text ("fail CI if any dimension drifted"); root cause in faircode/compare.py's _build_flags, which puts every kind of comparison flag - not just actual measured drift - into the single flags list --fail-on-drift checks.

The gap: flags also includes cases where a dimension simply couldn't be meaningfully compared at all (e.g. one side's values are banded ages and the other's are raw, so "drift comparison skipped"), which is not drift and involves no PSI or score-drop.

Repro:

$ python3 -c "
import pandas as pd
pd.DataFrame({'age': [25,30,35,40]*25}).to_csv('/tmp/age_num.csv', index=False)
pd.DataFrame({'age': ['1990-01-01','1985-05-05','1995-03-03','1980-07-07']*25}).to_csv('/tmp/age_dob.csv', index=False)
"
$ faircode compare /tmp/age_num.csv /tmp/age_dob.csv --json | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['dimensions'][0]['psi'], d['dimensions'][0]['drift_level'], d['flags'])"
0.0 none ['age: age values are banded (e.g. "18-30") in one dataset but left raw in the other - drift comparison skipped']

$ faircode compare /tmp/age_num.csv /tmp/age_dob.csv --fail-on-drift; echo "exit: $?"
error: representation drift detected (1 flag(s)) with --fail-on-drift set
exit: 1

PSI is 0.000 and drift_level is explicitly "none" - no drift was measured at all - yet --fail-on-drift still exits 1 with the message "representation drift detected."

Why it matters: a schema change between a training set and a production snapshot (a column reformatted, renamed, or dropped - a realistic real-world scenario) falsely triggers "drift detected" in a CI gate, when what actually happened is unrelated to demographic representation drift and contradicts what the flag's own help text and error message promise.

Suggested fix: either split _build_flags's output so --fail-on-drift only checks the overall-score-drop and drift_level != "none" entries specifically, or reword the help text/error message to clarify that a skipped/unmeasurable comparison counts as a failure too, so the behavior matches what's documented.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions