Where: faircode/proxy.py:34-62 (parse_held_out_specs).
The gap: the function already raises ValueError if a held-out column collides with a real column already in the profiled dataset (line 53-56), but never checks collision between two held-out specs themselves - held_out[column] = ... just overwrites silently.
Repro:
>>> import pandas as pd
>>> from faircode.proxy import parse_held_out_specs
>>> df = pd.DataFrame({'sex': ['M', 'F'] * 5})
>>> def read_table(p):
... return pd.DataFrame({'race': ['A', 'B'] * 5}) if p == 'a.csv' else pd.DataFrame({'race': ['X', 'Y'] * 5})
>>> held = parse_held_out_specs(['a.csv=race', 'b.csv=race'], df, read_table)
>>> held['race'].tolist()
['X', 'Y', 'X', 'Y', 'X', 'Y', 'X', 'Y', 'X', 'Y']
a.csv=race is silently discarded - no error, no warning - only b.csv=race's data survives.
Why it matters: --proxy-hints-with/held_out_with is designed to accept multiple specs (testing several dropped protected attributes at once); an accidental repeated column name - a typo, a copy-paste, or two different candidate source files meant for the same attribute - silently loses one of the two tests instead of erroring, the same way the sibling real-column collision check already does. This repo's own stated design philosophy elsewhere is loud failure over silent data loss.
Suggested fix: raise ValueError in parse_held_out_specs when column in held_out before assigning, mirroring the existing real-column collision check's error style.
Where:
faircode/proxy.py:34-62(parse_held_out_specs).The gap: the function already raises
ValueErrorif a held-out column collides with a real column already in the profiled dataset (line 53-56), but never checks collision between two held-out specs themselves -held_out[column] = ...just overwrites silently.Repro:
a.csv=raceis silently discarded - no error, no warning - onlyb.csv=race's data survives.Why it matters:
--proxy-hints-with/held_out_withis designed to accept multiple specs (testing several dropped protected attributes at once); an accidental repeated column name - a typo, a copy-paste, or two different candidate source files meant for the same attribute - silently loses one of the two tests instead of erroring, the same way the sibling real-column collision check already does. This repo's own stated design philosophy elsewhere is loud failure over silent data loss.Suggested fix: raise
ValueErrorinparse_held_out_specswhencolumn in held_outbefore assigning, mirroring the existing real-column collision check's error style.