Skip to content

parse_held_out_specs silently drops a duplicate held-out column name instead of erroring #463

Description

@yakew7

Where: faircode/proxy.py:34-62 (parse_held_out_specs).

The gap: the function already raises ValueError if a held-out column collides with a real column already in the profiled dataset (line 53-56), but never checks collision between two held-out specs themselves - held_out[column] = ... just overwrites silently.

Repro:

>>> import pandas as pd
>>> from faircode.proxy import parse_held_out_specs
>>> df = pd.DataFrame({'sex': ['M', 'F'] * 5})
>>> def read_table(p):
...     return pd.DataFrame({'race': ['A', 'B'] * 5}) if p == 'a.csv' else pd.DataFrame({'race': ['X', 'Y'] * 5})
>>> held = parse_held_out_specs(['a.csv=race', 'b.csv=race'], df, read_table)
>>> held['race'].tolist()
['X', 'Y', 'X', 'Y', 'X', 'Y', 'X', 'Y', 'X', 'Y']

a.csv=race is silently discarded - no error, no warning - only b.csv=race's data survives.

Why it matters: --proxy-hints-with/held_out_with is designed to accept multiple specs (testing several dropped protected attributes at once); an accidental repeated column name - a typo, a copy-paste, or two different candidate source files meant for the same attribute - silently loses one of the two tests instead of erroring, the same way the sibling real-column collision check already does. This repo's own stated design philosophy elsewhere is loud failure over silent data loss.

Suggested fix: raise ValueError in parse_held_out_specs when column in held_out before assigning, mirroring the existing real-column collision check's error style.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions