Skip to content

Add version-to-version data quality and drift checks - #181

Draft
developer-rpai wants to merge 1 commit into
CDCgov:mainfrom
developer-rpai:feature/data-quality-drift-checks
Draft

developer-rpai wants to merge 1 commit into
CDCgov:mainfrom
developer-rpai:feature/data-quality-drift-checks

Conversation

@developer-rpai

Copy link
Copy Markdown

What this adds

Pandera schemas (per-dataset, in catalog repos) validate a single version's
shape. Nothing in cfa.dataops checks that a new version still looks like
the dataset it claims to be. This PR adds a reusable
cfa.dataops.quality module for version-to-version checks:

  • freshness — newest record within max_age_days of the reference date
  • row_count — new version non-empty and within max_change_ratio of old
  • columns — no unexpected added/missing columns
  • dtypes — shared columns kept the same dtype
  • null_rates — per-column null fractions stable within max_delta
  • numeric_drift — column means stable within max_std_shift old-std units
  • categorical_values — no new/vanished categories in string columns

Results aggregate into a QualityReport (.passed, .failed, .summary(),
.as_dict(), .raise_on_failure() for CI gating), plus a
python -m cfa.dataops.quality --new/--old/--config CLI that exits non-zero
on failure. Docs in docs/data_quality.md, changelog entry included.

Relevant to #148 (Quality Assurance and Testing epic).

How it was tested

  • 47 new tests in tests/test_quality.py, including doctests (repo runs
    --doctest-modules on the package): 47 passed
  • ruff check and ruff format --check: clean
  • Full collectible suite (test_quality, test_soda, test_utils_*):
    99 passed. Note: several existing test modules cannot be collected in
    this sandbox (missing mako/pydantic/rich/tomli); that is a
    pre-existing environment limitation, unrelated to this change.
  • The new module imports only pandas + stdlib, so it adds no new
    dependencies.

Open questions for reviewers

  • Is cfa.dataops.quality the right home, or would you prefer this under a
    different module name?
  • Worth adding polars-native paths later, or is pandas sufficient given the
    Pandera-based validation in catalog repos is pandas-based?

Draft PR — happy to rework based on feedback before review.

New cfa.dataops.quality module: Pandera schemas validate a single version's
shape, but nothing checks that a new version still looks like the dataset it
claims to be. Adds seven checks (freshness, row count, columns, dtypes, null
rates, numeric drift, categorical values) aggregated into a QualityReport with
raise_on_failure() for CI gating, plus a `python -m cfa.dataops.quality` CLI
that exits non-zero on failure.

Includes 47 tests (incl. doctests, following the repo's --doctest-modules
convention), docs/data_quality.md, mkdocs nav entry, and a changelog entry.
Relevant to CDCgov#148 (Quality Assurance and Testing epic).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants