freshdata

The explainable cleaning layer for pandas — decision-preserving data hygiene.

One call turns a messy CSV, Excel, or SQL export into analysis- and ML-ready data, and tells you exactly what it changed and why.

Documentation · Quickstart · API Reference · Changelog

What is freshdata?

freshdata is an automated data-cleaning library for Python. A rule-based decision engine profiles every column — missing ratio, dtype, skewness, cardinality, inferred role — and chooses the right action per column. Every decision carries a rationale, a risk level, and a confidence score, so nothing happens silently and nothing is left unexplained.

It fills the gap between tools that only describe data and tools that only validate it: freshdata makes the cleaning decision and shows its work, producing reproducible, auditable, ML-ready output with an audit trail you can hand to a reviewer.

Key features

• One-call cleaning — fd.clean(df) handles missing values, outliers, duplicates, dtype repair, and messy column names.
• Per-column decision engine — infers each column's role and applies explicit, documented rules instead of one blunt global strategy.
• Explainable by design — every action carries a rationale, risk level, and confidence score; if a NaN survives, the report says why.
• Safe defaults — never imputes an identifier, modifies a target column, or removes outliers blindly.
• pandas-first, scalable when needed — pandas + NumPy core; pass a Polars frame and get one back, with optional Polars/DuckDB/Spark execution backends for larger-than-memory data.
• CLI included — clean, plan, apply-plan, profile, learn, and trust subcommands for scripting and CI pipelines without writing Python.
• Typed and tested — fully type-hinted (py.typed), vectorized, with a 93% coverage gate enforced in CI.

Installation

pip install freshdata-cleaner

The PyPI distribution is freshdata-cleaner; the import name is freshdata.

Requires Python >= 3.9 and pandas >= 1.5. The core install depends only on pandas and NumPy; everything else is an optional extra:

pip install "freshdata-cleaner[ml,polars]"

Extra	Adds
ml	KNN/model-based imputation
polars	Polars DataFrame support
duckdb	Out-of-core execution via DuckDB
spark	Out-of-core execution via PySpark
viz	Interactive HTML report rendering
privacy	PII detection and anonymization
enterprise	Compliance reporting, orchestration hooks, quality-ops exporters
all	Everything above

See the installation guide for the full list of extras (domain packs, format parsers, streaming, entity resolution, and more).

Quickstart

import pandas as pd
import freshdata as fd

df = pd.read_csv("messy_export.csv")

cleaned = fd.clean(df)                               # one line
cleaned, report = fd.clean(df, return_report=True)   # ... with a full audit trail
print(report.summary())

freshdata clean report
  rows:    525 -> 500 (-25)
  columns: 7 -> 6 (-1)
  missing: 421 -> 0 cell(s)
  memory:  100.8 KB -> 89.2 KB

The same operation is available from the command line:

freshdata clean messy_export.csv -o clean.csv --report audit.json

See the quickstart guide for strategies, reports, and CLI usage.

Beyond core cleaning

Optional layers, all off by default and covered in the documentation:

• Repair plans — suggest a reviewable plan, then apply exactly the approved actions.
• Context policies — compile plain-English cleaning rules into an enforceable policy.
• Streaming — micro-batch and time-series-aware cleaning with bounded memory.
• Privacy — PII detection, masking, and jurisdiction-aware anonymization policies.
• Plugins — extend the engine with your own experts, validators, and backends.
• AI Copilot (experimental) — deterministic, offline dataset analysis that returns an explainable cleaning plan and copy-ready freshdata code; no API key required.

Documentation and examples

• Documentation site — guides, API reference, benchmarks, and honest limitations.
• examples/ — runnable, self-contained scripts, indexed in examples/README.md.

Contributing

Contributions are welcome — standard GitHub flow: fork, branch, add tests, open a pull request. CI runs ruff, mypy, and the fast pytest lane on every PR. See CONTRIBUTING.md for setup and guidelines, and CODE_OF_CONDUCT.md for community standards.

License

MIT — see LICENSE.