docs: contributor onboarding — architecture map, roadmap, CoC/SECURITY fixes

ARCHITECTURE.md

@@ -0,0 +1,111 @@
+# Architecture
+
+This is a map for contributors: what lives where in `src/freshdata`, how a
+`clean()` call flows through the code, and where to add new behavior. It is
+deliberately high-level — read it once before your first code change, then let
+the module docstrings and tests fill in the detail.
+
+For *user*-facing explanations of features, see the
+[documentation site](https://freshcode-org.github.io/freshdata/). This file is
+about the *code*.
+
+## The one-sentence version
+
+`fd.clean(df)` profiles every column, asks a rule-based **decision engine** which
+action each column needs, applies those actions as small vectorized **steps**,
+and returns the cleaned frame plus a **report** that explains every change.
+
+## Core flow
+
+```
+fd.clean(df)                      # api.py — the public entry point
+      │
+      ▼
+  Cleaner                         # cleaner.py — orchestrates the pipeline
+      │
+      ├─ profile the frame        # profile.py — per-column stats & inferred role
+      │
+      ├─ decision engine          # engine/ — pick an action per column
+      │     ├─ missing.py         #   imputation strategy
+      │     ├─ outliers.py        #   outlier handling
+      │     ├─ context.py         #   column-role / context inference
+      │     └─ model_select.py    #   when to use model-based imputation
+      │
+      ├─ apply cleaning steps     # steps/ — pure vectorized transforms
+      │     ├─ missing.py  dtypes.py  duplicates.py
+      │     ├─ outliers.py strings.py columns.py
+      │     └─ prune.py    memory.py
+      │
+      └─ build the report         # report.py — every Action, with rationale,
+                                  #   risk, confidence, affected-cell count
+```
+
+Two invariants hold everywhere in this path and are enforced in review (see
+[CONTRIBUTING.md](CONTRIBUTING.md)):
+
+1. **Everything is reported.** A transformation that changes data must record an
+   `Action` (see `report.py`) with an affected-cell count.
+2. **Vectorized only.** No row-wise `apply` or Python loops over rows in the
+   cleaning path.
+
+## Top-level modules
+
+| Module | Responsibility |
+|---|---|
+| `api.py` | The public convenience functions: `clean`, `clean_csv`, `profile`, `plan`, `learn`, `mask_dataframe`, `link`. Start here to trace any public call. |
+| `cleaner.py` | The `Cleaner` front-end and the pipeline that wires profiling → engine → steps → report. |
+| `profile.py` | Read-only profiling: what is in a frame and what `clean()` *would* do, with no mutation. |
+| `plan.py` / `repairplan.py` | Dry-run plans (`plan.py`) and executable, reviewable repair plans (`repairplan.py`). |
+| `report.py`, `result.py`, `_reportframe.py` | The report objects returned to users — `Action` records, summaries, `to_dict()`. |
+| `provenance.py` | Provenance-aware cleaning for OCR / PDF / document-derived tables. |
+| `guard.py` | The hard safety gate that physically protects columns from modification. |
+| `findings.py`, `fieldcheck.py`, `textclean.py`, `textlint.py` | Per-cell validation (`fieldcheck`), field-aware text cleaning (`textclean`), and the normalized finding model (`findings`). |
+| `insight.py`, `stakeholder.py`, `explain.py` | Presentation layers: action-oriented insight output, business-language stakeholder summaries, and per-decision explanations. |
+| `config.py`, `_util.py`, `_sentinels.py` | Configuration and shared internals. |
+
+## Subpackages
+
+Each subpackage has a one-line docstring at the top of its `__init__.py` —
+these summaries come from that code.
+
+| Package | What it does |
+|---|---|
+| `engine/` | The rule-based decision engine behind `strategy=` — picks an action per column. |
+| `steps/` | Individual cleaning steps, each a pure vectorized function. |
+| `imputation/` | Internal imputation engines (mean/median/mode → KNN/model-based with the `ml` extra). |
+| `domains/` | Domain-specific validator packs (finance, healthcare, energy, retail, transport, education, agriculture, media) plus reference data. **The most self-contained place to start** — see [CONTRIBUTING_DOMAINS.md](CONTRIBUTING_DOMAINS.md). |
+| `semantic/` | The offline Semantic Cleaning Layer — experts, scoring, routing. Off by default. |
+| `context/` | The deterministic natural-language context compiler (plain-English rules → policy). |
+| `enterprise/` | Clustering / entity resolution, trust scoring, lineage, and PII masking. |
+| `compliance/` | Maps a `CleanReport` to named control frameworks (21 CFR Part 11, GDPR, HIPAA, SOX, ALCOA+). |
+| `execution/` | Pluggable out-of-core backends (Polars / DuckDB / Spark) behind a shared logical plan. |
+| `streaming/` | Micro-batch and time-series-aware cleaning in bounded memory. |
+| `parsers/` | Structural format readers (HL7 v2, GPX, SDMX, EDIFACT, FHIR). |
+| `render/` | Interactive HTML rendering of report objects (the `viz` extra). |
+| `models/` | Optional local model registry and runtime (`fd.models`). |
+| `learning/` | Paired-data learning: `fd.learn` and reusable `.fdprofile` profiles. |
+| `integrations/` | Run `clean` + a trust gate inside orchestrators (Dagster / Airflow / dbt). |
+| `adapters/` | Optional framework adapters (Polars, etc.). |
+| `experimental/` | Features whose APIs may change between minor releases (e.g. the AI Copilot). |
+| `plugins.py` + `testing.py` | The entry-point plugin system and `fd.testing` contract helpers for third-party experts/backends/validators. |
+
+## Where do I add…?
+
+- **A new cleaning behavior** → a step in `steps/`, wired through the engine in
+  `engine/`, with an `Action` recorded in the report. Add the "does not fire
+  when it shouldn't" test.
+- **A new domain validator** → a pack in `domains/`. Follow
+  [CONTRIBUTING_DOMAINS.md](CONTRIBUTING_DOMAINS.md) — this path is designed to
+  be self-contained and is the friendliest first code contribution.
+- **A new imputation method** → `imputation/`, selected from `engine/model_select.py`.
+- **A new execution backend** → `execution/` + a plugin registration; see
+  [`docs/plugins.md`](https://freshcode-org.github.io/freshdata/plugins/).
+- **A new output/report surface** → `render/`, `insight.py`, or `stakeholder.py`.
+- **A format parser** → `parsers/`.
+
+## What to read next
+
+- [CONTRIBUTING.md](CONTRIBUTING.md) — setup, the checks CI runs, and the PR workflow.
+- [CONTRIBUTING_DOMAINS.md](CONTRIBUTING_DOMAINS.md) — the deep dive for domain packs.
+- [Contributor roadmap](docs/community/contributor-roadmap.md) — tasks grouped by difficulty.
+- The module docstring and the tests next to any file you plan to change.

CODE_OF_CONDUCT.md

@@ -39,3 +39,49 @@ This Code of Conduct applies within all community spaces, and also applies when
 Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders at jyothiswaroop2803@gmail.com. All complaints will be reviewed and investigated promptly and fairly.
 
 All community leaders are obligated to respect the privacy and security of the reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

CONTRIBUTING.md

@@ -4,6 +4,18 @@ Thanks for your interest! freshdata aims to stay **small and sharp** — a
 focused data-cleaning library, not a framework. Contributions that fit that
 philosophy are very welcome.
 
+New to the project? The fastest paths in:
+
+- **Pick something to work on** — the [contributor roadmap](docs/community/contributor-roadmap.md)
+  groups open work by difficulty, from good first issues to architecture work.
+- **Find where your change belongs** — [ARCHITECTURE.md](ARCHITECTURE.md) maps
+  the `src/freshdata` layout and the `clean()` flow.
+- **Ask a question** — open a thread in
+  [Discussions](https://github.com/FreshCode-Org/freshdata/discussions); the
+  issue tracker is for reproducible bugs and concrete proposals.
+
+By participating you agree to the [Code of Conduct](CODE_OF_CONDUCT.md).
+
 ## Ground rules
 
 - **Safety first.** Every statistical change (imputation, outlier handling,
@@ -41,6 +53,10 @@ need network access and local datasets — they run in the nightly CI lane, not
 on PRs. If you change user-facing behavior, update the matching `docs/` page
 and add a note under `Unreleased` in `CHANGELOG.md`.
 
+The 93% coverage gate fires on every `pytest` run, so a single-file run fails it
+on its own. While iterating, add `--no-cov` (e.g. `pytest tests/test_foo.py
+--no-cov`), then run the full fast lane above before opening the PR.
+
 First time contributing? See the
 [first-PR walkthrough](docs/community/first-pr.md).
 

README.md

@@ -153,6 +153,16 @@ for setup and guidelines, and
 [CODE_OF_CONDUCT.md](https://github.com/FreshCode-Org/freshdata/blob/main/CODE_OF_CONDUCT.md)
 for community standards.
 
+New here? Good places to start:
+
+- [Good first issues](https://github.com/FreshCode-Org/freshdata/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
+  and the [contributor roadmap](https://freshcode-org.github.io/freshdata/community/contributor-roadmap/) —
+  open work grouped by difficulty.
+- [ARCHITECTURE.md](https://github.com/FreshCode-Org/freshdata/blob/main/ARCHITECTURE.md) —
+  how the code is laid out.
+- [Discussions](https://github.com/FreshCode-Org/freshdata/discussions) — ask
+  questions and float ideas before you build.
+
 ## License
 
 MIT — see [LICENSE](https://github.com/FreshCode-Org/freshdata/blob/main/LICENSE).

SECURITY.md

@@ -10,8 +10,8 @@ backport fixes to older releases — please upgrade to the latest.
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 1.1.x   | :white_check_mark: |
-| < 1.1   | :x:                |
+| 2.x     | :white_check_mark: |
+| < 2.0   | :x:                |
 
 ## Reporting a Vulnerability
 

SUPPORT.md

@@ -18,6 +18,10 @@ output you got vs expected.
 [private vulnerability reporting](https://github.com/FreshCode-Org/freshdata/security/advisories/new)
 (see SECURITY.md).
 
+**Want to contribute?** Start with [CONTRIBUTING.md](CONTRIBUTING.md), the
+[architecture map](ARCHITECTURE.md), and the
+[contributor roadmap](docs/community/contributor-roadmap.md).
+
 **Support expectations:** FreshData is maintained on a best-effort basis.
 Reproducible bug reports with a failing snippet get priority; "it doesn't
 work" reports without data shape/dtypes usually need a round-trip first.

docs/community/contributor-roadmap.md

@@ -0,0 +1,129 @@
+---
+title: Contributor roadmap
+description: >-
+  A map of freshdata contribution opportunities grouped by difficulty — from
+  good first issues and docs tasks to bug fixes, features, and architecture work.
+keywords: freshdata contributor roadmap, good first issue, open source data cleaning tasks
+---
+
+# Contributor roadmap
+
+This page groups the kinds of work freshdata needs by difficulty, so you can
+find something that matches the time and depth you want to invest. Every tier
+links to *live* issues where they exist — always check the
+[issue tracker](https://github.com/FreshCode-Org/freshdata/issues) for the
+current state, since issues get claimed and closed.
+
+New here? Read the [first-PR guide](first-pr.md) first, then come back to pick a
+tier. For the code layout behind these tasks, see
+[ARCHITECTURE.md](https://github.com/FreshCode-Org/freshdata/blob/main/ARCHITECTURE.md).
+
+!!! tip "Claim before you build"
+    Comment on an issue to claim it before opening a PR, so two people don't do
+    the same work. No issue for what you want to do? Open one (or start a
+    [Discussion](https://github.com/FreshCode-Org/freshdata/discussions)) so a
+    maintainer can confirm the direction before you invest time.
+
+## 🟢 Good first issues — small, self-contained, reviewable without touching the engine
+
+Best for your first PR. These change docs, examples, or an isolated helper, so
+review does not depend on understanding the whole cleaning pipeline.
+
+- **Recipes and examples** — short, runnable scripts showing freshdata next to
+  another tool. Open examples:
+  [pyjanitor interop (#9)](https://github.com/FreshCode-Org/freshdata/issues/9),
+  [Great Expectations recipe (#8)](https://github.com/FreshCode-Org/freshdata/issues/8),
+  [ydata-profiling comparison (#7)](https://github.com/FreshCode-Org/freshdata/issues/7).
+  Files live in [`examples/`](https://github.com/FreshCode-Org/freshdata/tree/main/examples).
+- **Small isolated cleanups** —
+  [de-duplicate `_has_outliers` (#33)](https://github.com/FreshCode-Org/freshdata/issues/33)
+  and
+  [fix boolean coercion in `_coerce_series` (#31)](https://github.com/FreshCode-Org/freshdata/issues/31)
+  are both labeled `good first issue` and name the exact function and file.
+
+**Skills:** Python, pandas basics. **You'll touch:** `examples/`, `docs/`, or one
+named module + its test.
+
+## 📖 Documentation tasks
+
+Docs changes are the fastest way to make a real, mergeable contribution.
+
+- Fix anything unclear or out of date in the
+  [docs site](https://freshcode-org.github.io/freshdata/) — pages live in
+  [`docs/`](https://github.com/FreshCode-Org/freshdata/tree/main/docs).
+- [Comparison-table refresh checklist (#2)](https://github.com/FreshCode-Org/freshdata/issues/2).
+- Add missing docstrings, or clarify a confusing feature guide.
+- Spotted a docs bug? File it with the **Documentation** issue template.
+
+**Skills:** clear writing, Markdown, `mkdocs serve` to preview. **You'll touch:**
+`docs/`, module docstrings.
+
+## 🧪 Testing tasks
+
+freshdata enforces a 93% coverage gate, so tests are always welcome — especially
+"does not fire when it shouldn't" cases and fixtures.
+
+- Add [benchmark preservation / trust-monotonicity / export tests (#82)](https://github.com/FreshCode-Org/freshdata/issues/82).
+- Add synthetic fixture generators:
+  [CRM & finance (#77)](https://github.com/FreshCode-Org/freshdata/issues/77),
+  [event-log CDC & wide-schema (#78)](https://github.com/FreshCode-Org/freshdata/issues/78),
+  [provenance & gold-label repair fixtures (#79)](https://github.com/FreshCode-Org/freshdata/issues/79).
+- Add an [online dataset fixture](https://github.com/FreshCode-Org/freshdata/blob/main/CONTRIBUTING.md#adding-an-online-dataset-fixture)
+  to widen real-world coverage.
+
+**Skills:** pytest, pandas, an eye for edge cases. **You'll touch:** `tests/`,
+`tests/fixtures/`.
+
+## 🐛 Bug fixes
+
+Reproducible bugs with a named symptom — a good step up once you've landed a
+first PR.
+
+- [Polars streaming disabled by default dedupe (#53)](https://github.com/FreshCode-Org/freshdata/issues/53)
+  and
+  [DuckDB `fetchdf()` defeats out-of-core execution (#52)](https://github.com/FreshCode-Org/freshdata/issues/52)
+  — both labeled `bug` + `benchmarks`, in `execution/` / `streaming/`.
+- [`explain_clean` computes `build_contexts` twice (#32)](https://github.com/FreshCode-Org/freshdata/issues/32).
+
+Reproduce first, add a failing test, then fix. See the "Prove-it" habit in the
+[first-PR guide](first-pr.md).
+
+**Skills:** pandas debugging, the affected subpackage. **You'll touch:** the named
+module + a regression test.
+
+## 🚀 Feature development
+
+Mid-sized, maintainer-guided additions. Confirm the API in the issue before
+building.
+
+- [First-class JSON export for `CleanReport` (#21)](https://github.com/FreshCode-Org/freshdata/issues/21)
+  (`help wanted`, mid level).
+- [`fd.clean_csv()` one-line CSV cleanup (#19)](https://github.com/FreshCode-Org/freshdata/issues/19)
+  (`help wanted`, junior level).
+- A **new domain validator pack** — the most self-contained feature path. Follow
+  [CONTRIBUTING_DOMAINS.md](https://github.com/FreshCode-Org/freshdata/blob/main/CONTRIBUTING_DOMAINS.md).
+
+**Skills:** the relevant subpackage, API design sense. **You'll touch:** `api.py`,
+a subpackage, docs, and tests.
+
+## 🏗️ Advanced architecture work
+
+Deep changes to the engine, execution backends, or CI. Discuss the design first.
+
+- CI / release automation:
+  [TestPyPI smoke-install (#6)](https://github.com/FreshCode-Org/freshdata/issues/6),
+  [public benchmark refresh workflow (#4)](https://github.com/FreshCode-Org/freshdata/issues/4),
+  [notebook smoke-test workflow (#3)](https://github.com/FreshCode-Org/freshdata/issues/3).
+- [Benchmark harness CLI with standardized metrics (#80)](https://github.com/FreshCode-Org/freshdata/issues/80).
+- Out-of-core execution correctness across Polars / DuckDB / Spark backends
+  (`execution/`), where the two open backend bugs above also live.
+
+**Skills:** the engine or execution internals, CI, performance. **You'll touch:**
+`engine/`, `execution/`, `.github/workflows/`.
+
+## Where to ask
+
+Not sure which tier fits, or want a maintainer to confirm an approach before you
+start? Open a thread in
+[Discussions](https://github.com/FreshCode-Org/freshdata/discussions) — that is
+the right place for "how should I…?" and "is this worth doing?" questions.