From c9c88821b769d3adf2857a248fefeea5689b5f88 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 14 Oct 2026 09:41:24 +0300 Subject: [PATCH 01/26] chore(deps-dev): update pydata-sphinx-theme requirement (#162) Updates the requirements on [pydata-sphinx-theme](https://github.com/pydata/pydata-sphinx-theme) to permit the latest version. - [Release notes](https://github.com/pydata/pydata-sphinx-theme/releases) - [Changelog](https://github.com/pydata/pydata-sphinx-theme/blob/main/RELEASE.md) - [Commits](https://github.com/pydata/pydata-sphinx-theme/compare/v0.16.1...v0.23.0) --- updated-dependencies: - dependency-name: pydata-sphinx-theme dependency-version: 0.23.0 dependency-type: direct:development ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 58a22c2e..10eb43b7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -59,7 +59,7 @@ docs = [ "jupyter-client>=8.7.0,<9.0.0", "nb2plots>=0.7.2,<0.8.0", "numpydoc>=1.10.0,<2.0.0", - "pydata-sphinx-theme>=0.16.1,<0.22.0", + "pydata-sphinx-theme>=0.16.1,<0.24.0", "sphinx>=9.0.4,<10.0.0", "sphinx-copybutton>=0.5.2,<0.6.0", ] From a1126966dda1c71405ae7be386e649bd28da0e0a Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:02:00 +0300 Subject: [PATCH 02/26] docs(164): write the detailed plan for the external data contribution flow --- .../ISSUE_TEMPLATE/grammar-add-template.md | 39 --- .github/ISSUE_TEMPLATE/graph-add-template.md | 78 ----- tasks/detailed_plan.md | 269 ++++++++++++++---- 3 files changed, 207 insertions(+), 179 deletions(-) delete mode 100644 .github/ISSUE_TEMPLATE/grammar-add-template.md delete mode 100644 .github/ISSUE_TEMPLATE/graph-add-template.md diff --git a/.github/ISSUE_TEMPLATE/grammar-add-template.md b/.github/ISSUE_TEMPLATE/grammar-add-template.md deleted file mode 100644 index f649461b..00000000 --- a/.github/ISSUE_TEMPLATE/grammar-add-template.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -name: ADD NEW GRAMMAR -about: Suggest new grammar template to this project -title: "[NEW GRAMMAR] " -labels: new data -assignees: rustam-azimov, vadyushkins - ---- - - - -## Info -| | | -|-----------|------------------------------------------------------------------------| -| Full Name | ```` | -| Version | ```` | -| Class | ```` | -| Kind | ```` | -| Origin | [link](````) | - -## Grammar Parameters - -| Parameter | Description | -|:------------------------------------:|:---------------------------------------:| -| ```` | ```` | - -## Grammar Template - - - -## Grammar Description - - - -## Example Grammars - - - - diff --git a/.github/ISSUE_TEMPLATE/graph-add-template.md b/.github/ISSUE_TEMPLATE/graph-add-template.md deleted file mode 100644 index 2e0be260..00000000 --- a/.github/ISSUE_TEMPLATE/graph-add-template.md +++ /dev/null @@ -1,78 +0,0 @@ -name: ADD NEW GRAPH -about: Suggest new graph to this project -title: "[NEW GRAPH] " -labels: new data -assignees: rustam-azimov, vadyushkins - ---- - - - -## Info -| | | -|---|---| -| Type | ```` | -| Full Name | ```` | -| Category | ``" for a new category (new graphs only)>`` | -| Version | ```` | -| Archive | [link](``.tar.gz, full or partial>``) | - -## Data format - -The main way to provide data is a **Google Drive link** to an archive prepared per the structure-validation tool. Two kinds of archives exist (both documented in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/graphs/index.html)): - -- **Full archive** — a new graph: `.tar.gz` unpacks to a self-contained directory with `README.md`, `graph/` (one Boolean MatrixMarket pattern file per stored edge label), and `queries/` (one directory per query in `cfpq/`, `rpq/`, or `mcfpq/`; each query directory holds every representation of the query — `.cnf`/`.rsm`, `.re`/`.rsm` (regular only), or `.mcfg` — plus one `results.mtx` with the constrained reachability facts). -- **Partial archive** — new queries for an existing graph: `.tar.gz` unpacks to a directory containing only `queries/` — the new query directories (same layout) plus a `README.md` fragment with one `## /` section per new query. - -Validate the archive before sharing it: - -``` -python utils/check_archive_structure.py .tar.gz # full archive -python utils/check_archive_structure.py .tar.gz --partial # partial archive -``` - -A partial archive is merged into the existing graph archive by the maintainer (`python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`), re-validated, and uploaded. The query file formats — including the recursive state machine (`.rsm`) format — are documented on the [FLPQ design page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/flpq.html). - -## Description document - -**Full archive.** The archive's `README.md` must answer all of these questions (one `##` section each): -| | | -|---|---| -| What is this graph? | ```` | -| Source | ```` | -| Construction | ```` | -| Nodes | ```` | -| Edges and labels | ```` | -| Query classes | ```` | -| License | ```` | -| Caveats | ```` | - -**Partial archive.** Instead of a full description document, the `queries/README.md` fragment carries one `## /` section per new query, each describing the language (in words and/or in LaTeX) and its purpose on this graph. - -*New graphs only — a partial archive adds no nodes or edges.* - -## Graph Statistics -| Num Nodes | Num Edges | -|:---:|:---:| -| ```` | ```` | - -## Edges Statistics -| Edge Label | Num Edge Label | -|---:|---:| -| ```` | ```` | - -List the stored labels only (no `_r` reversed-edge rows). A family of indexed labels (e.g. `load_0`, ..., `load_857`) is listed once with a placeholder subscript (e.g. `load_f`) and a note listing the index set. - -## Queries - -Canonical grammars are documented on the category page, not on the per-graph page. Pick the case that applies: - -1. **Graph for an existing category** — name the grammar(s) from the category's "Canonical grammars" section that apply to this graph; they become the table column(s) filled in for the new row. No new grammar text on the site. -2. **New grammar for an existing category** — provide the grammar below; it will be added to the category page and a new column to its table. -3. **Graph for a new category** — provide the canonical grammar(s) below; a new category page will be created with them. - -Every query you provide must ship inside the archive (a full archive ships all queries of the graph; a partial archive ships the new ones): one directory per query in `queries/cfpq/`, `queries/rpq/`, or `queries/mcfpq/`, holding every representation of the query (`.cnf`/`.rsm`, `.re`/`.rsm` — regular only, or `.mcfg`) plus one `results.mtx` with the constrained reachability facts (a Boolean pattern matrix with the graph's dimensions; an entry `(i, j)` means some path from node `i` to node `j` satisfies the query). Each query directory is described in a `## /` section of `queries/README.md`. A query may only use labels of this graph. - - - - diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index fd85bf34..64d41a41 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -1,62 +1,207 @@ -# Detailed Plan: Task 161 — Remove the Version row from per-graph Info tables - -Issue #161. Every per-graph docs page carries a `Version` row in its Info -list-table duplicating the dataset version. On the current pages -(`docs/graphs/data/`, 113 files) it is stale: it says `5.0.0` while the -dataset is `6.0.0` (the 6.0.0 migration re-pointed the download links in -#133 but missed this row). On the historical `old_graphs` pages -(`docs/old_graphs/data/`, 54 files) it says `4.0.0`, consistent with their -URLs but equally redundant. - -## Decision: remove, not automate - -The version is easily discoverable and is not per-graph information: - -- It is the dataset version — one value for all pages — defined once in - `flpq_data/config.py` (`DATASET_VERSION`). -- The "Direct download" URL on the same page already encodes it - (`.../6.0.0/graph/.tar.gz`, `.../4.0.0/graph/.tar.gz`). -- Nothing consumes the row: no tool, test, or template reads it from the - pages. `utils/audit_info_tables.py` and its tests use synthetic field - names; the "Version" field of the `.github/` contribution templates is - contribution metadata (which release an archive targets), not a page field. - -Per the single-source-of-truth principle, a manually maintained duplicate -that has already gone stale once is removed rather than automated. New -per-graph pages (the add-graph skill says to follow the structure of an -existing page) will no longer carry the row. - -## Subtasks - -### S1: Remove the Version row from the 113 current per-graph pages [docs] [done] (cc13b4e) - -**Code:** none (docs only). -**Files:** `docs/graphs/data/*.rst` — delete the two-line Info-table row -(`* - Version` / `- 5.0.0`) from each page; no other line changes. -**Tests:** skip (no code); verify with `python utils/audit_info_tables.py` -— no `Version` field, shapes reduce to `[Full Name, Direct download]` (50) -and `[Full Name, Direct download, Source]` (63). - -### S2: Remove the Version row from the 54 historical old_graphs pages [docs] [done] (850f392) - -**Code:** none (docs only). -**Files:** `docs/old_graphs/data/*.rst` — same deletion (`- 4.0.0` value); -the frozen historical family keeps one uniform page format with the current -one. -**Tests:** skip; verify with -`python utils/audit_info_tables.py --dir docs/old_graphs/data`. - -### S3: Record the removal in the changelog [docs] [done] (ed1d8e5) - -**Code:** none (docs only). -**Files:** `CHANGELOG.md` — a "Changed" entry under `[6.0.0]`: this task -lands on dev before the release merge, so it ships with 6.0.0. -**Tests:** skip; docs build green. - -## Verification - -- `python utils/audit_info_tables.py` on both directories: no `Version` - field in any shape. -- `grep -rn '^\s*\* - Version$' docs/graphs/data docs/old_graphs/data` → - empty. -- Docs build (quality gate) green, no warnings. +# Detailed Plan: Task 164 — External data contribution flow (issue-first) + +Issue #164. Redesign the external contribution path so new data enters via a +single issue carrying one or more validated archives (one graph each), with +mandatory motivation and explicit consistency invariants; delete the +standalone grammar path; document the full case matrix; add the +maintainer-side pipeline tooling. + +## Confirmed decisions (user guidance, verbatim on the issue) + +- One issue may contain **multiple archives** (a set of graphs/queries for a + category); each archive still unpacks to a single directory named after it + (one graph per archive — the fixed structure is unchanged). +- **Delete both** specialized PR templates; replace with one generic PR + template for code contributions. +- **Decline for now**: updates to an existing graph; new representation of an + existing query. Both are documented as not accepted via this flow. +- **Add the utility** deriving `reachable_pairs.csv` rows from an archive. +- Explicit invariants: every new graph ships at least one query; every query + ships reference results (`results.mtx`); no item without description and + motivation. A grammar is a representation of a query — a grammar definition + is accepted only if the same grammar is instantiated as a query directory + with `results.mtx` in a contributed archive. + +## Contribution cases (the matrix) + +| # | Case | Archive(s) | What's new | +|---|------|-----------|-----------| +| 1 | New graph(s), existing category, existing canonical grammar(s) | full | graphs + query instances | +| 2 | New graph(s), existing category, **new grammar** | full | + grammar: new column in the category table, possibly a generator module | +| 3 | New graph(s), **new category**, canonical grammar(s) | full | + category page | +| 4 | **Existing graph(s)**, new queries — any class; sub-case 4b: queries using a **new grammar** | partial | query dirs; + grammar definition if 4b | + +Not accepted via this flow: a grammar/query provided outside an archive (no +reference results); updates to an existing graph; a new representation of an +existing query; more than one graph inside a single archive. + +--- + +### S1: Unified "CONTRIBUTE NEW DATA" issue template + +**Code:** Restructure `.github/ISSUE_TEMPLATE/graph-add-template.md` into +`.github/ISSUE_TEMPLATE/new-data-template.md` (rename + rewrite); delete +`.github/ISSUE_TEMPLATE/grammar-add-template.md`. Reuse the existing +template's field definitions and wording where they survive (Info table, +description-document pointer, statistics conventions, data-format bullets) — +restructure, do not reinvent. + +**Tests:** none (no `.py` files changed). + +**Docs:** the template is the source of truth for the required fields; no +other doc changes in this subtask. + +**Spec:** +- Frontmatter: `name: CONTRIBUTE NEW DATA`, `about: Suggest new data + (graphs, queries, grammars) to this project`, `title: "[NEW DATA] "`, `labels: new data`, `assignees: rustam-azimov, vadyushkins`. +- **Contribution type** — one of the four cases above; an issue may combine + several archives of the same case. +- **Info table** — Version (current FLPQ_Data version); Category (existing + name or `new: `; N/A for case 4). +- **Archives** — one entry per archive (a set is allowed): Name, Kind + (full/partial), Target graph (partial only), Drive link to `.tar.gz`, + SHA256 of the archive. State that each archive unpacks to a single + directory named after it (one graph per archive). +- **Validation (mandatory, per archive)** — the exact command + (`python utils/check_archive_structure.py .tar.gz` or with + `--partial`) plus the pasted output and the FLPQ_Data version used. Include + the no-clone recipe: `pip install flpq-data`, download the single script + from GitHub raw, run it (all imports resolve from the installed package). +- **Motivation (mandatory)** — free text answering why this graph set / + grammar / category is useful to other users; must cover every new item. +- **Reference results (explicit requirement)** — every query directory must + contain `results.mtx`; queries without reference results are not accepted; + every new graph must ship at least one query; every new grammar must be + instantiated as at least one such query in a contributed archive. +- Keep: the data-format bullets (full vs partial layout), the + description-document pointer (the archive README answers the 8 mandatory + questions — not duplicated here), per-graph statistics subsections (new + graphs only: Graph Statistics + Edges Statistics tables with the existing + conventions), and the Queries section reorganized around the case matrix: + which canonical grammar(s) apply (case 1); grammar definition in LaTeX + + file format plus why no existing canonical grammar covers it (cases 2, 3, + 4b); category name + description (case 3). +- **Not accepted via this flow** — explicit list: a grammar/query provided + outside an archive (no reference results); updates to an existing graph; a + new representation of an existing query; more than one graph inside a + single archive. + +### S2: PR template cleanup + +**Code:** Delete `.github/PULL_REQUEST_TEMPLATE/new_graph.md` and +`.github/PULL_REQUEST_TEMPLATE/new_grammar.md`; add a generic +`.github/PULL_REQUEST_TEMPLATE/pull_request.md`. + +**Tests:** none (no `.py` files changed). + +**Docs:** none. + +**Spec:** The generic template is short: a description of the change, the +linked issue (`Closes #`), and a checklist (quality gate passes; docs +updated per the documentation skill; no secrets). Data contributions do not +use PRs — they use the CONTRIBUTE NEW DATA issue (pointer line in the +template). + +### S3: External contribution guide in the docs + +**Code:** Rewrite the "How to add a new graph?" section of +`docs/graphs/index.rst` (currently *"Just create a PR"*) into a +"Contributing data" section. + +**Tests:** Sphinx build via the quality gate; `utils/check_math_snippets.py` +if any math is added (none planned). + +**Docs:** this section becomes the single source of truth for the flow and +the case analysis; field-level details stay in the template (cross-reference, +do not duplicate). + +**Spec:** +- Flow overview: build archive(s) → validate each locally → publish to Drive + → one issue with all archives (link + SHA256 + validation output + + motivation) → maintainers handle upload/merge/registry/docs. +- The no-clone validation recipe (`pip install flpq-data` + raw script URL). +- The four-case table with what each requires (mirror the matrix above; + field details point at the template). +- Explicit consistency invariants: no graph without at least one query, no + query without reference results, no item without description and + motivation. +- Declined cases with reasons (standalone grammar/query without reference + results; graph updates; new representation of an existing query; multiple + graphs per archive). +- Note that a set of graphs for a category goes into one issue as multiple + archives. + +### S4: `utils/add_reachable_pairs.py` — derive CSV rows from an archive + +**Code:** New file `utils/add_reachable_pairs.py`. Reuse from +`check_archive_structure.py`: `unpack_single_dir`, `_mtx_header`, +`validate_archive` (and the `QUERY_CLASSES` mapping). No new parsing logic. + +**Tests:** New `tests/utils/test_add_reachable_pairs.py`. Reuse the tree +builders from `tests/utils/test_check_archive_structure.py` (`_valid_tree`, +`_partial_tree`, `_tarball`) by importing them from that module (same +directory; pytest prepends it to `sys.path`). + +**Docs:** numpydoc docstring with doctest examples on the public function; a +short section in `docs/utils.rst` following the existing per-tool pattern +(`.. _add_reachable_pairs:` anchor, usage line, behavior summary) — the +docstring stays the source of truth for details. + +**Spec:** +- CLI: `python utils/add_reachable_pairs.py ARCHIVE.tar.gz --graph NAME + --category CAT [--partial] [--dry-run]`. +- Validate the archive first (full or partial mode per `--partial`); refuse + to proceed on any structure problem. +- For each `queries///` directory emit one row: + `(graph, ., category, class, nnz)` where `` is the first + present of `cnf > re > mcfg > rsm` (must be an actual file in the query + directory; all current rows are `.cnf`) and `nnz` is the declared entry + count from the `results.mtx` header. +- Write to `flpq_data/dataset/reachable_pairs.csv` preserving the existing + alphabetical order by graph name (insert, don't resort); refuse a duplicate + `(graph, query_class, stem)` row; `--dry-run` prints the rows and writes + nothing. +- Exit non-zero on any problem (invalid archive, duplicate row, missing + representation file), printing each problem. + +### S5: Maintainer acceptance pipeline in the skills + +**Code:** Extend `.opencode/skills/add-graph/SKILL.md` with an "Accepting an +external contribution" section; update `.opencode/skills/add-grammar/SKILL.md` +(source-of-truth pointers + the grammar-arrives-with-data rule). + +**Tests:** none (markdown only). + +**Docs:** skills stay thin pointers to tools/docs — no duplication of tool +docstrings or template fields. + +**Spec:** +- `add-graph`: per archive of the issue — download from the Drive link → + verify SHA256 against the issue → re-run `check_archive_structure.py` + (full/`--partial`) → name uniqueness vs `registry.json` / + `reachable_pairs.csv` → add CSV rows via `utils/add_reachable_pairs.py` → + upload (full) or `merge_archive.py` (partial; re-validates and checks the + CSV rows). Once per issue — `generate_registry.py` → the existing docs + wiring steps (per-graph pages, category table(s), canonical grammars, new + category page) → `reachable_pairs_tables.py --update` + + `archive_sizes.py --update`. +- `add-grammar`: grammars arrive only together with a data contribution + (cases 2/3/4b of the CONTRIBUTE NEW DATA issue); pointer to the template; + the generator-module conventions stay as-is. + +### S6: Changelog + +**Code:** none. **Tests:** none. + +**Docs:** Unreleased entry in `CHANGELOG.md`: issue-first contribution flow +with multi-archive issues, unified CONTRIBUTE NEW DATA template, removed +specialized templates, new `utils/add_reachable_pairs.py`. + +--- + +## Execution order + +S1 → S2 → S3 → S4 → S5 → S6. One commit per subtask +(`feat(164-SN)` / `docs(164-SN)`), code review + quality gate before merge to +`dev`; the last subtask's commit carries `Closes #164`. From 4d1c42c60ca707626cd354a89026db4fd94ea9ad Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:02:00 +0300 Subject: [PATCH 03/26] docs(164-S1): replace graph and grammar templates with the unified CONTRIBUTE NEW DATA template Data contributions become issue-first: one issue carries one or more validated archives (one graph each) plus motivation, and maintainers do all repo work. The standalone grammar template is removed because a grammar is a representation of a query and must ship inside an archive with reference results. The new template makes the four contribution cases explicit, requires per-archive validation output and SHA256, mandates motivation, and states the consistency invariants (no graph without a query, no query without results.mtx) and the declined cases. README and skill pointers are updated to the new template. --- .github/ISSUE_TEMPLATE/new-data-template.md | 123 ++++++++++++++++++++ .opencode/skills/add-grammar/SKILL.md | 10 +- .opencode/skills/add-graph/SKILL.md | 10 +- README.rst | 9 +- 4 files changed, 136 insertions(+), 16 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/new-data-template.md diff --git a/.github/ISSUE_TEMPLATE/new-data-template.md b/.github/ISSUE_TEMPLATE/new-data-template.md new file mode 100644 index 00000000..71cb7b79 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/new-data-template.md @@ -0,0 +1,123 @@ +--- +name: CONTRIBUTE NEW DATA +about: Suggest new data (graphs, queries, grammars) to this project +title: "[NEW DATA] " +labels: new data +assignees: rustam-azimov, vadyushkins + +--- + + + +## Contribution type + +Pick the case that applies to this issue (an issue may combine several archives of the same case): + +1. **New graph(s), existing category, existing grammar(s)** — full archive(s); the queries instantiate the category's canonical grammar(s). +2. **New graph(s), existing category, new grammar** — full archive(s) plus a grammar definition (see "Queries and grammars" below). +3. **New graph(s), new category** — full archive(s) plus the category name/description and its canonical grammar(s). +4. **New queries for existing graph(s)** — partial archive(s); sub-case 4b if a query uses a new grammar. + +## Info +| | | +|---|---| +| Version | ```` | +| Category | ``" (N/A for case 4)>`` | + +## Archives + +One row per archive. An issue may carry a **set** of archives (e.g. all graphs of a new category) — but one graph per archive: each `.tar.gz` unpacks to a single directory named after it. + +| Name | Kind | Target graph | Archive | SHA256 | +|---|---|---|---|---| +| ```` | ```` | ```` | [link](``.tar.gz>``) | ``.tar.gz`>`` | + +## Validation (mandatory, per archive) + +Validate every archive **before** sharing it and paste the output below. Without cloning the repo: `pip install flpq-data`, download the single script [`check_archive_structure.py`](https://raw.githubusercontent.com/FormalLanguageConstrainedPathQuerying/FLPQ_Data/master/utils/check_archive_structure.py), and run it (all imports resolve from the installed package). + +### `.tar.gz` + +``` +python check_archive_structure.py .tar.gz # full archive +python check_archive_structure.py .tar.gz --partial # partial archive + structure ok +``` + +FLPQ_Data version used: ```` + +(One subsection per archive.) + +## Motivation (mandatory) + +```` + +## Reference results (mandatory) + +Every query directory ships one `results.mtx` with the constrained reachability facts — a Boolean pattern matrix with the graph's dimensions, where an entry `(i, j)` means some path from node `i` to node `j` satisfies the query. **Queries without reference results are not accepted.** Every new graph must ship at least one query; every new grammar must be instantiated as at least one such query in a contributed archive. + +## Data format + +Two kinds of archives exist (both documented in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/graphs/index.html)): + +- **Full archive** — a new graph: `.tar.gz` unpacks to a self-contained directory with `README.md`, `graph/` (one Boolean MatrixMarket pattern file per stored edge label), and `queries/` (one directory per query in `cfpq/`, `rpq/`, or `mcfpq/`; each query directory holds every representation of the query — `.cnf`/`.rsm`, `.re`/`.rsm` (regular only), or `.mcfg` — plus one `results.mtx`). +- **Partial archive** — new queries for an existing graph: `.tar.gz` unpacks to a directory named after that graph containing only `queries/` — the new query directories (same layout) plus a `README.md` fragment with one `## /` section per new query. Partial archives are merged into the existing graph archive by the maintainers. + +The query file formats — including the recursive state machine (`.rsm`) format — are documented on the [FLPQ design page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/flpq.html). + +## Description document + +**Full archive.** The archive's `README.md` must answer all of these questions (one `##` section each): +| | | +|---|---| +| What is this graph? | ```` | +| Source | ```` | +| Construction | ```` | +| Nodes | ```` | +| Edges and labels | ```` | +| Query classes | ```` | +| License | ```` | +| Caveats | ```` | + +**Partial archive.** Instead of a full description document, the `queries/README.md` fragment carries one `## /` section per new query, each describing the language (in words and/or in LaTeX) and its purpose on this graph. + +*New graphs only — a partial archive adds no nodes or edges.* + +## Graph statistics (new graphs only) + +One subsection per new graph: + +### `` + +| Num Nodes | Num Edges | +|:---:|:---:| +| ```` | ```` | + +| Edge Label | Num Edge Label | +|---:|---:| +| ```` | ```` | + +List the stored labels only (no `_r` reversed-edge rows). A family of indexed labels (e.g. `load_0`, ..., `load_857`) is listed once with a placeholder subscript (e.g. `load_f`) and a note listing the index set. + +## Queries and grammars + +Canonical grammars are documented on the category page, not on the per-graph page. Pick the case that applies: + +1. **Existing grammar(s)** — name the grammar(s) from the category's "Canonical grammars" section that apply; they become the table column(s) filled in for the new row(s). No new grammar text on the site. +2. **New grammar** — provide the grammar below; it will be added to the category page and a new column to its table. The same grammar must also be instantiated as at least one query directory with `results.mtx` in a contributed archive. +3. **New category** — provide the canonical grammar(s) below plus the category name and description; a new category page will be created with them. + +Every query you provide must ship inside an archive (a full archive ships all queries of the graph; a partial archive ships the new ones): one directory per query in `queries/cfpq/`, `queries/rpq/`, or `queries/mcfpq/`, holding every representation of the query plus one `results.mtx`. Each query directory is described in a `## /` section of `queries/README.md`. A query may only use labels of its graph. + + + + + + + +## Not accepted via this flow + +- A grammar or query provided outside an archive — without reference results. +- Updates to an existing graph (a new version of the source dataset). +- A new representation of an existing query (e.g. adding `.rsm` next to a `.cnf`-only query). +- More than one graph inside a single archive (use several archives in one issue instead). diff --git a/.opencode/skills/add-grammar/SKILL.md b/.opencode/skills/add-grammar/SKILL.md index 2bcb9d76..e15905b3 100644 --- a/.opencode/skills/add-grammar/SKILL.md +++ b/.opencode/skills/add-grammar/SKILL.md @@ -7,13 +7,13 @@ description: Use when adding a new grammar template to FLPQ_Data. Covers the gen ## Source of truth -The required fields and metadata live in the GitHub templates — do **not** -duplicate them here: +The required fields and metadata live in the GitHub issue template — do +**not** duplicate them here: -- `.github/PULL_REQUEST_TEMPLATE/new_grammar.md` -- `.github/ISSUE_TEMPLATE/grammar-add-template.md` +- `.github/ISSUE_TEMPLATE/new-data-template.md` (the "Queries and grammars" + section; a grammar arrives only together with a data contribution) -Fill in every field in the triangle brackets (`<>`) of those templates. +Fill in every field in the triangle brackets (`<>`) of that template. ## Package conventions diff --git a/.opencode/skills/add-graph/SKILL.md b/.opencode/skills/add-graph/SKILL.md index ff9d2576..aa85f8ab 100644 --- a/.opencode/skills/add-graph/SKILL.md +++ b/.opencode/skills/add-graph/SKILL.md @@ -7,13 +7,13 @@ description: Use when adding a new graph to FLPQ_Data. Covers the mtx-per-label ## Source of truth -The required fields and metadata live in the GitHub templates — do **not** -duplicate them here: +The required fields and metadata live in the GitHub issue template — do +**not** duplicate them here: -- `.github/PULL_REQUEST_TEMPLATE/new_graph.md` -- `.github/ISSUE_TEMPLATE/graph-add-template.md` +- `.github/ISSUE_TEMPLATE/new-data-template.md` (data contributions are + issue-first; there is no PR path for data) -Fill in every field in the triangle brackets (`<>`) of those templates. +Fill in every field in the triangle brackets (`<>`) of that template. ## Data format diff --git a/README.rst b/README.rst index daed4bf8..b63afd7c 100644 --- a/README.rst +++ b/README.rst @@ -85,13 +85,10 @@ The full developer guide — development setup, quality checks, the test pipeline, docs build and deployment, the package release process, and contribution guidelines — is on the `Developer `_ page. -How to add a new graph? -*********************** +How to contribute new data? +************************** -Just create - -- an ``Issue`` corresponding to the `"Issue template for adding a new graph" `_. -- a ``Pull Request`` corresponding to the `"Pull request template for adding a new graph" `_. +Create an ``Issue`` corresponding to the `"CONTRIBUTE NEW DATA" issue template `_ — one or more validated archives (one graph each), links to them, and the motivation. The maintainers upload and merge the data into the dataset; the full contribution guide is on the `Graphs `_ page. Citation ******** From b71e403babe167c99df6c6ef7f57c73e178837ab Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:02:55 +0300 Subject: [PATCH 04/26] docs(164): mark S1 done in the detailed plan --- tasks/detailed_plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index 64d41a41..fb11db72 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -37,7 +37,7 @@ existing query; more than one graph inside a single archive. --- -### S1: Unified "CONTRIBUTE NEW DATA" issue template +### S1: Unified "CONTRIBUTE NEW DATA" issue template [done] 4d1c42c **Code:** Restructure `.github/ISSUE_TEMPLATE/graph-add-template.md` into `.github/ISSUE_TEMPLATE/new-data-template.md` (rename + rewrite); delete From ba5d880979e0c0123f9151847f0705a0c2030dfa Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:04:53 +0300 Subject: [PATCH 05/26] docs(164-S2): replace specialized PR templates with a generic one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Data contributions are issue-first, so the new_graph and new_grammar PR templates — duplicates of the issue template's fields — are removed. A generic PR template covers code contributions and points data contributors to the CONTRIBUTE NEW DATA issue; docs/dataset.rst now says grammars arrive together with the data they apply to. --- .github/PULL_REQUEST_TEMPLATE/new_grammar.md | 30 -------- .github/PULL_REQUEST_TEMPLATE/new_graph.md | 70 ------------------- .github/PULL_REQUEST_TEMPLATE/pull_request.md | 15 ++++ docs/dataset.rst | 2 +- 4 files changed, 16 insertions(+), 101 deletions(-) delete mode 100644 .github/PULL_REQUEST_TEMPLATE/new_grammar.md delete mode 100644 .github/PULL_REQUEST_TEMPLATE/new_graph.md create mode 100644 .github/PULL_REQUEST_TEMPLATE/pull_request.md diff --git a/.github/PULL_REQUEST_TEMPLATE/new_grammar.md b/.github/PULL_REQUEST_TEMPLATE/new_grammar.md deleted file mode 100644 index e44382c9..00000000 --- a/.github/PULL_REQUEST_TEMPLATE/new_grammar.md +++ /dev/null @@ -1,30 +0,0 @@ - - -## Info -| | | -|-----------|------------------------------------------------------------------------| -| Full Name | ```` | -| Version | ```` | -| Class | ```` | -| Kind | ```` | -| Origin | [link](````) | - -## Grammar Parameters - -| Parameter | Description | -|:------------------------------------:|:---------------------------------------:| -| ```` | ```` | - -## Grammar Template - - - -## Grammar Description - - - -## Example Grammars - - - - diff --git a/.github/PULL_REQUEST_TEMPLATE/new_graph.md b/.github/PULL_REQUEST_TEMPLATE/new_graph.md deleted file mode 100644 index 1d5bc96d..00000000 --- a/.github/PULL_REQUEST_TEMPLATE/new_graph.md +++ /dev/null @@ -1,70 +0,0 @@ - - -## Info -| | | -|---|---| -| Type | ```` | -| Full Name | ```` | -| Category | ``" for a new category (new graphs only)>`` | -| Version | ```` | -| Archive | [link](``.tar.gz, full or partial>``) | - -## Data format - -The main way to provide data is a **Google Drive link** to an archive prepared per the structure-validation tool. Two kinds of archives exist (both documented in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/graphs/index.html)): - -- **Full archive** — a new graph: `.tar.gz` unpacks to a self-contained directory with `README.md`, `graph/` (one Boolean MatrixMarket pattern file per stored edge label), and `queries/` (one directory per query in `cfpq/`, `rpq/`, or `mcfpq/`; each query directory holds every representation of the query — `.cnf`/`.rsm`, `.re`/`.rsm` (regular only), or `.mcfg` — plus one `results.mtx` with the constrained reachability facts). -- **Partial archive** — new queries for an existing graph: `.tar.gz` unpacks to a directory containing only `queries/` — the new query directories (same layout) plus a `README.md` fragment with one `## /` section per new query. - -Validate the archive before sharing it: - -``` -python utils/check_archive_structure.py .tar.gz # full archive -python utils/check_archive_structure.py .tar.gz --partial # partial archive -``` - -A partial archive is merged into the existing graph archive by the maintainer (`python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`), re-validated, and uploaded. The query file formats — including the recursive state machine (`.rsm`) format — are documented on the [FLPQ design page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/flpq.html). - -## Description document - -**Full archive.** The archive's `README.md` must answer all of these questions (one `##` section each): -| | | -|---|---| -| What is this graph? | ```` | -| Source | ```` | -| Construction | ```` | -| Nodes | ```` | -| Edges and labels | ```` | -| Query classes | ```` | -| License | ```` | -| Caveats | ```` | - -**Partial archive.** Instead of a full description document, the `queries/README.md` fragment carries one `## /` section per new query, each describing the language (in words and/or in LaTeX) and its purpose on this graph. - -*New graphs only — a partial archive adds no nodes or edges.* - -## Graph Statistics -| Num Nodes | Num Edges | -|:---:|:---:| -| ```` | ```` | - -## Edges Statistics -| Edge Label | Num Edge Label | -|---:|---:| -| ```` | ```` | - -List the stored labels only (no `_r` reversed-edge rows). A family of indexed labels (e.g. `load_0`, ..., `load_857`) is listed once with a placeholder subscript (e.g. `load_f`) and a note listing the index set. - -## Queries - -Canonical grammars are documented on the category page, not on the per-graph page. Pick the case that applies: - -1. **Graph for an existing category** — name the grammar(s) from the category's "Canonical grammars" section that apply to this graph; they become the table column(s) filled in for the new row. No new grammar text on the site. -2. **New grammar for an existing category** — provide the grammar below; it will be added to the category page and a new column to its table. -3. **Graph for a new category** — provide the canonical grammar(s) below; a new category page will be created with them. - -Every query you provide must ship inside the archive (a full archive ships all queries of the graph; a partial archive ships the new ones): one directory per query in `queries/cfpq/`, `queries/rpq/`, or `queries/mcfpq/`, holding every representation of the query (`.cnf`/`.rsm`, `.re`/`.rsm` — regular only, or `.mcfg`) plus one `results.mtx` with the constrained reachability facts (a Boolean pattern matrix with the graph's dimensions; an entry `(i, j)` means some path from node `i` to node `j` satisfies the query). Each query directory is described in a `## /` section of `queries/README.md`. A query may only use labels of this graph. - - - - diff --git a/.github/PULL_REQUEST_TEMPLATE/pull_request.md b/.github/PULL_REQUEST_TEMPLATE/pull_request.md new file mode 100644 index 00000000..8b007aae --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/pull_request.md @@ -0,0 +1,15 @@ + + +## Description + + + +## Linked issue + +Closes # + +## Checklist + +- [ ] Quality gate passes (tests, style, type check, docs build) +- [ ] Documentation updated per the documentation conventions +- [ ] No secrets or credentials included diff --git a/docs/dataset.rst b/docs/dataset.rst index 757934de..f013c4a6 100644 --- a/docs/dataset.rst +++ b/docs/dataset.rst @@ -26,4 +26,4 @@ list of applicable graphs. How to add a new grammar? ------------------------- -Just create a PR (Pull Request) corresponding to the `"Template for adding a new grammar" `_. +Grammars are added together with the data they apply to — create an ``Issue`` corresponding to the `"CONTRIBUTE NEW DATA" issue template `_ (the "Queries and grammars" section). A grammar is accepted only if it is instantiated as at least one query with reference results in a contributed archive. From a8ef42e957428c5b016219feee3e6018b040adca Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:05:24 +0300 Subject: [PATCH 06/26] docs(164): mark S2 done in the detailed plan --- tasks/detailed_plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index fb11db72..47ec70c6 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -87,7 +87,7 @@ other doc changes in this subtask. new representation of an existing query; more than one graph inside a single archive. -### S2: PR template cleanup +### S2: PR template cleanup [done] ba5d880 **Code:** Delete `.github/PULL_REQUEST_TEMPLATE/new_graph.md` and `.github/PULL_REQUEST_TEMPLATE/new_grammar.md`; add a generic From f9076cd922ca60a6c77ad4ad31771a6c3b4b758a Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:18:36 +0300 Subject: [PATCH 07/26] docs(164-S3): rewrite the graph contribution section as the external contribution guide The old 'How to add a new graph?' section told contributors to open a PR, which contradicts the issue-first flow. The new 'Contributing data' section documents the flow (archives -> validation -> one issue -> maintainers), the four contribution cases, the per-archive validation command, the consistency invariants (no graph without a query, no query without reference results, no item without description and motivation), and the declined cases. Subsection underlines use the file's existing '^' style. --- docs/graphs/index.rst | 64 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 61 insertions(+), 3 deletions(-) diff --git a/docs/graphs/index.rst b/docs/graphs/index.rst index 8afe170b..6b3c9c12 100644 --- a/docs/graphs/index.rst +++ b/docs/graphs/index.rst @@ -9,10 +9,68 @@ Graphs :Release: |release| :Date: |today| -How to add a new graph? ------------------------ +.. _contributing_data: -Just create a PR (Pull Request) corresponding to the `"Template for adding a new graph" `_. +Contributing data +----------------- + +New data — graphs, queries, grammars — enters the dataset through a single +`"CONTRIBUTE NEW DATA" issue `_: +you prepare one or more archives (one graph each), validate them, publish +them (e.g. to Google Drive), and open the issue with the links, the +validation output, the SHA256 of every archive, and the motivation. The +maintainers upload the archives and merge them into the dataset — no pull +request is needed. + +Contribution cases +^^^^^^^^^^^^^^^^^^ + +1. **New graph(s), existing category, existing grammar(s)** — full + archive(s); the queries instantiate the category's canonical grammar(s). +2. **New graph(s), existing category, new grammar** — full archive(s) plus a + grammar definition; the grammar gets a new column in the category table. +3. **New graph(s), new category** — full archive(s) plus the category + name/description and its canonical grammar(s); a new category page is + created. +4. **New queries for existing graph(s)** — partial archive(s); sub-case 4b + if a query uses a new grammar. + +A set of graphs for a category goes into one issue as several archives. The +issue template lists every required field per case (statistics, grammar +definitions, category description). + +Validation +^^^^^^^^^^ + +Every archive must pass the structure checker before it is shared:: + + python utils/check_archive_structure.py .tar.gz # full archive + python utils/check_archive_structure.py .tar.gz --partial # partial archive + +The issue carries the pasted output of this command per archive, the +archive's SHA256, and the FLPQ_Data version used; the template has the +recipe for running the checker without cloning the repo. + +Consistency invariants +^^^^^^^^^^^^^^^^^^^^^^ + +- No graph without at least one query. +- No query without reference results (``results.mtx``). +- No item without description and motivation: every graph is described in + its archive ``README.md``, every query in ``queries/README.md``, and the + issue states why each new graph, grammar, or category is useful to other + users. + +Not accepted via this flow +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- A grammar or query provided outside an archive — without reference + results. +- Updates to an existing graph (a new version of the source dataset). +- A new representation of an existing query (e.g. adding ``.rsm`` next to a + ``.cnf``-only query). +- More than one graph inside a single archive (use several archives in one + issue instead). .. _graph_file_structure: From 85659797ad5b4d78521688606d10964992f29b49 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:19:49 +0300 Subject: [PATCH 08/26] docs(164): mark S3 done in the detailed plan --- tasks/detailed_plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index 47ec70c6..0dcd3557 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -103,7 +103,7 @@ updated per the documentation skill; no secrets). Data contributions do not use PRs — they use the CONTRIBUTE NEW DATA issue (pointer line in the template). -### S3: External contribution guide in the docs +### S3: External contribution guide in the docs [done] f9076cd **Code:** Rewrite the "How to add a new graph?" section of `docs/graphs/index.rst` (currently *"Just create a PR"*) into a From 23f00116a5bdd572835bebef2313a4d4e128dba1 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:44:55 +0300 Subject: [PATCH 09/26] feat(164-S4): add utils/add_reachable_pairs.py to derive CSV rows from an archive Accepting a contributed archive requires a reachable_pairs.csv row per new query (generate_registry reads the graph list and categories from the CSV, and merge_archive refuses to merge without the rows). Typing those rows by hand is error-prone, so the tool derives them from a validated archive: one row per queries/// directory, with the query directory name as the grammar stem (representation picked cnf > re > mcfg > rsm, the alphabetically first file per extension) and the results.mtx declared entry count as num_reachable_pairs. The archive must pass the structure checker first, a tarball must be named after the graph, and the CSV's (graph, grammar) sort order is preserved; duplicate rows and category disagreements are reported without writing. --dry-run prints instead of writing. --- docs/utils.rst | 23 ++ tests/utils/test_add_reachable_pairs.py | 223 ++++++++++++++++++ utils/add_reachable_pairs.py | 291 ++++++++++++++++++++++++ 3 files changed, 537 insertions(+) create mode 100644 tests/utils/test_add_reachable_pairs.py create mode 100644 utils/add_reachable_pairs.py diff --git a/docs/utils.rst b/docs/utils.rst index 8f497e4c..87033ff9 100644 --- a/docs/utils.rst +++ b/docs/utils.rst @@ -187,6 +187,29 @@ step with the archives; an unregistered query aborts the merge. The output file must be named after the graph (``.tar.gz``): the upload tool stores it under that name, replacing the previous archive. +.. _add_reachable_pairs: + +Add reachable-pairs rows +------------------------ + +``utils/add_reachable_pairs.py`` derives the reachable-pairs rows of one +validated archive and inserts them into +``flpq_data/dataset/reachable_pairs.csv``:: + + python utils/add_reachable_pairs.py ARCHIVE.tar.gz --graph NAME \ + --category CAT [--partial] [--dry-run] + +For every ``queries///`` directory the tool emits one row: +the query directory name as the stem of the ``grammar`` file (the +representation is picked in the order ``cnf > re > mcfg > rsm``) and the +declared entry count of the query's ``results.mtx`` as +``num_reachable_pairs``. The archive must pass the structure checker first +(full mode, or ``--partial`` for a partial archive), and a tarball must be +named after the graph. The CSV's ``(graph, grammar)`` sort order is +preserved; a row that already exists, or a category that disagrees with the +graph's existing rows, is reported and nothing is written. ``--dry-run`` +prints the rows instead of writing them. + .. _reachable_pairs_tables: Reachable pair counts diff --git a/tests/utils/test_add_reachable_pairs.py b/tests/utils/test_add_reachable_pairs.py new file mode 100644 index 00000000..ccb9fa02 --- /dev/null +++ b/tests/utils/test_add_reachable_pairs.py @@ -0,0 +1,223 @@ +import csv +import pathlib + +import add_reachable_pairs +from add_reachable_pairs import add_rows, derive_rows, main, representation_file +from test_check_archive_structure import _partial_tree, _tarball, _valid_tree + +HEADER = ["graph", "grammar", "category", "query_class", "num_reachable_pairs"] + + +def _write_csv(path: pathlib.Path, rows: list[list[str]]) -> None: + with path.open("w", newline="", encoding="utf-8") as f: + writer = csv.writer(f) + writer.writerow(HEADER) + writer.writerows(rows) + + +def _read_rows(path: pathlib.Path) -> list[list[str]]: + with path.open(newline="", encoding="utf-8") as f: + reader = csv.reader(f) + next(reader) + return [row for row in reader] + + +def test_derive_rows_full_archive(tmp_path): + tree = _valid_tree(tmp_path) + rows = derive_rows(tree, "g", "cat") + assert rows == [ + { + "graph": "g", + "grammar": "s.cnf", + "category": "cat", + "query_class": "cfpq", + "num_reachable_pairs": "1", + } + ] + + +def test_derive_rows_partial_archive(tmp_path): + tree = _partial_tree(tmp_path) + rows = derive_rows(tree, "g", "cat") + assert rows == [ + { + "graph": "g", + "grammar": "s.cnf", + "category": "cat", + "query_class": "cfpq", + "num_reachable_pairs": "1", + } + ] + + +def test_representation_file_prefers_cnf_over_rsm(tmp_path): + d = tmp_path / "q" + d.mkdir() + (d / "q.rsm").write_text("S -> a*") + (d / "q.cnf").write_text("S\ta\tN1\nN1\ta\n\nCount:\nS") + rep = representation_file(d) + assert rep is not None + assert rep.name == "q.cnf" + + +def test_representation_file_falls_back_to_rsm(tmp_path): + d = tmp_path / "q" + d.mkdir() + (d / "q.rsm").write_text("S -> a*") + rep = representation_file(d) + assert rep is not None + assert rep.name == "q.rsm" + + +def test_representation_file_none_without_representations(tmp_path): + d = tmp_path / "q" + d.mkdir() + (d / "results.mtx").write_text("x") + assert representation_file(d) is None + + +def test_add_rows_inserts_sorted(tmp_path): + csv_path = tmp_path / "r.csv" + _write_csv( + csv_path, + [ + ["b", "q.cnf", "cat", "cfpq", "1"], + ["d", "q.cnf", "cat", "cfpq", "2"], + ], + ) + row = { + "graph": "c", + "grammar": "q.cnf", + "category": "cat", + "query_class": "cfpq", + "num_reachable_pairs": "3", + } + problems = add_rows(csv_path, [row]) + assert problems == [] + assert _read_rows(csv_path) == [ + ["b", "q.cnf", "cat", "cfpq", "1"], + ["c", "q.cnf", "cat", "cfpq", "3"], + ["d", "q.cnf", "cat", "cfpq", "2"], + ] + + +def test_add_rows_refuses_duplicate(tmp_path): + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, [["g", "s.cnf", "cat", "cfpq", "1"]]) + row = { + "graph": "g", + "grammar": "s.rsm", + "category": "cat", + "query_class": "cfpq", + "num_reachable_pairs": "9", + } + problems = add_rows(csv_path, [row]) + assert len(problems) == 1 + assert "already exists" in problems[0] + assert _read_rows(csv_path) == [["g", "s.cnf", "cat", "cfpq", "1"]] + + +def test_add_rows_refuses_category_mismatch(tmp_path): + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, [["g", "other.cnf", "old_cat", "cfpq", "1"]]) + row = { + "graph": "g", + "grammar": "s.cnf", + "category": "new_cat", + "query_class": "cfpq", + "num_reachable_pairs": "2", + } + problems = add_rows(csv_path, [row]) + assert len(problems) == 1 + assert "disagrees" in problems[0] + assert _read_rows(csv_path) == [["g", "other.cnf", "old_cat", "cfpq", "1"]] + + +def test_main_full_archive(tmp_path, monkeypatch): + tree = _valid_tree(tmp_path) + tarball = _tarball(tree, tmp_path / "g.tar.gz") + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, []) + monkeypatch.setattr(add_reachable_pairs, "REACHABLE_PAIRS_CSV", csv_path) + status = main([str(tarball), "--graph", "g", "--category", "cat"]) + assert status == 0 + assert _read_rows(csv_path) == [["g", "s.cnf", "cat", "cfpq", "1"]] + + +def test_main_partial_archive(tmp_path, monkeypatch): + tree = _partial_tree(tmp_path) + tarball = _tarball(tree, tmp_path / "g.tar.gz") + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, [["g", "old.cnf", "cat", "cfpq", "5"]]) + monkeypatch.setattr(add_reachable_pairs, "REACHABLE_PAIRS_CSV", csv_path) + status = main([str(tarball), "--graph", "g", "--category", "cat", "--partial"]) + assert status == 0 + assert _read_rows(csv_path) == [ + ["g", "old.cnf", "cat", "cfpq", "5"], + ["g", "s.cnf", "cat", "cfpq", "1"], + ] + + +def test_main_refuses_invalid_archive(tmp_path, monkeypatch): + tree = _partial_tree(tmp_path) + tarball = _tarball(tree, tmp_path / "g.tar.gz") + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, []) + monkeypatch.setattr(add_reachable_pairs, "REACHABLE_PAIRS_CSV", csv_path) + status = main([str(tarball), "--graph", "g", "--category", "cat"]) + assert status == 1 + assert _read_rows(csv_path) == [] + + +def test_main_refuses_graph_name_mismatch(tmp_path, monkeypatch): + tree = _valid_tree(tmp_path) + tarball = _tarball(tree, tmp_path / "g.tar.gz") + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, []) + monkeypatch.setattr(add_reachable_pairs, "REACHABLE_PAIRS_CSV", csv_path) + status = main([str(tarball), "--graph", "other", "--category", "cat"]) + assert status == 1 + assert _read_rows(csv_path) == [] + + +def test_main_dry_run_writes_nothing(tmp_path, monkeypatch): + tree = _valid_tree(tmp_path) + tarball = _tarball(tree, tmp_path / "g.tar.gz") + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, []) + monkeypatch.setattr(add_reachable_pairs, "REACHABLE_PAIRS_CSV", csv_path) + status = main([str(tarball), "--graph", "g", "--category", "cat", "--dry-run"]) + assert status == 0 + assert _read_rows(csv_path) == [] + + +def test_main_refuses_archive_not_named_after_graph(tmp_path, monkeypatch): + tree = _valid_tree(tmp_path) + tarball = _tarball(tree, tmp_path / "other.tar.gz") + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, []) + monkeypatch.setattr(add_reachable_pairs, "REACHABLE_PAIRS_CSV", csv_path) + status = main([str(tarball), "--graph", "g", "--category", "cat"]) + assert status == 1 + assert _read_rows(csv_path) == [] + + +def test_main_refuses_non_tar_gz(tmp_path, monkeypatch): + not_an_archive = tmp_path / "g.tar.gz" + not_an_archive.write_text("plain text") + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, []) + monkeypatch.setattr(add_reachable_pairs, "REACHABLE_PAIRS_CSV", csv_path) + status = main([str(not_an_archive), "--graph", "g", "--category", "cat"]) + assert status == 1 + assert _read_rows(csv_path) == [] + + +def test_main_refuses_existing_row(tmp_path, monkeypatch): + tree = _valid_tree(tmp_path) + tarball = _tarball(tree, tmp_path / "g.tar.gz") + csv_path = tmp_path / "r.csv" + _write_csv(csv_path, [["g", "s.cnf", "cat", "cfpq", "1"]]) + monkeypatch.setattr(add_reachable_pairs, "REACHABLE_PAIRS_CSV", csv_path) + status = main([str(tarball), "--graph", "g", "--category", "cat"]) + assert status == 1 diff --git a/utils/add_reachable_pairs.py b/utils/add_reachable_pairs.py new file mode 100644 index 00000000..3b6f9914 --- /dev/null +++ b/utils/add_reachable_pairs.py @@ -0,0 +1,291 @@ +"""Add reachable-pairs rows for the queries of one archive. + +``flpq_data/dataset/reachable_pairs.csv`` is the single source of truth +behind the registry and the site's applicable-graphs renderings: every +graph x query pair needs a row. This tool derives those rows from a +validated archive instead of typing them by hand: for every +``queries///`` directory it emits one row with the query +directory name as the stem of the ``grammar`` file (the representation +file is picked in the order ``cnf > re > mcfg > rsm``) and the declared +entry count of the query's ``results.mtx`` as ``num_reachable_pairs``. + +The archive must pass the structure checker first (full mode, or +``--partial`` for a partial archive); any structure problem aborts the +tool. Rows are inserted into the CSV preserving its ``(graph, grammar)`` +sort order; a row that already exists, or a ``--category`` that disagrees +with the graph's existing rows, is reported and nothing is written. + +Usage (from any directory):: + + python utils/add_reachable_pairs.py ARCHIVE.tar.gz --graph NAME \ + --category CAT [--partial] [--dry-run] +""" + +import argparse +import csv +import pathlib +import tarfile +import tempfile +from typing import Optional, Sequence + +from check_archive_structure import ( + QUERY_CLASSES, + _mtx_header, + unpack_single_dir, + validate_archive, +) +from config import MAIN_FOLDER + +__all__ = [ + "REACHABLE_PAIRS_CSV", + "REPRESENTATION_PRIORITY", + "representation_file", + "derive_rows", + "add_rows", + "main", +] + +REACHABLE_PAIRS_CSV = MAIN_FOLDER / "flpq_data" / "dataset" / "reachable_pairs.csv" + +#: The order in which a query directory's representation files are +#: preferred for the row's ``grammar`` file (all current rows are ``.cnf``). +REPRESENTATION_PRIORITY = ("cnf", "re", "mcfg", "rsm") + + +def representation_file(query_dir: pathlib.Path) -> Optional[pathlib.Path]: + """Return the query directory's preferred representation file. + + Parameters + ---------- + query_dir : Path + A ``queries///`` directory. + + Examples + -------- + >>> import pathlib, tempfile + >>> with tempfile.TemporaryDirectory() as tmp: + ... d = pathlib.Path(tmp) / "q" + ... d.mkdir() + ... (d / "q.rsm").write_text("x") + ... (d / "q.cnf").write_text("x") + ... representation_file(d).name + 'q.cnf' + + Returns + ------- + file : Path or None + The alphabetically first file whose extension is in + :data:`REPRESENTATION_PRIORITY` (a query may hold two files of one + extension), or None when the directory holds no representation + file. + """ + by_ext: dict[str, pathlib.Path] = {} + for path in sorted(query_dir.iterdir(), key=lambda p: p.name): + if path.is_file(): + by_ext.setdefault(path.suffix.lstrip("."), path) + for ext in REPRESENTATION_PRIORITY: + if ext in by_ext: + return by_ext[ext] + return None + + +def derive_rows(root: pathlib.Path, graph: str, category: str) -> list[dict[str, str]]: + """Derive the reachable-pairs rows of one validated archive. + + Parameters + ---------- + root : Path + The unpacked archive directory (it must have passed + :func:`check_archive_structure.validate_archive`, so every query + directory holds a representation file and a parseable + ``results.mtx``). + graph : str + The graph name the rows are recorded under. + category : str + The graph's category. + + Returns + ------- + rows : list[dict[str, str]] + One row per query directory, with the CSV's five fields as string + values. + + Raises + ------ + ValueError + If a query directory lacks a representation file or a parseable + ``results.mtx`` (only possible for an unvalidated tree). + """ + rows: list[dict[str, str]] = [] + queries_dir = root / "queries" + for cls in sorted(QUERY_CLASSES): + cls_dir = queries_dir / cls + if not cls_dir.is_dir(): + continue + for query_dir in sorted(cls_dir.iterdir()): + if not query_dir.is_dir(): + continue + rep = representation_file(query_dir) + if rep is None: + raise ValueError( + f"queries/{cls}/{query_dir.name}: no representation file" + ) + header = _mtx_header(query_dir / "results.mtx") + if header is None: + raise ValueError( + f"queries/{cls}/{query_dir.name}/results.mtx: cannot be parsed" + ) + rows.append( + { + "graph": graph, + "grammar": rep.name, + "category": category, + "query_class": cls, + "num_reachable_pairs": str(header[2]), + } + ) + return rows + + +def add_rows(csv_path: pathlib.Path, rows: list[dict[str, str]]) -> list[str]: + """Insert ``rows`` into the reachable-pairs CSV. + + The file's ``(graph, grammar)`` sort order is preserved. Nothing is + written when any problem is found. + + Parameters + ---------- + csv_path : Path + The reachable_pairs.csv file to update in place. + rows : list[dict[str, str]] + The rows to add (the CSV's five fields as string values). + + Returns + ------- + problems : list[str] + Every duplicate row and category disagreement found; an empty list + means the rows were written. + """ + with csv_path.open(newline="", encoding="utf-8") as f: + reader = csv.reader(f) + header = next(reader) + existing = [row for row in reader] + + problems: list[str] = [] + keys = {(row[0], row[3], pathlib.Path(row[1]).stem) for row in existing} + categories = {row[0]: row[2] for row in existing} + for row in rows: + key = (row["graph"], row["query_class"], pathlib.Path(row["grammar"]).stem) + if key in keys: + problems.append( + f"{row['graph']},{row['grammar']}: row already exists in the CSV" + ) + elif row["graph"] in categories and categories[row["graph"]] != row["category"]: + problems.append( + f"{row['graph']}: category {row['category']!r} disagrees with " + f"the existing rows' {categories[row['graph']]!r}" + ) + if problems: + return problems + + merged = sorted( + existing + [list(row.values()) for row in rows], key=lambda r: (r[0], r[1]) + ) + with csv_path.open("w", newline="", encoding="utf-8") as f: + writer = csv.writer(f) + writer.writerow(header) + writer.writerows(merged) + return [] + + +def main(argv: Optional[Sequence[str]] = None) -> int: + """Command-line entry point. + + Validates one archive, derives its reachable-pairs rows, and inserts + them into ``flpq_data/dataset/reachable_pairs.csv`` (or prints them + with ``--dry-run``). + + Returns + ------- + status : int + 0 when the rows were added (or printed), 1 otherwise. + """ + parser = argparse.ArgumentParser( + description=( + "Add the reachable-pairs rows of one validated archive to " + "flpq_data/dataset/reachable_pairs.csv." + ) + ) + parser.add_argument( + "path", help="path to a .tar.gz archive or an unpacked archive directory" + ) + parser.add_argument( + "--graph", required=True, help="the graph name the rows are recorded under" + ) + parser.add_argument("--category", required=True, help="the graph's category") + parser.add_argument( + "--partial", + action="store_true", + help="treat the archive as a partial one (new queries for an existing graph)", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="print the rows instead of writing the CSV", + ) + args = parser.parse_args(argv) + + path = pathlib.Path(args.path) + if path.is_dir(): + return _run(path, args) + if path.suffix != ".gz" or not tarfile.is_tarfile(path): + print(f"error: {path.name}: not a .tar.gz archive") + return 1 + if path.name[: -len(".tar.gz")] != args.graph: + print(f"error: {path.name} must be named after the graph ({args.graph}.tar.gz)") + return 1 + with tempfile.TemporaryDirectory() as tmp: + try: + root = unpack_single_dir(path, pathlib.Path(tmp)) + except ValueError as error: + print(f"error: {error}") + return 1 + return _run(root, args) + + +def _run(root: pathlib.Path, args: argparse.Namespace) -> int: + """Validate ``root`` and add (or print) its rows.""" + if root.name != args.graph: + print( + f"error: top-level directory {root.name!r} " + f"does not match --graph {args.graph!r}" + ) + return 1 + + problems = validate_archive(root, partial=args.partial) + if problems: + for problem in problems: + print(f"error: {problem}") + return 1 + + rows = derive_rows(root, args.graph, args.category) + if args.dry_run: + for row in rows: + print(",".join(row.values())) + return 0 + + problems = add_rows(REACHABLE_PAIRS_CSV, rows) + if problems: + for problem in problems: + print(f"error: {problem}") + return 1 + for row in rows: + print( + f"added {row['graph']},{row['grammar']} ({row['query_class']}, " + f"{row['num_reachable_pairs']} pairs)" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 9a0f874f4a3d2790ff30adf9cf61e6a52e4f6edd Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:45:20 +0300 Subject: [PATCH 10/26] docs(164): mark S4 done in the detailed plan --- tasks/detailed_plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index 0dcd3557..c20dd97e 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -132,7 +132,7 @@ do not duplicate). - Note that a set of graphs for a category goes into one issue as multiple archives. -### S4: `utils/add_reachable_pairs.py` — derive CSV rows from an archive +### S4: `utils/add_reachable_pairs.py` — derive CSV rows from an archive [done] 23f0011 **Code:** New file `utils/add_reachable_pairs.py`. Reuse from `check_archive_structure.py`: `unpack_single_dir`, `_mtx_header`, From 8e98011902cf1a8a18ad934a729dbd387ab3c194 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:48:03 +0300 Subject: [PATCH 11/26] docs(164-S5): document the maintainer acceptance pipeline in the skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The add-graph skill gains an 'Accepting an external contribution' section: per archive — download from the Drive link, verify the SHA256 against the issue, re-run the structure checker, check name uniqueness against the registry and the reachable-pairs CSV, derive the CSV rows with utils/add_reachable_pairs.py, then upload (full) or merge (partial); once per issue — regenerate the registry, do the docs wiring, and refresh the reachable-pairs tables and archive sizes. Stale 'PR template' references in both skills point at the issue template now, and add-grammar records that grammars arrive via cases 2/3/4b of a CONTRIBUTE NEW DATA issue. --- .opencode/skills/add-grammar/SKILL.md | 12 ++++++- .opencode/skills/add-graph/SKILL.md | 45 +++++++++++++++++++++------ 2 files changed, 47 insertions(+), 10 deletions(-) diff --git a/.opencode/skills/add-grammar/SKILL.md b/.opencode/skills/add-grammar/SKILL.md index e15905b3..9549a82e 100644 --- a/.opencode/skills/add-grammar/SKILL.md +++ b/.opencode/skills/add-grammar/SKILL.md @@ -1,6 +1,6 @@ --- name: add-grammar -description: Use when adding a new grammar template to FLPQ_Data. Covers the generator/readwrite/converter module conventions and the PR/issue templates. +description: Use when adding a new grammar template to FLPQ_Data. Covers the generator/readwrite/converter module conventions and the issue template. --- # Add a grammar template @@ -15,6 +15,16 @@ The required fields and metadata live in the GitHub issue template — do Fill in every field in the triangle brackets (`<>`) of that template. +## Where grammars come from + +Grammars arrive via cases 2, 3, or 4b of a CONTRIBUTE NEW DATA issue: the +issue carries the grammar definition (LaTeX plus file-format text), and the +same grammar is instantiated as at least one query directory with +`results.mtx` in a contributed archive. When the grammar is parameterized, +implement its generator module under `flpq_data/queries//generators/` +per the package conventions below; the docs wiring (category page, table +column) follows the "Documentation" section of the `add-graph` skill. + ## Package conventions Query code lives under `flpq_data/queries//` (`cfpq`, `rpq`, `mcfpq`): diff --git a/.opencode/skills/add-graph/SKILL.md b/.opencode/skills/add-graph/SKILL.md index aa85f8ab..c0e0844e 100644 --- a/.opencode/skills/add-graph/SKILL.md +++ b/.opencode/skills/add-graph/SKILL.md @@ -1,6 +1,6 @@ --- name: add-graph -description: Use when adding a new graph to FLPQ_Data. Covers the mtx-per-label archive format, the registry.json wiring, and the PR/issue templates. +description: Use when adding a new graph to FLPQ_Data. Covers the mtx-per-label archive format, accepting external contributions (the issue-first pipeline), and the registry.json wiring. --- # Add a graph to the dataset @@ -19,16 +19,43 @@ Fill in every field in the triangle brackets (`<>`) of that template. The archive layout and the label conventions are documented once in the "File structure" section of `docs/graphs/index.rst` — follow it exactly. -Pack every query of the graph into the archive per the "Queries" section of -the PR template, then validate with +Pack every query of the graph into the archive per the "Queries and +grammars" section of the issue template, then validate with `python utils/check_archive_structure.py .tar.gz` (the upload tool runs the same check and refuses invalid archives). New queries for an existing graph ship as a **partial archive** instead — -see the "Data format" section of the PR template; it is validated with +see the "Data format" section of the issue template; it is validated with `--partial` and merged into the existing archive with `python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`. +## Accepting an external contribution + +Data arrives as a CONTRIBUTE NEW DATA issue carrying one or more archives. +Per archive, in order: + +1. Download it from the Drive link in the issue. +2. Verify its SHA256 against the value in the issue — a mismatch means the + shared file is not the validated one; stop and ask the contributor. +3. Re-run `python utils/check_archive_structure.py .tar.gz` (add + `--partial` for partial archives) — it must pass. +4. Check the graph name against `flpq_data/dataset/registry.json` and + `flpq_data/dataset/reachable_pairs.csv`: a full archive's name must be + new, a partial archive's target must exist. +5. Add its reachable-pairs rows with + `python utils/add_reachable_pairs.py .tar.gz --graph NAME --category CAT [--partial]` + (it re-validates the archive and refuses duplicates). +6. Put it in place: a full archive is uploaded to the dataset URL (see + "Wiring a new graph into the code"); a partial archive is merged with + `python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz` + and the result uploaded under the graph's name. + +Once per issue, after every archive is in: regenerate the registry +(`python utils/generate_registry.py`), do the "Documentation" steps (per-graph +pages, category table(s), canonical grammars, new category page), then run +`python utils/reachable_pairs_tables.py --update` and +`python utils/archive_sizes.py --update`. + ## Wiring a new graph into the code 1. The graph archive must be uploaded under the dataset URL @@ -44,11 +71,11 @@ see the "Data format" section of the PR template; it is validated with 1. Create the per-graph page `docs/graphs/data/.rst`, following the structure of an existing page (description, statistics tables). The page - carries **no** grammar section: canonical grammars are documented once on - the category page. Statistics conventions: the PR template is the source - of truth. -2. Handle the grammar case from the PR template's "Canonical grammars" - section: + carries **no** grammar section: canonical grammars are documented once on + the category page. Statistics conventions: the issue template is the + source of truth. +2. Handle the grammar case from the issue template's "Queries and grammars" + section: - *existing category, existing grammar(s)* — nothing to add; the new table row fills in the matching column(s); - *new grammar for an existing category* — append the grammar to the From e824a7e9893896700083da0b4fc1a49c2da86b50 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:48:21 +0300 Subject: [PATCH 12/26] docs(164): mark S5 done in the detailed plan --- tasks/detailed_plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index c20dd97e..0f427810 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -165,7 +165,7 @@ docstring stays the source of truth for details. - Exit non-zero on any problem (invalid archive, duplicate row, missing representation file), printing each problem. -### S5: Maintainer acceptance pipeline in the skills +### S5: Maintainer acceptance pipeline in the skills [done] 8e98011 **Code:** Extend `.opencode/skills/add-graph/SKILL.md` with an "Accepting an external contribution" section; update `.opencode/skills/add-grammar/SKILL.md` From 0f20cd101af8be2ebe21ca40863ca4d7ac093519 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:49:18 +0300 Subject: [PATCH 13/26] docs(164-S6): record the contribution-flow change in the changelog Issue-first data contributions (unified CONTRIBUTE NEW DATA template, multi-archive issues, motivation and reference-results requirements), the new utils/add_reachable_pairs.py, and the removal of the specialized graph and grammar templates are recorded under Unreleased. Closes #164 --- CHANGELOG.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ed9ee26..7be9a46e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- `utils/add_reachable_pairs.py`: derives the reachable-pairs rows of one + validated archive (one row per query directory, with the declared entry + count of its `results.mtx` as `num_reachable_pairs`) and inserts them into + `flpq_data/dataset/reachable_pairs.csv`, preserving the file's sort order + and refusing duplicate rows and category disagreements. + +### Changed + +- Data contributions are issue-first: a single "CONTRIBUTE NEW DATA" issue + carries one or more validated archives (one graph each), the per-archive + validation output and SHA256, and the motivation; maintainers upload and + merge the data into the dataset. The Graphs page documents the flow, the + four contribution cases, and the consistency invariants — no graph without + a query, no query without reference results, no item without description + and motivation. + +### Removed + +- The `ADD NEW GRAPH` / `ADD NEW GRAMMAR` issue templates and the + `new_graph` / `new_grammar` PR templates: data has no PR path, and a + grammar is accepted only together with the archive that instantiates it as + a query with reference results. A generic PR template covers code + contributions. + ## [6.0.0] - 2026-10-13 ### Added From 2fcbde1cbd1047a9ae3adc23b868edab836b989d Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 14:49:44 +0300 Subject: [PATCH 14/26] docs(164): mark S6 done in the detailed plan --- tasks/detailed_plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index 0f427810..4894135b 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -190,7 +190,7 @@ docstrings or template fields. (cases 2/3/4b of the CONTRIBUTE NEW DATA issue); pointer to the template; the generator-module conventions stay as-is. -### S6: Changelog +### S6: Changelog [done] 0f20cd1 **Code:** none. **Tests:** none. From 6ea48db4578a1b532f836189d580299d26fa6bdf Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 15:23:16 +0300 Subject: [PATCH 15/26] docs(164-S3): point the contribution guide at the template for cases and declined cases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Code review found the four-case list and the declined-cases list duplicated near-verbatim between the issue template and the docs guide. Per the single-source-of-truth convention (templates own field and case content), the guide now summarizes both in one sentence each and points at the template's 'Contribution type' and 'Not accepted via this flow' sections; the guide keeps what is uniquely its own — the flow, the validation command, and the consistency invariants. --- docs/graphs/index.rst | 30 ++++++++++-------------------- 1 file changed, 10 insertions(+), 20 deletions(-) diff --git a/docs/graphs/index.rst b/docs/graphs/index.rst index 6b3c9c12..cc6ac661 100644 --- a/docs/graphs/index.rst +++ b/docs/graphs/index.rst @@ -25,19 +25,12 @@ request is needed. Contribution cases ^^^^^^^^^^^^^^^^^^ -1. **New graph(s), existing category, existing grammar(s)** — full - archive(s); the queries instantiate the category's canonical grammar(s). -2. **New graph(s), existing category, new grammar** — full archive(s) plus a - grammar definition; the grammar gets a new column in the category table. -3. **New graph(s), new category** — full archive(s) plus the category - name/description and its canonical grammar(s); a new category page is - created. -4. **New queries for existing graph(s)** — partial archive(s); sub-case 4b - if a query uses a new grammar. - -A set of graphs for a category goes into one issue as several archives. The -issue template lists every required field per case (statistics, grammar -definitions, category description). +The issue template's "Contribution type" section lists the four contribution +cases — new graph(s) for an existing or a new category, with an existing or +a new grammar, and new queries for existing graph(s) — and what each +requires; its remaining sections carry every required field (statistics, +grammar definitions, category description). A set of graphs for a category +goes into one issue as several archives (one graph per archive). Validation ^^^^^^^^^^ @@ -64,13 +57,10 @@ Consistency invariants Not accepted via this flow ^^^^^^^^^^^^^^^^^^^^^^^^^^ -- A grammar or query provided outside an archive — without reference - results. -- Updates to an existing graph (a new version of the source dataset). -- A new representation of an existing query (e.g. adding ``.rsm`` next to a - ``.cnf``-only query). -- More than one graph inside a single archive (use several archives in one - issue instead). +The issue template's "Not accepted via this flow" section lists the declined +cases: a grammar or query provided outside an archive (without reference +results), updates to an existing graph, a new representation of an existing +query, and more than one graph inside a single archive. .. _graph_file_structure: From a94a890d03fa659da02f113981e5ae7b46894be3 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 15:26:02 +0300 Subject: [PATCH 16/26] docs(164-S1): state the no-graph-without-query rule once in the template Code review found the grammar-instantiation rule stated twice inside the template (Reference results and Queries and grammars case 2). The Reference results section now keeps only the graph-side rule, noting that the structure checker enforces it; the grammar-side rule stays with case 2. --- .github/ISSUE_TEMPLATE/new-data-template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/new-data-template.md b/.github/ISSUE_TEMPLATE/new-data-template.md index 71cb7b79..e433b3fe 100644 --- a/.github/ISSUE_TEMPLATE/new-data-template.md +++ b/.github/ISSUE_TEMPLATE/new-data-template.md @@ -54,7 +54,7 @@ FLPQ_Data version used: ```` ## Reference results (mandatory) -Every query directory ships one `results.mtx` with the constrained reachability facts — a Boolean pattern matrix with the graph's dimensions, where an entry `(i, j)` means some path from node `i` to node `j` satisfies the query. **Queries without reference results are not accepted.** Every new graph must ship at least one query; every new grammar must be instantiated as at least one such query in a contributed archive. +Every query directory ships one `results.mtx` with the constrained reachability facts — a Boolean pattern matrix with the graph's dimensions, where an entry `(i, j)` means some path from node `i` to node `j` satisfies the query. **Queries without reference results are not accepted.** Every new graph must ship at least one query (the structure checker refuses archives without one). ## Data format From f0cff39cf5502901fb38a479bdb2f605fe1463d5 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 15:30:09 +0300 Subject: [PATCH 17/26] docs(164-S3): link the new issue template on dev, not master The CONTRIBUTE NEW DATA template does not exist on master until a release lands, so blob/master links to it would fail the CI link check on dev. The links now point at blob/dev, where the file exists and which is also the version contributors should follow; the links keep resolving after release. --- README.rst | 2 +- docs/dataset.rst | 2 +- docs/graphs/index.rst | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index b63afd7c..5ba65a39 100644 --- a/README.rst +++ b/README.rst @@ -88,7 +88,7 @@ contribution guidelines — is on the `Developer `_ — one or more validated archives (one graph each), links to them, and the motivation. The maintainers upload and merge the data into the dataset; the full contribution guide is on the `Graphs `_ page. +Create an ``Issue`` corresponding to the `"CONTRIBUTE NEW DATA" issue template `_ — one or more validated archives (one graph each), links to them, and the motivation. The maintainers upload and merge the data into the dataset; the full contribution guide is on the `Graphs `_ page. Citation ******** diff --git a/docs/dataset.rst b/docs/dataset.rst index f013c4a6..2a372c9b 100644 --- a/docs/dataset.rst +++ b/docs/dataset.rst @@ -26,4 +26,4 @@ list of applicable graphs. How to add a new grammar? ------------------------- -Grammars are added together with the data they apply to — create an ``Issue`` corresponding to the `"CONTRIBUTE NEW DATA" issue template `_ (the "Queries and grammars" section). A grammar is accepted only if it is instantiated as at least one query with reference results in a contributed archive. +Grammars are added together with the data they apply to — create an ``Issue`` corresponding to the `"CONTRIBUTE NEW DATA" issue template `_ (the "Queries and grammars" section). A grammar is accepted only if it is instantiated as at least one query with reference results in a contributed archive. diff --git a/docs/graphs/index.rst b/docs/graphs/index.rst index cc6ac661..3773bcef 100644 --- a/docs/graphs/index.rst +++ b/docs/graphs/index.rst @@ -15,7 +15,7 @@ Contributing data ----------------- New data — graphs, queries, grammars — enters the dataset through a single -`"CONTRIBUTE NEW DATA" issue `_: +`"CONTRIBUTE NEW DATA" issue `_: you prepare one or more archives (one graph each), validate them, publish them (e.g. to Google Drive), and open the issue with the links, the validation output, the SHA256 of every archive, and the motivation. The From b04acdc1b45489a35249a1661381b77ca03a6445 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 17:16:23 +0300 Subject: [PATCH 18/26] docs(165): write the detailed plan for packaging the archive checker --- tasks/detailed_plan.md | 314 ++++++++++++++++------------------------- 1 file changed, 123 insertions(+), 191 deletions(-) diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index 4894135b..43e4bb0a 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -1,207 +1,139 @@ -# Detailed Plan: Task 164 — External data contribution flow (issue-first) +# Detailed Plan: Task 165 — Package the archive checker as `flpq-check-archive` -Issue #164. Redesign the external contribution path so new data enters via a -single issue carrying one or more validated archives (one graph each), with -mandatory motivation and explicit consistency invariants; delete the -standalone grammar path; document the full case matrix; add the -maintainer-side pipeline tooling. +Issue #165. Move the archive structure checker from `utils/` into the +`flpq_data` package and expose it as a console entry point, so external +contributors validate archives with `pip install flpq-data` alone — no +manual script download. ## Confirmed decisions (user guidance, verbatim on the issue) -- One issue may contain **multiple archives** (a set of graphs/queries for a - category); each archive still unpacks to a single directory named after it - (one graph per archive — the fixed structure is unchanged). -- **Delete both** specialized PR templates; replace with one generic PR - template for code contributions. -- **Decline for now**: updates to an existing graph; new representation of an - existing query. Both are documented as not accepted via this flow. -- **Add the utility** deriving `reachable_pairs.csv` rows from an archive. -- Explicit invariants: every new graph ships at least one query; every query - ships reference results (`results.mtx`); no item without description and - motivation. A grammar is a representation of a query — a grammar definition - is accepted only if the same grammar is instantiated as a query directory - with `results.mtx` in a contributed archive. - -## Contribution cases (the matrix) - -| # | Case | Archive(s) | What's new | -|---|------|-----------|-----------| -| 1 | New graph(s), existing category, existing canonical grammar(s) | full | graphs + query instances | -| 2 | New graph(s), existing category, **new grammar** | full | + grammar: new column in the category table, possibly a generator module | -| 3 | New graph(s), **new category**, canonical grammar(s) | full | + category page | -| 4 | **Existing graph(s)**, new queries — any class; sub-case 4b: queries using a **new grammar** | partial | query dirs; + grammar definition if 4b | - -Not accepted via this flow: a grammar/query provided outside an archive (no -reference results); updates to an existing graph; a new representation of an -existing query; more than one graph inside a single archive. +- Scope: **checker only**. The other utils are repo-bound maintainer tools + (S3 credentials, in-repo CSV/docs paths) that would fail outside a + checkout. +- Entry point name: **`flpq-check-archive`** (e.g. + `flpq-check-archive my_graph.tar.gz --partial`). + +## Design notes + +- The checker imports only from `flpq_data.*` + `pyformlang` — no + repo-bound paths — so it moves cleanly into the package. +- boto3 is a dev-only dependency; the `--audit` subcommand (S3 audit) is + maintainer-side. `create_s3_client` + `DEFAULT_BUCKET` / + `DEFAULT_ENDPOINT_URL` move to `flpq_data/dataset/s3.py` with boto3 + imported lazily inside the function, so the package imports without + boto3; `--audit` reports a clean "boto3 required" error when it is + missing. +- The wheel build (`packages = ["flpq_data"]`) picks up the new submodule + automatically; `[project.scripts]` registers the entry point; uv installs + the project itself, so the command works in-repo too. +- Coverage: the moved lines now count toward the 95% flpq_data gate; the + existing tests cover the module (incl. mock-tested `_audit`); one new + branch (missing boto3) gets a test. --- -### S1: Unified "CONTRIBUTE NEW DATA" issue template [done] 4d1c42c - -**Code:** Restructure `.github/ISSUE_TEMPLATE/graph-add-template.md` into -`.github/ISSUE_TEMPLATE/new-data-template.md` (rename + rewrite); delete -`.github/ISSUE_TEMPLATE/grammar-add-template.md`. Reuse the existing -template's field definitions and wording where they survive (Info table, -description-document pointer, statistics conventions, data-format bullets) — -restructure, do not reinvent. - -**Tests:** none (no `.py` files changed). - -**Docs:** the template is the source of truth for the required fields; no -other doc changes in this subtask. - -**Spec:** -- Frontmatter: `name: CONTRIBUTE NEW DATA`, `about: Suggest new data - (graphs, queries, grammars) to this project`, `title: "[NEW DATA] "`, `labels: new data`, `assignees: rustam-azimov, vadyushkins`. -- **Contribution type** — one of the four cases above; an issue may combine - several archives of the same case. -- **Info table** — Version (current FLPQ_Data version); Category (existing - name or `new: `; N/A for case 4). -- **Archives** — one entry per archive (a set is allowed): Name, Kind - (full/partial), Target graph (partial only), Drive link to `.tar.gz`, - SHA256 of the archive. State that each archive unpacks to a single - directory named after it (one graph per archive). -- **Validation (mandatory, per archive)** — the exact command - (`python utils/check_archive_structure.py .tar.gz` or with - `--partial`) plus the pasted output and the FLPQ_Data version used. Include - the no-clone recipe: `pip install flpq-data`, download the single script - from GitHub raw, run it (all imports resolve from the installed package). -- **Motivation (mandatory)** — free text answering why this graph set / - grammar / category is useful to other users; must cover every new item. -- **Reference results (explicit requirement)** — every query directory must - contain `results.mtx`; queries without reference results are not accepted; - every new graph must ship at least one query; every new grammar must be - instantiated as at least one such query in a contributed archive. -- Keep: the data-format bullets (full vs partial layout), the - description-document pointer (the archive README answers the 8 mandatory - questions — not duplicated here), per-graph statistics subsections (new - graphs only: Graph Statistics + Edges Statistics tables with the existing - conventions), and the Queries section reorganized around the case matrix: - which canonical grammar(s) apply (case 1); grammar definition in LaTeX + - file format plus why no existing canonical grammar covers it (cases 2, 3, - 4b); category name + description (case 3). -- **Not accepted via this flow** — explicit list: a grammar/query provided - outside an archive (no reference results); updates to an existing graph; a - new representation of an existing query; more than one graph inside a - single archive. - -### S2: PR template cleanup [done] ba5d880 - -**Code:** Delete `.github/PULL_REQUEST_TEMPLATE/new_graph.md` and -`.github/PULL_REQUEST_TEMPLATE/new_grammar.md`; add a generic -`.github/PULL_REQUEST_TEMPLATE/pull_request.md`. - -**Tests:** none (no `.py` files changed). - -**Docs:** none. - -**Spec:** The generic template is short: a description of the change, the -linked issue (`Closes #`), and a checklist (quality gate passes; docs -updated per the documentation skill; no secrets). Data contributions do not -use PRs — they use the CONTRIBUTE NEW DATA issue (pointer line in the -template). - -### S3: External contribution guide in the docs [done] f9076cd - -**Code:** Rewrite the "How to add a new graph?" section of -`docs/graphs/index.rst` (currently *"Just create a PR"*) into a -"Contributing data" section. - -**Tests:** Sphinx build via the quality gate; `utils/check_math_snippets.py` -if any math is added (none planned). - -**Docs:** this section becomes the single source of truth for the flow and -the case analysis; field-level details stay in the template (cross-reference, -do not duplicate). - -**Spec:** -- Flow overview: build archive(s) → validate each locally → publish to Drive - → one issue with all archives (link + SHA256 + validation output + - motivation) → maintainers handle upload/merge/registry/docs. -- The no-clone validation recipe (`pip install flpq-data` + raw script URL). -- The four-case table with what each requires (mirror the matrix above; - field details point at the template). -- Explicit consistency invariants: no graph without at least one query, no - query without reference results, no item without description and - motivation. -- Declined cases with reasons (standalone grammar/query without reference - results; graph updates; new representation of an existing query; multiple - graphs per archive). -- Note that a set of graphs for a category goes into one issue as multiple - archives. - -### S4: `utils/add_reachable_pairs.py` — derive CSV rows from an archive [done] 23f0011 - -**Code:** New file `utils/add_reachable_pairs.py`. Reuse from -`check_archive_structure.py`: `unpack_single_dir`, `_mtx_header`, -`validate_archive` (and the `QUERY_CLASSES` mapping). No new parsing logic. - -**Tests:** New `tests/utils/test_add_reachable_pairs.py`. Reuse the tree -builders from `tests/utils/test_check_archive_structure.py` (`_valid_tree`, -`_partial_tree`, `_tarball`) by importing them from that module (same -directory; pytest prepends it to `sys.path`). - -**Docs:** numpydoc docstring with doctest examples on the public function; a -short section in `docs/utils.rst` following the existing per-tool pattern -(`.. _add_reachable_pairs:` anchor, usage line, behavior summary) — the -docstring stays the source of truth for details. +### S1: Move the checker into the package + console entry point + +One atomic increment — moving the module without its tests would break the +suite, so code and tests move together. + +**Code:** +- `git mv utils/check_archive_structure.py` → + `flpq_data/tools/check_archive_structure.py`; update the module docstring + Usage block to `flpq-check-archive ...`. +- New `flpq_data/tools/__init__.py` — star import of the checker module, + following the `flpq_data/dataset/__init__.py` pattern. +- New `flpq_data/dataset/s3.py`: `DEFAULT_ENDPOINT_URL`, `DEFAULT_BUCKET`, + `create_s3_client` (moved from `utils/upload_to_s3.py`; boto3 imported + lazily inside the function body; numpydoc docstring with doctest-safe + examples). Star-imported from `flpq_data/dataset/__init__.py`. +- `pyproject.toml`: `[project.scripts]` with + `flpq-check-archive = "flpq_data.tools.check_archive_structure:main"`. +- Checker's `--audit` branch: import the S3 helpers from + `flpq_data.dataset.s3`; wrap client creation so a missing boto3 prints + `error: --audit requires boto3 (pip install boto3)` and returns 1. +- Utils importers updated to the package path: `merge_archive.py`, + `add_reachable_pairs.py`, `reachable_pairs_tables.py`, + `strip_reverse_edges.py`, `generate_registry.py`; `upload_to_s3.py` drops + its own constant/client definitions (imports them from the package; its + `__all__` keeps re-exporting them so existing imports keep working) and + its lazy `validate_archive` import points at the package module. + +**Tests:** +- Move `tests/utils/test_check_archive_structure.py` → + `tests/tools/test_check_archive_structure.py`; imports from + `flpq_data.tools.check_archive_structure`. +- Extract the shared tree builders (`README`, `RESULTS`, `_query_dir`, + `_valid_tree`, `_partial_tree`, `_tarball`) into `tests/archive_helpers.py`; + new root `tests/conftest.py` inserting `tests/` on `sys.path` (the four + utils test files — `test_merge_archive`, `test_add_reachable_pairs`, + `test_strip_reverse_edges`, `test_upload_to_s3` — import those helpers; + their import lines point at `archive_helpers`). +- New tests: the missing-boto3 `--audit` branch (mocked import failure); + the declared console entry point via `importlib.metadata` (name + `flpq-check-archive`, value `flpq_data.tools.check_archive_structure:main`). + +**Docs:** none in this subtask beyond the moved module's docstring (the +docs sweep is S2). **Spec:** -- CLI: `python utils/add_reachable_pairs.py ARCHIVE.tar.gz --graph NAME - --category CAT [--partial] [--dry-run]`. -- Validate the archive first (full or partial mode per `--partial`); refuse - to proceed on any structure problem. -- For each `queries///` directory emit one row: - `(graph, ., category, class, nnz)` where `` is the first - present of `cnf > re > mcfg > rsm` (must be an actual file in the query - directory; all current rows are `.cnf`) and `nnz` is the declared entry - count from the `results.mtx` header. -- Write to `flpq_data/dataset/reachable_pairs.csv` preserving the existing - alphabetical order by graph name (insert, don't resort); refuse a duplicate - `(graph, query_class, stem)` row; `--dry-run` prints the rows and writes - nothing. -- Exit non-zero on any problem (invalid archive, duplicate row, missing - representation file), printing each problem. - -### S5: Maintainer acceptance pipeline in the skills [done] 8e98011 - -**Code:** Extend `.opencode/skills/add-graph/SKILL.md` with an "Accepting an -external contribution" section; update `.opencode/skills/add-grammar/SKILL.md` -(source-of-truth pointers + the grammar-arrives-with-data rule). - -**Tests:** none (markdown only). - -**Docs:** skills stay thin pointers to tools/docs — no duplication of tool -docstrings or template fields. +- The public API and behavior are unchanged — this is a move, not a rewrite; + `validate_archive`, `unpack_single_dir`, `stored_reverse_forward`, + `parse_readme_sections`, `QUERY_CLASSES`, `MANDATORY_README_SECTIONS` keep + their signatures. +- `flpq_data/dataset/s3.py` must import cleanly without boto3 installed + (lazy import inside `create_s3_client` only). +- After the move, no file outside `tests/` may import the bare + `check_archive_structure` module name; all in-repo consumers use + `flpq_data.tools.check_archive_structure`. +- Verify: full suite + coverage gate green (canonical command from + `docs/developer.rst`). + +### S2: Docs and flow sweep + +**Code:** none (docs/templates/skills only). + +**Tests:** fresh docs build under the no-warnings policy. + +**Docs:** +- New reference page `docs/reference/tools/index.rst` following the + `docs/reference/dataset/index.rst` pattern (`.. automodule::` + + autosummary of the checker's public API, `generated/` toctree); it carries + the existing `.. _archive_structure:` anchor (so `:ref:`archive_structure` + from `docs/flpq.rst` and `docs/utils.rst` keeps resolving); add + `tools/index` to the `docs/reference/index.rst` toctree. +- Remove the "Archive structure checker" section from `docs/utils.rst` (the + tool is no longer a repo util). +- `docs/flpq.rst`: update the checker path mention in the migration-path + note. +- `.github/ISSUE_TEMPLATE/new-data-template.md` "Validation" section: the + recipe becomes `pip install flpq-data` + + `flpq-check-archive .tar.gz [--partial]` — no raw-URL script + download. +- `docs/graphs/index.rst`: "Contributing data" → Validation subsection and + the "File structure" partial-validation command mention use the console + command. +- `.opencode/skills/add-graph/SKILL.md`: the two checker invocations use the + console command. +- `AGENTS.md`: the checker path in the data-archives principle line and a + `flpq_data/tools/` line in the package layout. +- `CHANGELOG.md` Unreleased: Added — `flpq-check-archive` console command, + validation with `pip install flpq-data` alone; Changed — checker moved + from `utils/` to `flpq_data/tools/`, S3 client helpers moved to + `flpq_data.dataset.s3`. **Spec:** -- `add-graph`: per archive of the issue — download from the Drive link → - verify SHA256 against the issue → re-run `check_archive_structure.py` - (full/`--partial`) → name uniqueness vs `registry.json` / - `reachable_pairs.csv` → add CSV rows via `utils/add_reachable_pairs.py` → - upload (full) or `merge_archive.py` (partial; re-validates and checks the - CSV rows). Once per issue — `generate_registry.py` → the existing docs - wiring steps (per-graph pages, category table(s), canonical grammars, new - category page) → `reachable_pairs_tables.py --update` + - `archive_sizes.py --update`. -- `add-grammar`: grammars arrive only together with a data contribution - (cases 2/3/4b of the CONTRIBUTE NEW DATA issue); pointer to the template; - the generator-module conventions stay as-is. - -### S6: Changelog [done] 0f20cd1 - -**Code:** none. **Tests:** none. - -**Docs:** Unreleased entry in `CHANGELOG.md`: issue-first contribution flow -with multi-archive issues, unified CONTRIBUTE NEW DATA template, removed -specialized templates, new `utils/add_reachable_pairs.py`. +- Every remaining mention of `utils/check_archive_structure.py` in + docs/templates/skills/AGENTS.md is updated (the historical CHANGELOG entry + stays as-is). +- The contributor flow documented end-to-end: `pip install flpq-data`, then + `flpq-check-archive .tar.gz [--partial]`. --- ## Execution order -S1 → S2 → S3 → S4 → S5 → S6. One commit per subtask -(`feat(164-SN)` / `docs(164-SN)`), code review + quality gate before merge to -`dev`; the last subtask's commit carries `Closes #164`. +S1 → S2. One commit per subtask (`feat(165-SN)` / `docs(165-SN)`), code +review + quality gate before merge to `dev`; the last subtask's commit +carries `Closes #165`. Final verification includes a one-off `uv build` +wheel inspection (module present, entry point registered). From 9abbf5d3885096db8adb63ed6f0d84296f7f5d5a Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 18:43:23 +0300 Subject: [PATCH 19/26] feat(165-S1): package the archive checker as flpq-check-archive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move utils/check_archive_structure.py into flpq_data/tools/ and register the flpq-check-archive console entry point, so external contributors validate archives with pip install flpq-data alone instead of downloading the script from the repo. The S3 client factory and bucket constants move to flpq_data/dataset/s3.py (boto3 imported lazily — it is a dev-only dependency), and --audit reports a clean error when boto3 is missing. The utils importers now use the package path; upload_to_s3 keeps re-exporting the moved names so existing imports work unchanged. The module now counts toward the 95% coverage gate, which exposed 28 never-tested branch edges. Sixteen new tests cover the real defensive branches (malformed MTX headers and entry lines, nnz mismatches, blank body lines, unexpected queries/ entries, missing queries README, subdirectories in query dirs, forbidden label components, same-directory terminal-set consistency, multi-top-level tarballs, broken symlinks, the no-path CLI). Two provably dead constructs are removed: the indexed- reversed label check (every stem it matches also matches the reverse regex, so it could never fire) and the __main__ guard (replaced by the console entry point; no other package module has one). Shared archive-tree builders move to tests/archive_helpers.py with a root tests/conftest.py putting tests/ on sys.path; the four utils test files import from there. New tests also pin the missing-boto3 --audit branch and the declared console entry point via importlib.metadata. --- flpq_data/dataset/__init__.py | 1 + flpq_data/dataset/s3.py | 65 ++++ flpq_data/tools/__init__.py | 1 + .../tools}/check_archive_structure.py | 47 ++- pyproject.toml | 7 +- tests/archive_helpers.py | 85 +++++ tests/conftest.py | 4 + .../test_check_archive_structure.py | 301 +++++++++++++----- tests/utils/test_add_reachable_pairs.py | 2 +- tests/utils/test_merge_archive.py | 7 +- tests/utils/test_strip_reverse_edges.py | 8 +- tests/utils/test_upload_to_s3.py | 12 +- utils/add_reachable_pairs.py | 8 +- utils/generate_registry.py | 6 +- utils/merge_archive.py | 15 +- utils/reachable_pairs_tables.py | 3 +- utils/strip_reverse_edges.py | 5 +- utils/upload_to_s3.py | 45 +-- uv.lock | 4 +- 19 files changed, 452 insertions(+), 174 deletions(-) create mode 100644 flpq_data/dataset/s3.py create mode 100644 flpq_data/tools/__init__.py rename {utils => flpq_data/tools}/check_archive_structure.py (96%) create mode 100644 tests/archive_helpers.py create mode 100644 tests/conftest.py rename tests/{utils => tools}/test_check_archive_structure.py (74%) diff --git a/flpq_data/dataset/__init__.py b/flpq_data/dataset/__init__.py index 71c3be56..4a5f76a7 100644 --- a/flpq_data/dataset/__init__.py +++ b/flpq_data/dataset/__init__.py @@ -2,3 +2,4 @@ from flpq_data.dataset.data import * from flpq_data.dataset.reachable_pairs import * from flpq_data.dataset.registry import * +from flpq_data.dataset.s3 import * diff --git a/flpq_data/dataset/s3.py b/flpq_data/dataset/s3.py new file mode 100644 index 00000000..2241db80 --- /dev/null +++ b/flpq_data/dataset/s3.py @@ -0,0 +1,65 @@ +"""S3 access to the dataset object storage. + +The dataset archives live in a Yandex Object Storage bucket reachable via +the S3 API. This module holds the storage endpoint and bucket name and the +client factory used by the maintainer tools (``utils/upload_to_s3.py`` and +the ``--audit`` mode of :mod:`flpq_data.tools.check_archive_structure`). + +``boto3`` is imported lazily inside :func:`create_s3_client`, so importing +this module does not require it. +""" + +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from botocore.client import BaseClient + +__all__ = [ + "DEFAULT_ENDPOINT_URL", + "DEFAULT_BUCKET", + "create_s3_client", +] + +#: The S3 API endpoint of the dataset object storage. +DEFAULT_ENDPOINT_URL = "https://s3.yandexcloud.net" + +#: The bucket holding the dataset archives. +DEFAULT_BUCKET = "cfpq-data" + + +def create_s3_client( + access_key_id: str, + secret_access_key: str, + endpoint_url: str = DEFAULT_ENDPOINT_URL, +) -> "BaseClient": + """Create a boto3 S3 client for the dataset object storage. + + Parameters + ---------- + access_key_id : str + Yandex Cloud IAM service account key ID (provided from the CLI). + secret_access_key : str + Yandex Cloud IAM service account secret key (provided from the CLI). + endpoint_url : str, optional + S3 API endpoint of the bucket's region. + Default: ``DEFAULT_ENDPOINT_URL``. + + Returns + ------- + client : botocore.client.BaseClient + Configured S3 client. + + Raises + ------ + ImportError + If ``boto3`` is not installed (it is a development dependency; the + callers that need it report a friendly error instead). + """ + import boto3 + + return boto3.client( + "s3", + aws_access_key_id=access_key_id, + aws_secret_access_key=secret_access_key, + endpoint_url=endpoint_url, + ) diff --git a/flpq_data/tools/__init__.py b/flpq_data/tools/__init__.py new file mode 100644 index 00000000..655862e2 --- /dev/null +++ b/flpq_data/tools/__init__.py @@ -0,0 +1 @@ +from flpq_data.tools.check_archive_structure import * diff --git a/utils/check_archive_structure.py b/flpq_data/tools/check_archive_structure.py similarity index 96% rename from utils/check_archive_structure.py rename to flpq_data/tools/check_archive_structure.py index f8bfa8e3..f9b92150 100644 --- a/utils/check_archive_structure.py +++ b/flpq_data/tools/check_archive_structure.py @@ -12,11 +12,11 @@ only ``queries/`` (the new query directories plus a ``README.md`` fragment with their sections) and is validated with ``--partial``. -Usage (from any directory):: +The command-line entry point is installed with the package:: - python utils/check_archive_structure.py ARCHIVE.tar.gz - python utils/check_archive_structure.py UNPACKED_DIR - python utils/check_archive_structure.py PARTIAL.tar.gz --partial + flpq-check-archive ARCHIVE.tar.gz + flpq-check-archive UNPACKED_DIR + flpq-check-archive PARTIAL.tar.gz --partial """ import argparse @@ -310,7 +310,6 @@ def _mtx_range_problems(graph_dir: pathlib.Path) -> list[str]: # --------------------------------------------------------------------------- # _FORBIDDEN_COMPONENTS = ("bar", "rev") -_INDEXED_RE = re.compile(r"^(.+)_(\d+)$") _INDEXED_REV_RE = re.compile(r"^(.*)_r_(\d+)$") @@ -359,7 +358,7 @@ def _label_problems(graph_dir: pathlib.Path) -> list[str]: stems = {path.stem for path in graph_dir.glob("*.mtx")} for stem in sorted(stems): - # Rule 1: no forbidden components (bar, rev) + # No forbidden components (bar, rev) parts = stem.replace("_", " ").split() for part in parts: if part in _FORBIDDEN_COMPONENTS: @@ -369,17 +368,7 @@ def _label_problems(graph_dir: pathlib.Path) -> list[str]: ) break - # Rule 2: indexed-reversed order must be _r_, not __r - m = _INDEXED_RE.fullmatch(stem) - if m and not _INDEXED_REV_RE.fullmatch(stem): - base, idx = m.groups() - if base.endswith("_r"): - problems.append( - f"graph/{stem}.mtx: indexed-reversed label must be " - f"{base[:-2]}_r_{idx} (got {stem})" - ) - - # Rule 3: no stored reverses — if the forward label is stored, neither its + # No stored reverses — if the forward label is stored, neither its # unindexed reverse (L_r) nor its indexed reverse (B_r_) may be stored. for stem in sorted(stems): forward = stored_reverse_forward(stem, stems) @@ -769,7 +758,11 @@ def main(argv: Optional[Sequence[str]] = None) -> int: args = parser.parse_args(argv) if args.audit: - from upload_to_s3 import DEFAULT_BUCKET, DEFAULT_ENDPOINT_URL, create_s3_client + from flpq_data.dataset.s3 import ( + DEFAULT_BUCKET, + DEFAULT_ENDPOINT_URL, + create_s3_client, + ) if not args.access_key_id or not args.secret_access_key: print( @@ -777,11 +770,15 @@ def main(argv: Optional[Sequence[str]] = None) -> int: "required for --audit" ) return 1 - client = create_s3_client( - args.access_key_id, - args.secret_access_key, - args.endpoint_url or DEFAULT_ENDPOINT_URL, - ) + try: + client = create_s3_client( + args.access_key_id, + args.secret_access_key, + args.endpoint_url or DEFAULT_ENDPOINT_URL, + ) + except ImportError: + print("error: --audit requires boto3 (pip install boto3)") + return 1 problems = _audit(client, args.bucket or DEFAULT_BUCKET, args.prefix) elif args.path is None: print("error: give a path to validate or use --audit") @@ -797,7 +794,3 @@ def main(argv: Optional[Sequence[str]] = None) -> int: kind = "partial archive" if args.partial else "archive" print(f"{kind} structure ok") return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/pyproject.toml b/pyproject.toml index 10eb43b7..9b766feb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -42,6 +42,9 @@ Documentation = "https://formallanguageconstrainedpathquerying.github.io/FLPQ_Da "Source Code" = "https://github.com/FormalLanguageConstrainedPathQuerying/FLPQ_Data" "Bug Tracker" = "https://github.com/FormalLanguageConstrainedPathQuerying/FLPQ_Data/issues" +[project.scripts] +flpq-check-archive = "flpq_data.tools.check_archive_structure:main" + [dependency-groups] dev = [ "boto3>=1.43.86,<2.0.0", @@ -91,11 +94,11 @@ select = ["E", "F", "W", "I"] "tests/utils/conftest.py" = ["E402"] [tool.ty.environment] -extra-paths = ["utils"] +extra-paths = ["utils", "tests"] [tool.pyright] pythonVersion = "3.11" -extraPaths = ["utils"] +extraPaths = ["utils", "tests"] # Local scratch/working directory (gitignored), not part of the package. exclude = ["temporal_cfpq"] diff --git a/tests/archive_helpers.py b/tests/archive_helpers.py new file mode 100644 index 00000000..943c58a4 --- /dev/null +++ b/tests/archive_helpers.py @@ -0,0 +1,85 @@ +"""Builders for graph-archive trees shared by the archive-related tests.""" + +import pathlib +import tarfile + +README = """# g + +## What is this graph? +A test graph. + +## Source +A test source. + +## Construction +Generated. + +## Nodes +Two nodes. + +## Edges and labels +Label a: one edge. + +## Query classes +CFPQ applies. + +## License +Apache-2.0. + +## Caveats +None. +""" + +RESULTS = ( + "%%MatrixMarket matrix coordinate pattern general\n" + "%%GraphBLAS type bool\n2 2 1\n0 1\n" +) + + +def _query_dir( + tree: pathlib.Path, cls: str, name: str, representation: str +) -> pathlib.Path: + query = tree / "queries" / cls / name + query.mkdir(parents=True) + (query / f"{name}{representation}").write_text( + { + ".cnf": "S\ta\tN1\nN1\ta\n\nCount:\nS", + ".rsm": "S -> a*", + ".re": "a*", + ".mcfg": "A(a)\nS(x1) <- A(x1)", + }[representation] + ) + (query / "results.mtx").write_text(RESULTS) + return query + + +def _valid_tree(root: pathlib.Path, name: str = "g") -> pathlib.Path: + d = root / name + (d / "graph").mkdir(parents=True) + for cls in ("cfpq", "rpq", "mcfpq"): + (d / "queries" / cls).mkdir(parents=True) + (d / "README.md").write_text(README) + (d / "graph" / "a.mtx").write_text( + "%%MatrixMarket matrix coordinate pattern general\n" + "%%GraphBLAS type bool\n2 2 1\n0 1\n" + ) + _query_dir(d, "cfpq", "s", ".cnf") + (d / "queries" / "README.md").write_text( + "# Queries for g\n\n## cfpq/s\n- Language: a*.\n- Purpose: test.\n" + ) + return d + + +def _partial_tree(root: pathlib.Path, name: str = "g") -> pathlib.Path: + d = root / name + _query_dir(d, "cfpq", "s", ".cnf") + (d / "queries" / "README.md").write_text( + "# Queries for g\n\n## cfpq/s\n- Language: a*.\n- Purpose: test.\n" + ) + return d + + +def _tarball(tree: pathlib.Path, dest: pathlib.Path) -> pathlib.Path: + with tarfile.open(dest, "w:gz") as tarball: + tarball.add(tree, arcname=tree.name) + return dest diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 00000000..13f82945 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,4 @@ +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) diff --git a/tests/utils/test_check_archive_structure.py b/tests/tools/test_check_archive_structure.py similarity index 74% rename from tests/utils/test_check_archive_structure.py rename to tests/tools/test_check_archive_structure.py index d18191e4..b16fdda1 100644 --- a/tests/utils/test_check_archive_structure.py +++ b/tests/tools/test_check_archive_structure.py @@ -1,90 +1,19 @@ import pathlib import shutil +import sys import tarfile from unittest import mock -from check_archive_structure import _audit, main, validate_archive - -README = """# g - -## What is this graph? -A test graph. - -## Source -A test source. - -## Construction -Generated. - -## Nodes -Two nodes. - -## Edges and labels -Label a: one edge. - -## Query classes -CFPQ applies. - -## License -Apache-2.0. - -## Caveats -None. -""" - -RESULTS = ( - "%%MatrixMarket matrix coordinate pattern general\n" - "%%GraphBLAS type bool\n2 2 1\n0 1\n" +from archive_helpers import ( + README, + RESULTS, + _partial_tree, + _query_dir, + _tarball, + _valid_tree, ) - -def _query_dir( - tree: pathlib.Path, cls: str, name: str, representation: str -) -> pathlib.Path: - query = tree / "queries" / cls / name - query.mkdir(parents=True) - (query / f"{name}{representation}").write_text( - { - ".cnf": "S\ta\tN1\nN1\ta\n\nCount:\nS", - ".rsm": "S -> a*", - ".re": "a*", - ".mcfg": "A(a)\nS(x1) <- A(x1)", - }[representation] - ) - (query / "results.mtx").write_text(RESULTS) - return query - - -def _valid_tree(root: pathlib.Path, name: str = "g") -> pathlib.Path: - d = root / name - (d / "graph").mkdir(parents=True) - for cls in ("cfpq", "rpq", "mcfpq"): - (d / "queries" / cls).mkdir(parents=True) - (d / "README.md").write_text(README) - (d / "graph" / "a.mtx").write_text( - "%%MatrixMarket matrix coordinate pattern general\n" - "%%GraphBLAS type bool\n2 2 1\n0 1\n" - ) - _query_dir(d, "cfpq", "s", ".cnf") - (d / "queries" / "README.md").write_text( - "# Queries for g\n\n## cfpq/s\n- Language: a*.\n- Purpose: test.\n" - ) - return d - - -def _partial_tree(root: pathlib.Path, name: str = "g") -> pathlib.Path: - d = root / name - _query_dir(d, "cfpq", "s", ".cnf") - (d / "queries" / "README.md").write_text( - "# Queries for g\n\n## cfpq/s\n- Language: a*.\n- Purpose: test.\n" - ) - return d - - -def _tarball(tree: pathlib.Path, dest: pathlib.Path) -> pathlib.Path: - with tarfile.open(dest, "w:gz") as tarball: - tarball.add(tree, arcname=tree.name) - return dest +from flpq_data.tools.check_archive_structure import _audit, main, validate_archive def test_valid_directory_passes(tmp_path): @@ -627,6 +556,191 @@ def test_main_partial_reports_problems(tmp_path, capsys): assert "error:" in out and "structure violation" in out +def test_unexpected_entry_in_queries(tmp_path): + tree = _valid_tree(tmp_path) + (tree / "queries" / "notes.txt").write_text("extra") + + problems = validate_archive(tree) + + assert any("queries/notes.txt: unexpected entry" in p for p in problems) + + +def test_missing_queries_readme(tmp_path): + tree = _valid_tree(tmp_path) + (tree / "queries" / "README.md").unlink() + + problems = validate_archive(tree) + + assert any("queries/README.md: missing" in p for p in problems) + + +def test_partial_unexpected_entry_in_queries(tmp_path): + tree = _partial_tree(tmp_path) + (tree / "queries" / "notes.txt").write_text("extra") + + problems = validate_archive(tree, partial=True) + + assert any("queries/notes.txt: unexpected entry" in p for p in problems) + + +def test_partial_missing_queries_readme(tmp_path): + tree = _partial_tree(tmp_path) + (tree / "queries" / "README.md").unlink() + + problems = validate_archive(tree, partial=True) + + assert any("queries/README.md: missing" in p for p in problems) + + +def test_subdirectory_in_query_dir(tmp_path): + tree = _valid_tree(tmp_path) + (tree / "queries" / "cfpq" / "s" / "sub").mkdir() + + problems = validate_archive(tree) + + assert any("only files are allowed" in p for p in problems) + + +def test_second_header_line_mismatch(tmp_path): + tree = _valid_tree(tmp_path) + (tree / "graph" / "a.mtx").write_text( + "%%MatrixMarket matrix coordinate pattern general\n" + "wrong second line\n2 2 1\n0 1\n" + ) + + problems = validate_archive(tree) + + assert any(p.startswith("graph/:") for p in problems) + + +def test_non_numeric_dimension_line(tmp_path): + tree = _valid_tree(tmp_path) + (tree / "graph" / "a.mtx").write_text( + "%%MatrixMarket matrix coordinate pattern general\n" + "%%GraphBLAS type bool\nx y z\n0 1\n" + ) + + problems = validate_archive(tree) + + assert any(p.startswith("graph/:") for p in problems) + + +def test_blank_line_in_mtx_body_is_skipped(tmp_path): + tree = _valid_tree(tmp_path) + (tree / "graph" / "a.mtx").write_text( + "%%MatrixMarket matrix coordinate pattern general\n" + "%%GraphBLAS type bool\n2 2 1\n0 1\n\n" + ) + + assert validate_archive(tree) == [] + + +def test_graph_mtx_entry_count_mismatch(tmp_path): + tree = _valid_tree(tmp_path) + (tree / "graph" / "a.mtx").write_text( + "%%MatrixMarket matrix coordinate pattern general\n" + "%%GraphBLAS type bool\n2 2 2\n0 1\n" + ) + + problems = validate_archive(tree) + + assert any(p.startswith("graph/:") for p in problems) + + +def test_graph_mtx_malformed_entry_line(tmp_path): + tree = _valid_tree(tmp_path) + (tree / "graph" / "a.mtx").write_text( + "%%MatrixMarket matrix coordinate pattern general\n" + "%%GraphBLAS type bool\n2 2 1\nx y\n" + ) + + problems = validate_archive(tree) + + assert any(p.startswith("graph/:") for p in problems) + + +def test_forbidden_label_component(tmp_path): + tree = _valid_tree(tmp_path) + _write_mtx(tree / "graph" / "a_bar.mtx") + + problems = validate_archive(tree) + + assert any("forbidden component 'bar'" in p for p in problems) + + +def test_same_dir_different_terminals(tmp_path): + tree = _valid_tree(tmp_path) + query = tree / "queries" / "cfpq" / "t" + query.mkdir(parents=True) + (query / "t.cnf").write_text("S\tb\tN1\nN1\tb\n\nCount:\nS") + (query / "t.rsm").write_text("S -> c*") + (query / "results.mtx").write_text(RESULTS) + (tree / "queries" / "README.md").write_text( + "# Queries for g\n\n## cfpq/s\n- Language: a*.\n\n## cfpq/t\n- Language: b*.\n" + ) + + problems = validate_archive(tree) + + assert any("use different terminal sets" in p for p in problems) + + +def test_same_dir_subset_terminals(tmp_path): + tree = _valid_tree(tmp_path) + query = tree / "queries" / "cfpq" / "t" + query.mkdir(parents=True) + (query / "t.cnf").write_text("S\ta\tN1\nN1\ta\nS\tb\tN2\nN2\tb\n\nCount:\nS") + (query / "t.rsm").write_text("S -> a*") + (query / "results.mtx").write_text(RESULTS) + (tree / "queries" / "README.md").write_text( + "# Queries for g\n\n## cfpq/s\n- Language: a*.\n\n## cfpq/t\n- Language: ab.\n" + ) + + problems = validate_archive(tree) + + assert any("use different terminal sets" in p for p in problems) + + +def test_same_dir_equal_terminals_pass(tmp_path): + tree = _valid_tree(tmp_path) + query = tree / "queries" / "cfpq" / "t" + query.mkdir(parents=True) + (query / "t.cnf").write_text("S\ta\tN1\nN1\ta\n\nCount:\nS") + (query / "t.rsm").write_text("S -> a*") + (query / "results.mtx").write_text(RESULTS) + (tree / "queries" / "README.md").write_text( + "# Queries for g\n\n## cfpq/s\n- Language: a*.\n\n## cfpq/t\n- Language: a*.\n" + ) + + assert validate_archive(tree) == [] + + +def test_tarball_with_two_top_level_dirs(tmp_path): + tree = _valid_tree(tmp_path, name="g") + (tmp_path / "other").mkdir() + dest = tmp_path / "g.tar.gz" + with tarfile.open(dest, "w:gz") as tarball: + tarball.add(tree, arcname="g") + tarball.add(tmp_path / "other", arcname="other") + + problems = validate_archive(dest) + + assert any("must contain a single top-level directory" in p for p in problems) + + +def test_broken_symlink_is_rejected(tmp_path): + link = tmp_path / "g.tar.gz" + link.symlink_to(tmp_path / "nowhere") + + problems = validate_archive(link) + + assert any("not a .tar.gz archive" in p for p in problems) + + +def test_main_without_path_reports_it(capsys): + assert main([]) == 1 + assert "give a path to validate or use --audit" in capsys.readouterr().out + + def _mock_s3(archives: dict[str, bytes]) -> mock.Mock: client = mock.Mock() client.get_paginator.return_value.paginate.return_value = [ @@ -676,7 +790,7 @@ def test_main_audit_requires_credentials(capsys): def test_main_audit_validates_bucket(tmp_path, monkeypatch, capsys): tarball = _tarball(_valid_tree(tmp_path), tmp_path / "g.tar.gz") client = _mock_s3({"5.0.0/graph/g.tar.gz": tarball.read_bytes()}) - monkeypatch.setattr("upload_to_s3.boto3.client", lambda *args, **kwargs: client) + monkeypatch.setattr("boto3.client", lambda *args, **kwargs: client) status = main( [ @@ -692,3 +806,28 @@ def test_main_audit_validates_bucket(tmp_path, monkeypatch, capsys): assert status == 0 assert "archive structure ok" in capsys.readouterr().out + + +def test_main_audit_without_boto3_reports_it(monkeypatch, capsys): + monkeypatch.setitem(sys.modules, "boto3", None) + + status = main( + [ + "--audit", + "--access-key-id", + "key-id", + "--secret-access-key", + "secret", + ] + ) + + assert status == 1 + assert "--audit requires boto3" in capsys.readouterr().out + + +def test_console_entry_point_is_registered(): + from importlib.metadata import entry_points + + eps = entry_points(group="console_scripts", name="flpq-check-archive") + + assert [ep.value for ep in eps] == ["flpq_data.tools.check_archive_structure:main"] diff --git a/tests/utils/test_add_reachable_pairs.py b/tests/utils/test_add_reachable_pairs.py index ccb9fa02..1c9056e4 100644 --- a/tests/utils/test_add_reachable_pairs.py +++ b/tests/utils/test_add_reachable_pairs.py @@ -3,7 +3,7 @@ import add_reachable_pairs from add_reachable_pairs import add_rows, derive_rows, main, representation_file -from test_check_archive_structure import _partial_tree, _tarball, _valid_tree +from archive_helpers import _partial_tree, _tarball, _valid_tree HEADER = ["graph", "grammar", "category", "query_class", "num_reachable_pairs"] diff --git a/tests/utils/test_merge_archive.py b/tests/utils/test_merge_archive.py index a34b0c6c..02172568 100644 --- a/tests/utils/test_merge_archive.py +++ b/tests/utils/test_merge_archive.py @@ -3,14 +3,15 @@ import merge_archive import pytest -from check_archive_structure import validate_archive -from merge_archive import main, merge_archives -from test_check_archive_structure import ( +from archive_helpers import ( _partial_tree, _query_dir, _tarball, _valid_tree, ) +from merge_archive import main, merge_archives + +from flpq_data.tools.check_archive_structure import validate_archive @pytest.fixture(autouse=True) diff --git a/tests/utils/test_strip_reverse_edges.py b/tests/utils/test_strip_reverse_edges.py index 82a28bc5..dde1b697 100644 --- a/tests/utils/test_strip_reverse_edges.py +++ b/tests/utils/test_strip_reverse_edges.py @@ -1,9 +1,13 @@ import pathlib import tempfile -from check_archive_structure import unpack_single_dir, validate_archive +from archive_helpers import README, _tarball, _valid_tree from strip_reverse_edges import main, strip_reverse_edges -from test_check_archive_structure import README, _tarball, _valid_tree + +from flpq_data.tools.check_archive_structure import ( + unpack_single_dir, + validate_archive, +) MTX = ( "%%MatrixMarket matrix coordinate pattern general\n" diff --git a/tests/utils/test_upload_to_s3.py b/tests/utils/test_upload_to_s3.py index 864b45e0..d175af9b 100644 --- a/tests/utils/test_upload_to_s3.py +++ b/tests/utils/test_upload_to_s3.py @@ -2,7 +2,7 @@ from unittest import mock import pytest -from test_check_archive_structure import README, _tarball, _valid_tree +from archive_helpers import README, _tarball, _valid_tree from upload_to_s3 import ( DEFAULT_BUCKET, DEFAULT_ENDPOINT_URL, @@ -21,7 +21,7 @@ def fake_client(service, **kwargs): calls.update(kwargs) return "client" - monkeypatch.setattr("upload_to_s3.boto3.client", fake_client) + monkeypatch.setattr("boto3.client", fake_client) client = create_s3_client("key-id", "secret", endpoint_url="https://example.com") @@ -41,7 +41,7 @@ def fake_client(service, **kwargs): calls.update(kwargs) return "client" - monkeypatch.setattr("upload_to_s3.boto3.client", fake_client) + monkeypatch.setattr("boto3.client", fake_client) create_s3_client("key-id", "secret") @@ -140,7 +140,7 @@ def test_main_uploads_valid_archive_and_reports_size(tmp_path, monkeypatch, caps file = _valid_tarball(tmp_path) client = mock.Mock() client.head_object.return_value = {"ContentLength": file.stat().st_size} - monkeypatch.setattr("upload_to_s3.boto3.client", lambda *args, **kwargs: client) + monkeypatch.setattr("boto3.client", lambda *args, **kwargs: client) status = main( [ @@ -175,7 +175,7 @@ def test_main_reports_size_above_one_megabyte(tmp_path, monkeypatch, capsys): file = _tarball(tree, tmp_path / "g.tar.gz") client = mock.Mock() client.head_object.return_value = {"ContentLength": file.stat().st_size} - monkeypatch.setattr("upload_to_s3.boto3.client", lambda *args, **kwargs: client) + monkeypatch.setattr("boto3.client", lambda *args, **kwargs: client) main( [ @@ -196,7 +196,7 @@ def test_main_refuses_invalid_archive_without_uploading(tmp_path, monkeypatch, c (tree / "README.md").write_text(README.replace("## License\nApache-2.0.\n", "")) file = _tarball(tree, tmp_path / "g.tar.gz") client = mock.Mock() - monkeypatch.setattr("upload_to_s3.boto3.client", lambda *args, **kwargs: client) + monkeypatch.setattr("boto3.client", lambda *args, **kwargs: client) status = main( [ diff --git a/utils/add_reachable_pairs.py b/utils/add_reachable_pairs.py index 3b6f9914..ff3b1ad7 100644 --- a/utils/add_reachable_pairs.py +++ b/utils/add_reachable_pairs.py @@ -28,13 +28,14 @@ import tempfile from typing import Optional, Sequence -from check_archive_structure import ( +from config import MAIN_FOLDER + +from flpq_data.tools.check_archive_structure import ( QUERY_CLASSES, _mtx_header, unpack_single_dir, validate_archive, ) -from config import MAIN_FOLDER __all__ = [ "REACHABLE_PAIRS_CSV", @@ -96,7 +97,8 @@ def derive_rows(root: pathlib.Path, graph: str, category: str) -> list[dict[str, ---------- root : Path The unpacked archive directory (it must have passed - :func:`check_archive_structure.validate_archive`, so every query + :func:`flpq_data.tools.check_archive_structure.validate_archive`, so + every query directory holds a representation file and a parseable ``results.mtx``). graph : str diff --git a/utils/generate_registry.py b/utils/generate_registry.py index 31717f8b..e8aa1354 100644 --- a/utils/generate_registry.py +++ b/utils/generate_registry.py @@ -32,12 +32,16 @@ from typing import Optional, Sequence import requests -from check_archive_structure import QUERY_CLASSES, _graph_dimensions, _mtx_header from config import MAIN_FOLDER from reachable_pairs_tables import load_rows from flpq_data.config import DATASET_VERSION from flpq_data.dataset.data import DATASET_URL +from flpq_data.tools.check_archive_structure import ( + QUERY_CLASSES, + _graph_dimensions, + _mtx_header, +) __all__ = [ "REGISTRY_CSV", diff --git a/utils/merge_archive.py b/utils/merge_archive.py index 6fcd7571..5f0e1722 100644 --- a/utils/merge_archive.py +++ b/utils/merge_archive.py @@ -1,6 +1,6 @@ """Merge a partial archive into an existing graph archive. -A partial archive (validated with ``check_archive_structure.py --partial``) +A partial archive (validated with ``flpq-check-archive --partial``) provides new queries for an existing graph: it contains only ``queries/`` — the new query directories plus a ``README.md`` fragment with their sections. This tool copies the new query directories into the existing archive, @@ -26,12 +26,6 @@ import tempfile from typing import Optional, Sequence, Union -from check_archive_structure import ( - QUERY_CLASSES, - parse_readme_sections, - unpack_single_dir, - validate_archive, -) from config import MAIN_FOLDER from migrate_gdrive_to_s3 import ( DRIVE_FILE_ID_RE, @@ -40,6 +34,13 @@ ) from reachable_pairs_tables import load_rows +from flpq_data.tools.check_archive_structure import ( + QUERY_CLASSES, + parse_readme_sections, + unpack_single_dir, + validate_archive, +) + __all__ = ["REGISTRY_CSV", "merge_archives", "main"] #: The reachable-pairs registry, the single source of truth for applicable diff --git a/utils/reachable_pairs_tables.py b/utils/reachable_pairs_tables.py index aa48194b..5d35559c 100644 --- a/utils/reachable_pairs_tables.py +++ b/utils/reachable_pairs_tables.py @@ -36,9 +36,10 @@ from typing import Optional, Sequence from archive_sizes import iter_graph_tables -from check_archive_structure import QUERY_CLASSES from config import MAIN_FOLDER +from flpq_data.tools.check_archive_structure import QUERY_CLASSES + __all__ = [ "BEGIN_MARKER", "END_MARKER", diff --git a/utils/strip_reverse_edges.py b/utils/strip_reverse_edges.py index 21f9915a..a2988941 100644 --- a/utils/strip_reverse_edges.py +++ b/utils/strip_reverse_edges.py @@ -5,7 +5,8 @@ auto-generated by :func:`flpq_data.graphs.utils.add_reverse_edges`. Some legacy archives still bundle reverse matrices — the unindexed ``L_r.mtx`` or the indexed ``_r_.mtx`` — which -:func:`check_archive_structure.stored_reverse_forward` flags. +:func:`flpq_data.tools.check_archive_structure.stored_reverse_forward` +flags. This tool removes those matrices, updates the archive ``README.md`` ("Edges and labels": the reverse bullet lines are dropped and the stored-edge total is @@ -29,7 +30,7 @@ from dataclasses import dataclass, field from typing import Optional, Sequence, Set, Union -from check_archive_structure import ( +from flpq_data.tools.check_archive_structure import ( stored_reverse_forward, unpack_single_dir, validate_archive, diff --git a/utils/upload_to_s3.py b/utils/upload_to_s3.py index 042af309..f1682150 100644 --- a/utils/upload_to_s3.py +++ b/utils/upload_to_s3.py @@ -5,10 +5,15 @@ import pathlib from typing import Optional, Sequence, Union -import boto3 from botocore.client import BaseClient from sizes import format_size_mb +from flpq_data.dataset.s3 import ( + DEFAULT_BUCKET, + DEFAULT_ENDPOINT_URL, + create_s3_client, +) + __all__ = [ "DEFAULT_ENDPOINT_URL", "DEFAULT_BUCKET", @@ -18,39 +23,6 @@ "main", ] -DEFAULT_ENDPOINT_URL = "https://s3.yandexcloud.net" -DEFAULT_BUCKET = "cfpq-data" - - -def create_s3_client( - access_key_id: str, - secret_access_key: str, - endpoint_url: str = DEFAULT_ENDPOINT_URL, -) -> BaseClient: - """Create a boto3 S3 client for Yandex Object Storage. - - Parameters - ---------- - access_key_id : str - Yandex Cloud IAM service account key ID (provided from the CLI). - secret_access_key : str - Yandex Cloud IAM service account secret key (provided from the CLI). - endpoint_url : str, optional - S3 API endpoint of the bucket's region. - Default: ``DEFAULT_ENDPOINT_URL``. - - Returns - ------- - client : botocore.client.BaseClient - Configured S3 client. - """ - return boto3.client( - "s3", - aws_access_key_id=access_key_id, - aws_secret_access_key=secret_access_key, - endpoint_url=endpoint_url, - ) - def upload_file( client: BaseClient, @@ -165,7 +137,8 @@ def main(argv: Optional[Sequence[str]] = None) -> int: Credentials are always taken from the command line and are never read from environment variables or config files. Any ``.tar.gz`` upload is first validated as a graph archive (see - :mod:`check_archive_structure`); an invalid archive is refused. + :mod:`flpq_data.tools.check_archive_structure`); an invalid archive is + refused. Returns ------- @@ -202,7 +175,7 @@ def main(argv: Optional[Sequence[str]] = None) -> int: args = parser.parse_args(argv) if args.file.endswith(".tar.gz"): - from check_archive_structure import validate_archive + from flpq_data.tools.check_archive_structure import validate_archive problems = validate_archive(args.file) if problems: diff --git a/uv.lock b/uv.lock index 3380d074..7d5a8a40 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 3 +revision = 5 requires-python = ">=3.11, <3.14" resolution-markers = [ "python_full_version >= '3.12' and sys_platform == 'win32'", @@ -595,7 +595,7 @@ docs = [ { name = "jupyter-client", specifier = ">=8.7.0,<9.0.0" }, { name = "nb2plots", specifier = ">=0.7.2,<0.8.0" }, { name = "numpydoc", specifier = ">=1.10.0,<2.0.0" }, - { name = "pydata-sphinx-theme", specifier = ">=0.16.1,<0.22.0" }, + { name = "pydata-sphinx-theme", specifier = ">=0.16.1,<0.24.0" }, { name = "sphinx", specifier = ">=9.0.4,<10.0.0" }, { name = "sphinx-copybutton", specifier = ">=0.5.2,<0.6.0" }, ] From a7da6ffee593eca06f22992df2775955b49b4d9b Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 18:44:08 +0300 Subject: [PATCH 20/26] docs(165): mark S1 done in the detailed plan --- tasks/detailed_plan.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/tasks/detailed_plan.md b/tasks/detailed_plan.md index 43e4bb0a..879ef6e2 100644 --- a/tasks/detailed_plan.md +++ b/tasks/detailed_plan.md @@ -26,13 +26,16 @@ manual script download. - The wheel build (`packages = ["flpq_data"]`) picks up the new submodule automatically; `[project.scripts]` registers the entry point; uv installs the project itself, so the command works in-repo too. -- Coverage: the moved lines now count toward the 95% flpq_data gate; the - existing tests cover the module (incl. mock-tested `_audit`); one new - branch (missing boto3) gets a test. +- Coverage: the moved lines now count toward the 95% flpq_data gate. The + existing tests did not cover 28 defensive branch edges of the module; + S1 adds tests for every reachable one and removes two provably dead + constructs (the indexed-reversed label check — every stem it matches + also matches the reverse regex — and the `__main__` guard, replaced by + the console entry point). --- -### S1: Move the checker into the package + console entry point +### S1: Move the checker into the package + console entry point [done] 9abbf5d One atomic increment — moving the module without its tests would break the suite, so code and tests move together. From 3c0ab2387d06643678e9b7c337f5335a738e8262 Mon Sep 17 00:00:00 2001 From: gsvgit Date: Wed, 14 Oct 2026 19:05:52 +0300 Subject: [PATCH 21/26] docs(165-S2): document flpq-check-archive and sweep the old path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The checker is now part of the installed package, so every documented invocation becomes the flpq-check-archive console command: the contributor recipe in the issue template and the Graphs page is now pip install flpq-data (no script download), and the maintainer steps in the add-graph skill use the command too. The "Archive structure" section moves from docs/utils.rst to a new docs/reference/tools/index.rst reference page (automodule + autosummary of the checker's public API, generated stubs committed), carrying the archive_structure anchor so existing :ref: links keep resolving. The Labels check bullet drops the indexed-reversed ordering clause — the checker never enforced it (the code path was dead and is removed in S1). AGENTS.md points at the new module path and gains a flpq_data/tools/ layout line; the changelog records the command and the move. Closes #165 --- .github/ISSUE_TEMPLATE/new-data-template.md | 6 +- .opencode/skills/add-graph/SKILL.md | 8 +- AGENTS.md | 6 +- CHANGELOG.md | 7 ++ docs/flpq.rst | 2 +- docs/graphs/index.rst | 10 +-- docs/reference/index.rst | 1 + docs/reference/tools/index.rst | 82 +++++++++++++++++++++ docs/utils.rst | 58 --------------- 9 files changed, 107 insertions(+), 73 deletions(-) create mode 100644 docs/reference/tools/index.rst diff --git a/.github/ISSUE_TEMPLATE/new-data-template.md b/.github/ISSUE_TEMPLATE/new-data-template.md index e433b3fe..5480c612 100644 --- a/.github/ISSUE_TEMPLATE/new-data-template.md +++ b/.github/ISSUE_TEMPLATE/new-data-template.md @@ -34,13 +34,13 @@ One row per archive. An issue may carry a **set** of archives (e.g. all graphs o ## Validation (mandatory, per archive) -Validate every archive **before** sharing it and paste the output below. Without cloning the repo: `pip install flpq-data`, download the single script [`check_archive_structure.py`](https://raw.githubusercontent.com/FormalLanguageConstrainedPathQuerying/FLPQ_Data/master/utils/check_archive_structure.py), and run it (all imports resolve from the installed package). +Validate every archive **before** sharing it and paste the output below. Without cloning the repo: `pip install flpq-data` — the checker is installed with the package as the `flpq-check-archive` command. ### `.tar.gz` ``` -python check_archive_structure.py .tar.gz # full archive -python check_archive_structure.py .tar.gz --partial # partial archive +flpq-check-archive .tar.gz # full archive +flpq-check-archive .tar.gz --partial # partial archive structure ok ``` diff --git a/.opencode/skills/add-graph/SKILL.md b/.opencode/skills/add-graph/SKILL.md index c0e0844e..3aacb41e 100644 --- a/.opencode/skills/add-graph/SKILL.md +++ b/.opencode/skills/add-graph/SKILL.md @@ -21,8 +21,8 @@ The archive layout and the label conventions are documented once in the "File structure" section of `docs/graphs/index.rst` — follow it exactly. Pack every query of the graph into the archive per the "Queries and grammars" section of the issue template, then validate with -`python utils/check_archive_structure.py .tar.gz` (the upload tool -runs the same check and refuses invalid archives). +`flpq-check-archive .tar.gz` (the upload tool runs the same check +and refuses invalid archives). New queries for an existing graph ship as a **partial archive** instead — see the "Data format" section of the issue template; it is validated with @@ -37,8 +37,8 @@ Per archive, in order: 1. Download it from the Drive link in the issue. 2. Verify its SHA256 against the value in the issue — a mismatch means the shared file is not the validated one; stop and ask the contributor. -3. Re-run `python utils/check_archive_structure.py .tar.gz` (add - `--partial` for partial archives) — it must pass. +3. Re-run `flpq-check-archive .tar.gz` (add `--partial` for partial + archives) — it must pass. 4. Check the graph name against `flpq_data/dataset/registry.json` and `flpq_data/dataset/reachable_pairs.csv`: a full archive's name must be new, a partial archive's target must exist. diff --git a/AGENTS.md b/AGENTS.md index 8b4af487..4dd825c0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,8 +25,8 @@ gate → merge, and points to the other workflow skills below. code: design decisions, non-trivial constraints. Skills stay thin pointers to docs, code, or CI — they never re-describe them. * Data archives are self-contained and must align with the checker - (`utils/check_archive_structure.py`) — the source of truth for any data - structure. + (`flpq_data/tools/check_archive_structure.py`, console command + `flpq-check-archive`) — the source of truth for any data structure. * Tools, not instructions. If you can do something with existing tool --- do it. No thinking, no long instructions, no manual analysis. You want to analyze code coverage? Just run coverage tool and analyze report. No workaround for regular tasks. If there is a tool for regular task it must be installed and configured appropriately. * Always learn, never forget — encode patterns before session ends @@ -39,6 +39,8 @@ gate → merge, and points to the other workflow skills below. - `flpq_data/queries/` — per-class query modules: `cfpq/` (`generators`, `readwrite`, `converters`, `utils`), `rpq/` (`readwrite`), `mcfpq/` (`readwrite`). +- `flpq_data/tools/` — the archive structure checker (console command + `flpq-check-archive`). - `tests/` mirrors the package structure (pytest + doctests). ## Skills diff --git a/CHANGELOG.md b/CHANGELOG.md index 7be9a46e..1e2669ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- The `flpq-check-archive` console command (installed with the package): + validates a graph archive's structure, so contributors check their + archives with `pip install flpq-data` alone — no script download. - `utils/add_reachable_pairs.py`: derives the reachable-pairs rows of one validated archive (one row per query directory, with the declared entry count of its `results.mtx` as `num_reachable_pairs`) and inserts them into @@ -17,6 +20,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- The archive structure checker moved from `utils/check_archive_structure.py` + to `flpq_data/tools/` (console command `flpq-check-archive`); the S3 client + factory and bucket constants moved to `flpq_data.dataset.s3`. + - Data contributions are issue-first: a single "CONTRIBUTE NEW DATA" issue carries one or more validated archives (one graph each), the per-archive validation output and SHA256, and the motivation; maintainers upload and diff --git a/docs/flpq.rst b/docs/flpq.rst index bc1cad6f..58b36dcf 100644 --- a/docs/flpq.rst +++ b/docs/flpq.rst @@ -419,7 +419,7 @@ Migration path: ``queries/cfpq//`` directory each, add the per-query ``results.mtx`` (the constrained reachability facts), write the mandatory-question ``README.md``, and validate with - ``utils/check_archive_structure.py`` (:ref:`archive_structure`). + ``flpq-check-archive`` (:ref:`archive_structure`). - The example query archives (``4.0.0/grammar/example/``) are dropped — every query file lives inside a graph archive. - Re-point ``DATASET_URL`` at the new prefix as part of the version bump diff --git a/docs/graphs/index.rst b/docs/graphs/index.rst index 3773bcef..872717a7 100644 --- a/docs/graphs/index.rst +++ b/docs/graphs/index.rst @@ -37,12 +37,12 @@ Validation Every archive must pass the structure checker before it is shared:: - python utils/check_archive_structure.py .tar.gz # full archive - python utils/check_archive_structure.py .tar.gz --partial # partial archive + flpq-check-archive .tar.gz # full archive + flpq-check-archive .tar.gz --partial # partial archive The issue carries the pasted output of this command per archive, the archive's SHA256, and the FLPQ_Data version used; the template has the -recipe for running the checker without cloning the repo. +recipe — ``pip install flpq-data`` installs the checker. Consistency invariants ^^^^^^^^^^^^^^^^^^^^^^ @@ -112,8 +112,8 @@ A **partial archive** provides new queries for an existing graph: it unpacks to a directory named after that graph and contains only ``queries/`` — the new query directories (same layout) plus a ``README.md`` fragment with one section per new query. It is validated with -``python utils/check_archive_structure.py .tar.gz --partial`` and -merged into the existing archive with :ref:`merge_archive`. +``flpq-check-archive .tar.gz --partial`` and merged into the existing +archive with :ref:`merge_archive`. The pre-migration graphs on the :ref:`old_graphs` page use the old CSV format instead. diff --git a/docs/reference/index.rst b/docs/reference/index.rst index fa70a98f..84761836 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -25,3 +25,4 @@ Reference dataset/index graphs/index queries/index + tools/index diff --git a/docs/reference/tools/index.rst b/docs/reference/tools/index.rst new file mode 100644 index 00000000..0de8fc56 --- /dev/null +++ b/docs/reference/tools/index.rst @@ -0,0 +1,82 @@ +.. _tools: + +.. currentmodule:: flpq_data + +***** +Tools +***** + +.. only:: html + + :Release: |release| + :Date: |today| + +.. automodule:: flpq_data.tools +.. autosummary:: + :toctree: generated/ + + MANDATORY_README_SECTIONS + QUERY_CLASSES + parse_readme_sections + stored_reverse_forward + unpack_single_dir + validate_archive + +.. _archive_structure: + +Archive structure checker +------------------------- + +``flpq-check-archive`` (installed with the package) validates that a graph +archive has the fixed self-contained structure documented in the "File +structure" section of the :ref:`graphs` page:: + + flpq-check-archive ARCHIVE.tar.gz + flpq-check-archive UNPACKED_DIR + flpq-check-archive PARTIAL.tar.gz --partial + flpq-check-archive --audit --prefix 6.0.0/graph/ \ + --access-key-id KEY_ID --secret-access-key SECRET + +A **partial archive** provides new queries for an existing graph: it contains +only ``queries/`` — the new query directories plus a ``README.md`` fragment +with their sections. With ``--partial`` the tool checks that skeleton, that +every representation file parses (the label and dimension checks are skipped +— there is no graph to check against), and that the fragment describes +exactly the query directories it contains. + +The checks: + +- **Skeleton.** A single top-level directory named after the archive, + containing exactly ``README.md``, ``graph/`` (non-empty, only ``.mtx`` + files), and ``queries/`` with its ``README.md`` and the three class + directories. Each class directory holds query directories only (flat query + files are rejected); each query directory contains at least one + representation file with a class extension (``.cnf``/``.rsm`` for CFPQ, + ``.re``/``.rsm`` for RPQ, ``.mcfg`` for MCFPQ) and exactly one + ``results.mtx``. +- **Graph.** Every MTX file parses as a Boolean pattern matrix, every edge + endpoint fits the declared dimensions, and all label matrices declare the + same dimensions. +- **Labels.** No forbidden label components (``bar``, ``rev``); and reversed + edges are not stored — neither ``