diff --git a/.github/ISSUE_TEMPLATE/grammar-add-template.md b/.github/ISSUE_TEMPLATE/grammar-add-template.md deleted file mode 100644 index f649461b..00000000 --- a/.github/ISSUE_TEMPLATE/grammar-add-template.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -name: ADD NEW GRAMMAR -about: Suggest new grammar template to this project -title: "[NEW GRAMMAR] " -labels: new data -assignees: rustam-azimov, vadyushkins - ---- - - - -## Info -| | | -|-----------|------------------------------------------------------------------------| -| Full Name | ```` | -| Version | ```` | -| Class | ```` | -| Kind | ```` | -| Origin | [link](````) | - -## Grammar Parameters - -| Parameter | Description | -|:------------------------------------:|:---------------------------------------:| -| ```` | ```` | - -## Grammar Template - - - -## Grammar Description - - - -## Example Grammars - - - - diff --git a/.github/ISSUE_TEMPLATE/graph-add-template.md b/.github/ISSUE_TEMPLATE/graph-add-template.md deleted file mode 100644 index 2e0be260..00000000 --- a/.github/ISSUE_TEMPLATE/graph-add-template.md +++ /dev/null @@ -1,78 +0,0 @@ -name: ADD NEW GRAPH -about: Suggest new graph to this project -title: "[NEW GRAPH] " -labels: new data -assignees: rustam-azimov, vadyushkins - ---- - - - -## Info -| | | -|---|---| -| Type | ```` | -| Full Name | ```` | -| Category | ``" for a new category (new graphs only)>`` | -| Version | ```` | -| Archive | [link](``.tar.gz, full or partial>``) | - -## Data format - -The main way to provide data is a **Google Drive link** to an archive prepared per the structure-validation tool. Two kinds of archives exist (both documented in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/graphs/index.html)): - -- **Full archive** — a new graph: `.tar.gz` unpacks to a self-contained directory with `README.md`, `graph/` (one Boolean MatrixMarket pattern file per stored edge label), and `queries/` (one directory per query in `cfpq/`, `rpq/`, or `mcfpq/`; each query directory holds every representation of the query — `.cnf`/`.rsm`, `.re`/`.rsm` (regular only), or `.mcfg` — plus one `results.mtx` with the constrained reachability facts). -- **Partial archive** — new queries for an existing graph: `.tar.gz` unpacks to a directory containing only `queries/` — the new query directories (same layout) plus a `README.md` fragment with one `## /` section per new query. - -Validate the archive before sharing it: - -``` -python utils/check_archive_structure.py .tar.gz # full archive -python utils/check_archive_structure.py .tar.gz --partial # partial archive -``` - -A partial archive is merged into the existing graph archive by the maintainer (`python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`), re-validated, and uploaded. The query file formats — including the recursive state machine (`.rsm`) format — are documented on the [FLPQ design page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/flpq.html). - -## Description document - -**Full archive.** The archive's `README.md` must answer all of these questions (one `##` section each): -| | | -|---|---| -| What is this graph? | ```` | -| Source | ```` | -| Construction | ```` | -| Nodes | ```` | -| Edges and labels | ```` | -| Query classes | ```` | -| License | ```` | -| Caveats | ```` | - -**Partial archive.** Instead of a full description document, the `queries/README.md` fragment carries one `## /` section per new query, each describing the language (in words and/or in LaTeX) and its purpose on this graph. - -*New graphs only — a partial archive adds no nodes or edges.* - -## Graph Statistics -| Num Nodes | Num Edges | -|:---:|:---:| -| ```` | ```` | - -## Edges Statistics -| Edge Label | Num Edge Label | -|---:|---:| -| ```` | ```` | - -List the stored labels only (no `_r` reversed-edge rows). A family of indexed labels (e.g. `load_0`, ..., `load_857`) is listed once with a placeholder subscript (e.g. `load_f`) and a note listing the index set. - -## Queries - -Canonical grammars are documented on the category page, not on the per-graph page. Pick the case that applies: - -1. **Graph for an existing category** — name the grammar(s) from the category's "Canonical grammars" section that apply to this graph; they become the table column(s) filled in for the new row. No new grammar text on the site. -2. **New grammar for an existing category** — provide the grammar below; it will be added to the category page and a new column to its table. -3. **Graph for a new category** — provide the canonical grammar(s) below; a new category page will be created with them. - -Every query you provide must ship inside the archive (a full archive ships all queries of the graph; a partial archive ships the new ones): one directory per query in `queries/cfpq/`, `queries/rpq/`, or `queries/mcfpq/`, holding every representation of the query (`.cnf`/`.rsm`, `.re`/`.rsm` — regular only, or `.mcfg`) plus one `results.mtx` with the constrained reachability facts (a Boolean pattern matrix with the graph's dimensions; an entry `(i, j)` means some path from node `i` to node `j` satisfies the query). Each query directory is described in a `## /` section of `queries/README.md`. A query may only use labels of this graph. - - - - diff --git a/.github/ISSUE_TEMPLATE/new-data-template.md b/.github/ISSUE_TEMPLATE/new-data-template.md new file mode 100644 index 00000000..5480c612 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/new-data-template.md @@ -0,0 +1,123 @@ +--- +name: CONTRIBUTE NEW DATA +about: Suggest new data (graphs, queries, grammars) to this project +title: "[NEW DATA] " +labels: new data +assignees: rustam-azimov, vadyushkins + +--- + + + +## Contribution type + +Pick the case that applies to this issue (an issue may combine several archives of the same case): + +1. **New graph(s), existing category, existing grammar(s)** — full archive(s); the queries instantiate the category's canonical grammar(s). +2. **New graph(s), existing category, new grammar** — full archive(s) plus a grammar definition (see "Queries and grammars" below). +3. **New graph(s), new category** — full archive(s) plus the category name/description and its canonical grammar(s). +4. **New queries for existing graph(s)** — partial archive(s); sub-case 4b if a query uses a new grammar. + +## Info +| | | +|---|---| +| Version | ```` | +| Category | ``" (N/A for case 4)>`` | + +## Archives + +One row per archive. An issue may carry a **set** of archives (e.g. all graphs of a new category) — but one graph per archive: each `.tar.gz` unpacks to a single directory named after it. + +| Name | Kind | Target graph | Archive | SHA256 | +|---|---|---|---|---| +| ```` | ```` | ```` | [link](``.tar.gz>``) | ``.tar.gz`>`` | + +## Validation (mandatory, per archive) + +Validate every archive **before** sharing it and paste the output below. Without cloning the repo: `pip install flpq-data` — the checker is installed with the package as the `flpq-check-archive` command. + +### `.tar.gz` + +``` +flpq-check-archive .tar.gz # full archive +flpq-check-archive .tar.gz --partial # partial archive + structure ok +``` + +FLPQ_Data version used: ```` + +(One subsection per archive.) + +## Motivation (mandatory) + +```` + +## Reference results (mandatory) + +Every query directory ships one `results.mtx` with the constrained reachability facts — a Boolean pattern matrix with the graph's dimensions, where an entry `(i, j)` means some path from node `i` to node `j` satisfies the query. **Queries without reference results are not accepted.** Every new graph must ship at least one query (the structure checker refuses archives without one). + +## Data format + +Two kinds of archives exist (both documented in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/graphs/index.html)): + +- **Full archive** — a new graph: `.tar.gz` unpacks to a self-contained directory with `README.md`, `graph/` (one Boolean MatrixMarket pattern file per stored edge label), and `queries/` (one directory per query in `cfpq/`, `rpq/`, or `mcfpq/`; each query directory holds every representation of the query — `.cnf`/`.rsm`, `.re`/`.rsm` (regular only), or `.mcfg` — plus one `results.mtx`). +- **Partial archive** — new queries for an existing graph: `.tar.gz` unpacks to a directory named after that graph containing only `queries/` — the new query directories (same layout) plus a `README.md` fragment with one `## /` section per new query. Partial archives are merged into the existing graph archive by the maintainers. + +The query file formats — including the recursive state machine (`.rsm`) format — are documented on the [FLPQ design page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/flpq.html). + +## Description document + +**Full archive.** The archive's `README.md` must answer all of these questions (one `##` section each): +| | | +|---|---| +| What is this graph? | ```` | +| Source | ```` | +| Construction | ```` | +| Nodes | ```` | +| Edges and labels | ```` | +| Query classes | ```` | +| License | ```` | +| Caveats | ```` | + +**Partial archive.** Instead of a full description document, the `queries/README.md` fragment carries one `## /` section per new query, each describing the language (in words and/or in LaTeX) and its purpose on this graph. + +*New graphs only — a partial archive adds no nodes or edges.* + +## Graph statistics (new graphs only) + +One subsection per new graph: + +### `` + +| Num Nodes | Num Edges | +|:---:|:---:| +| ```` | ```` | + +| Edge Label | Num Edge Label | +|---:|---:| +| ```` | ```` | + +List the stored labels only (no `_r` reversed-edge rows). A family of indexed labels (e.g. `load_0`, ..., `load_857`) is listed once with a placeholder subscript (e.g. `load_f`) and a note listing the index set. + +## Queries and grammars + +Canonical grammars are documented on the category page, not on the per-graph page. Pick the case that applies: + +1. **Existing grammar(s)** — name the grammar(s) from the category's "Canonical grammars" section that apply; they become the table column(s) filled in for the new row(s). No new grammar text on the site. +2. **New grammar** — provide the grammar below; it will be added to the category page and a new column to its table. The same grammar must also be instantiated as at least one query directory with `results.mtx` in a contributed archive. +3. **New category** — provide the canonical grammar(s) below plus the category name and description; a new category page will be created with them. + +Every query you provide must ship inside an archive (a full archive ships all queries of the graph; a partial archive ships the new ones): one directory per query in `queries/cfpq/`, `queries/rpq/`, or `queries/mcfpq/`, holding every representation of the query plus one `results.mtx`. Each query directory is described in a `## /` section of `queries/README.md`. A query may only use labels of its graph. + + + + + + + +## Not accepted via this flow + +- A grammar or query provided outside an archive — without reference results. +- Updates to an existing graph (a new version of the source dataset). +- A new representation of an existing query (e.g. adding `.rsm` next to a `.cnf`-only query). +- More than one graph inside a single archive (use several archives in one issue instead). diff --git a/.github/PULL_REQUEST_TEMPLATE/new_grammar.md b/.github/PULL_REQUEST_TEMPLATE/new_grammar.md deleted file mode 100644 index e44382c9..00000000 --- a/.github/PULL_REQUEST_TEMPLATE/new_grammar.md +++ /dev/null @@ -1,30 +0,0 @@ - - -## Info -| | | -|-----------|------------------------------------------------------------------------| -| Full Name | ```` | -| Version | ```` | -| Class | ```` | -| Kind | ```` | -| Origin | [link](````) | - -## Grammar Parameters - -| Parameter | Description | -|:------------------------------------:|:---------------------------------------:| -| ```` | ```` | - -## Grammar Template - - - -## Grammar Description - - - -## Example Grammars - - - - diff --git a/.github/PULL_REQUEST_TEMPLATE/new_graph.md b/.github/PULL_REQUEST_TEMPLATE/new_graph.md deleted file mode 100644 index 1d5bc96d..00000000 --- a/.github/PULL_REQUEST_TEMPLATE/new_graph.md +++ /dev/null @@ -1,70 +0,0 @@ - - -## Info -| | | -|---|---| -| Type | ```` | -| Full Name | ```` | -| Category | ``" for a new category (new graphs only)>`` | -| Version | ```` | -| Archive | [link](``.tar.gz, full or partial>``) | - -## Data format - -The main way to provide data is a **Google Drive link** to an archive prepared per the structure-validation tool. Two kinds of archives exist (both documented in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/graphs/index.html)): - -- **Full archive** — a new graph: `.tar.gz` unpacks to a self-contained directory with `README.md`, `graph/` (one Boolean MatrixMarket pattern file per stored edge label), and `queries/` (one directory per query in `cfpq/`, `rpq/`, or `mcfpq/`; each query directory holds every representation of the query — `.cnf`/`.rsm`, `.re`/`.rsm` (regular only), or `.mcfg` — plus one `results.mtx` with the constrained reachability facts). -- **Partial archive** — new queries for an existing graph: `.tar.gz` unpacks to a directory containing only `queries/` — the new query directories (same layout) plus a `README.md` fragment with one `## /` section per new query. - -Validate the archive before sharing it: - -``` -python utils/check_archive_structure.py .tar.gz # full archive -python utils/check_archive_structure.py .tar.gz --partial # partial archive -``` - -A partial archive is merged into the existing graph archive by the maintainer (`python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`), re-validated, and uploaded. The query file formats — including the recursive state machine (`.rsm`) format — are documented on the [FLPQ design page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/flpq.html). - -## Description document - -**Full archive.** The archive's `README.md` must answer all of these questions (one `##` section each): -| | | -|---|---| -| What is this graph? | ```` | -| Source | ```` | -| Construction | ```` | -| Nodes | ```` | -| Edges and labels | ```` | -| Query classes | ```` | -| License | ```` | -| Caveats | ```` | - -**Partial archive.** Instead of a full description document, the `queries/README.md` fragment carries one `## /` section per new query, each describing the language (in words and/or in LaTeX) and its purpose on this graph. - -*New graphs only — a partial archive adds no nodes or edges.* - -## Graph Statistics -| Num Nodes | Num Edges | -|:---:|:---:| -| ```` | ```` | - -## Edges Statistics -| Edge Label | Num Edge Label | -|---:|---:| -| ```` | ```` | - -List the stored labels only (no `_r` reversed-edge rows). A family of indexed labels (e.g. `load_0`, ..., `load_857`) is listed once with a placeholder subscript (e.g. `load_f`) and a note listing the index set. - -## Queries - -Canonical grammars are documented on the category page, not on the per-graph page. Pick the case that applies: - -1. **Graph for an existing category** — name the grammar(s) from the category's "Canonical grammars" section that apply to this graph; they become the table column(s) filled in for the new row. No new grammar text on the site. -2. **New grammar for an existing category** — provide the grammar below; it will be added to the category page and a new column to its table. -3. **Graph for a new category** — provide the canonical grammar(s) below; a new category page will be created with them. - -Every query you provide must ship inside the archive (a full archive ships all queries of the graph; a partial archive ships the new ones): one directory per query in `queries/cfpq/`, `queries/rpq/`, or `queries/mcfpq/`, holding every representation of the query (`.cnf`/`.rsm`, `.re`/`.rsm` — regular only, or `.mcfg`) plus one `results.mtx` with the constrained reachability facts (a Boolean pattern matrix with the graph's dimensions; an entry `(i, j)` means some path from node `i` to node `j` satisfies the query). Each query directory is described in a `## /` section of `queries/README.md`. A query may only use labels of this graph. - - - - diff --git a/.github/PULL_REQUEST_TEMPLATE/pull_request.md b/.github/PULL_REQUEST_TEMPLATE/pull_request.md new file mode 100644 index 00000000..8b007aae --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/pull_request.md @@ -0,0 +1,15 @@ + + +## Description + + + +## Linked issue + +Closes # + +## Checklist + +- [ ] Quality gate passes (tests, style, type check, docs build) +- [ ] Documentation updated per the documentation conventions +- [ ] No secrets or credentials included diff --git a/.opencode/skills/add-grammar/SKILL.md b/.opencode/skills/add-grammar/SKILL.md index 2bcb9d76..9549a82e 100644 --- a/.opencode/skills/add-grammar/SKILL.md +++ b/.opencode/skills/add-grammar/SKILL.md @@ -1,19 +1,29 @@ --- name: add-grammar -description: Use when adding a new grammar template to FLPQ_Data. Covers the generator/readwrite/converter module conventions and the PR/issue templates. +description: Use when adding a new grammar template to FLPQ_Data. Covers the generator/readwrite/converter module conventions and the issue template. --- # Add a grammar template ## Source of truth -The required fields and metadata live in the GitHub templates — do **not** -duplicate them here: +The required fields and metadata live in the GitHub issue template — do +**not** duplicate them here: -- `.github/PULL_REQUEST_TEMPLATE/new_grammar.md` -- `.github/ISSUE_TEMPLATE/grammar-add-template.md` +- `.github/ISSUE_TEMPLATE/new-data-template.md` (the "Queries and grammars" + section; a grammar arrives only together with a data contribution) -Fill in every field in the triangle brackets (`<>`) of those templates. +Fill in every field in the triangle brackets (`<>`) of that template. + +## Where grammars come from + +Grammars arrive via cases 2, 3, or 4b of a CONTRIBUTE NEW DATA issue: the +issue carries the grammar definition (LaTeX plus file-format text), and the +same grammar is instantiated as at least one query directory with +`results.mtx` in a contributed archive. When the grammar is parameterized, +implement its generator module under `flpq_data/queries//generators/` +per the package conventions below; the docs wiring (category page, table +column) follows the "Documentation" section of the `add-graph` skill. ## Package conventions diff --git a/.opencode/skills/add-graph/SKILL.md b/.opencode/skills/add-graph/SKILL.md index ff9d2576..3aacb41e 100644 --- a/.opencode/skills/add-graph/SKILL.md +++ b/.opencode/skills/add-graph/SKILL.md @@ -1,34 +1,61 @@ --- name: add-graph -description: Use when adding a new graph to FLPQ_Data. Covers the mtx-per-label archive format, the registry.json wiring, and the PR/issue templates. +description: Use when adding a new graph to FLPQ_Data. Covers the mtx-per-label archive format, accepting external contributions (the issue-first pipeline), and the registry.json wiring. --- # Add a graph to the dataset ## Source of truth -The required fields and metadata live in the GitHub templates — do **not** -duplicate them here: +The required fields and metadata live in the GitHub issue template — do +**not** duplicate them here: -- `.github/PULL_REQUEST_TEMPLATE/new_graph.md` -- `.github/ISSUE_TEMPLATE/graph-add-template.md` +- `.github/ISSUE_TEMPLATE/new-data-template.md` (data contributions are + issue-first; there is no PR path for data) -Fill in every field in the triangle brackets (`<>`) of those templates. +Fill in every field in the triangle brackets (`<>`) of that template. ## Data format The archive layout and the label conventions are documented once in the "File structure" section of `docs/graphs/index.rst` — follow it exactly. -Pack every query of the graph into the archive per the "Queries" section of -the PR template, then validate with -`python utils/check_archive_structure.py .tar.gz` (the upload tool -runs the same check and refuses invalid archives). +Pack every query of the graph into the archive per the "Queries and +grammars" section of the issue template, then validate with +`flpq-check-archive .tar.gz` (the upload tool runs the same check +and refuses invalid archives). New queries for an existing graph ship as a **partial archive** instead — -see the "Data format" section of the PR template; it is validated with +see the "Data format" section of the issue template; it is validated with `--partial` and merged into the existing archive with `python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`. +## Accepting an external contribution + +Data arrives as a CONTRIBUTE NEW DATA issue carrying one or more archives. +Per archive, in order: + +1. Download it from the Drive link in the issue. +2. Verify its SHA256 against the value in the issue — a mismatch means the + shared file is not the validated one; stop and ask the contributor. +3. Re-run `flpq-check-archive .tar.gz` (add `--partial` for partial + archives) — it must pass. +4. Check the graph name against `flpq_data/dataset/registry.json` and + `flpq_data/dataset/reachable_pairs.csv`: a full archive's name must be + new, a partial archive's target must exist. +5. Add its reachable-pairs rows with + `python utils/add_reachable_pairs.py .tar.gz --graph NAME --category CAT [--partial]` + (it re-validates the archive and refuses duplicates). +6. Put it in place: a full archive is uploaded to the dataset URL (see + "Wiring a new graph into the code"); a partial archive is merged with + `python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz` + and the result uploaded under the graph's name. + +Once per issue, after every archive is in: regenerate the registry +(`python utils/generate_registry.py`), do the "Documentation" steps (per-graph +pages, category table(s), canonical grammars, new category page), then run +`python utils/reachable_pairs_tables.py --update` and +`python utils/archive_sizes.py --update`. + ## Wiring a new graph into the code 1. The graph archive must be uploaded under the dataset URL @@ -44,11 +71,11 @@ see the "Data format" section of the PR template; it is validated with 1. Create the per-graph page `docs/graphs/data/.rst`, following the structure of an existing page (description, statistics tables). The page - carries **no** grammar section: canonical grammars are documented once on - the category page. Statistics conventions: the PR template is the source - of truth. -2. Handle the grammar case from the PR template's "Canonical grammars" - section: + carries **no** grammar section: canonical grammars are documented once on + the category page. Statistics conventions: the issue template is the + source of truth. +2. Handle the grammar case from the issue template's "Queries and grammars" + section: - *existing category, existing grammar(s)* — nothing to add; the new table row fills in the matching column(s); - *new grammar for an existing category* — append the grammar to the diff --git a/AGENTS.md b/AGENTS.md index 8b4af487..4dd825c0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,8 +25,8 @@ gate → merge, and points to the other workflow skills below. code: design decisions, non-trivial constraints. Skills stay thin pointers to docs, code, or CI — they never re-describe them. * Data archives are self-contained and must align with the checker - (`utils/check_archive_structure.py`) — the source of truth for any data - structure. + (`flpq_data/tools/check_archive_structure.py`, console command + `flpq-check-archive`) — the source of truth for any data structure. * Tools, not instructions. If you can do something with existing tool --- do it. No thinking, no long instructions, no manual analysis. You want to analyze code coverage? Just run coverage tool and analyze report. No workaround for regular tasks. If there is a tool for regular task it must be installed and configured appropriately. * Always learn, never forget — encode patterns before session ends @@ -39,6 +39,8 @@ gate → merge, and points to the other workflow skills below. - `flpq_data/queries/` — per-class query modules: `cfpq/` (`generators`, `readwrite`, `converters`, `utils`), `rpq/` (`readwrite`), `mcfpq/` (`readwrite`). +- `flpq_data/tools/` — the archive structure checker (console command + `flpq-check-archive`). - `tests/` mirrors the package structure (pytest + doctests). ## Skills diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ed9ee26..ebdfbf0b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [6.1.0] - 2026-10-14 + +### Added + +- The `flpq-check-archive` console command (installed with the package): + validates a graph archive's structure, so contributors check their + archives with `pip install flpq-data` alone — no script download. +- `utils/add_reachable_pairs.py`: derives the reachable-pairs rows of one + validated archive (one row per query directory, with the declared entry + count of its `results.mtx` as `num_reachable_pairs`) and inserts them into + `flpq_data/dataset/reachable_pairs.csv`, preserving the file's sort order + and refusing duplicate rows and category disagreements. + +### Changed + +- The archive structure checker moved from `utils/check_archive_structure.py` + to `flpq_data/tools/` (console command `flpq-check-archive`); the S3 client + factory and bucket constants moved to `flpq_data.dataset.s3`. + +- Data contributions are issue-first: a single "CONTRIBUTE NEW DATA" issue + carries one or more validated archives (one graph each), the per-archive + validation output and SHA256, and the motivation; maintainers upload and + merge the data into the dataset. The Graphs page documents the flow, the + four contribution cases, and the consistency invariants — no graph without + a query, no query without reference results, no item without description + and motivation. + +### Removed + +- The `ADD NEW GRAPH` / `ADD NEW GRAMMAR` issue templates and the + `new_graph` / `new_grammar` PR templates: data has no PR path, and a + grammar is accepted only together with the archive that instantiates it as + a query with reference results. A generic PR template covers code + contributions. + ## [6.0.0] - 2026-10-13 ### Added diff --git a/README.rst b/README.rst index daed4bf8..1b1d021a 100644 --- a/README.rst +++ b/README.rst @@ -85,13 +85,10 @@ The full developer guide — development setup, quality checks, the test pipeline, docs build and deployment, the package release process, and contribution guidelines — is on the `Developer `_ page. -How to add a new graph? -*********************** +How to contribute new data? +*************************** -Just create - -- an ``Issue`` corresponding to the `"Issue template for adding a new graph" `_. -- a ``Pull Request`` corresponding to the `"Pull request template for adding a new graph" `_. +Create an ``Issue`` corresponding to the `"CONTRIBUTE NEW DATA" issue template `_ — one or more validated archives (one graph each), links to them, and the motivation. The maintainers upload and merge the data into the dataset; the full contribution guide is on the `Graphs `_ page. Citation ******** diff --git a/docs/dataset.rst b/docs/dataset.rst index 757934de..2a372c9b 100644 --- a/docs/dataset.rst +++ b/docs/dataset.rst @@ -26,4 +26,4 @@ list of applicable graphs. How to add a new grammar? ------------------------- -Just create a PR (Pull Request) corresponding to the `"Template for adding a new grammar" `_. +Grammars are added together with the data they apply to — create an ``Issue`` corresponding to the `"CONTRIBUTE NEW DATA" issue template `_ (the "Queries and grammars" section). A grammar is accepted only if it is instantiated as at least one query with reference results in a contributed archive. diff --git a/docs/flpq.rst b/docs/flpq.rst index bc1cad6f..58b36dcf 100644 --- a/docs/flpq.rst +++ b/docs/flpq.rst @@ -419,7 +419,7 @@ Migration path: ``queries/cfpq//`` directory each, add the per-query ``results.mtx`` (the constrained reachability facts), write the mandatory-question ``README.md``, and validate with - ``utils/check_archive_structure.py`` (:ref:`archive_structure`). + ``flpq-check-archive`` (:ref:`archive_structure`). - The example query archives (``4.0.0/grammar/example/``) are dropped — every query file lives inside a graph archive. - Re-point ``DATASET_URL`` at the new prefix as part of the version bump diff --git a/docs/graphs/index.rst b/docs/graphs/index.rst index 8afe170b..872717a7 100644 --- a/docs/graphs/index.rst +++ b/docs/graphs/index.rst @@ -9,10 +9,58 @@ Graphs :Release: |release| :Date: |today| -How to add a new graph? ------------------------ +.. _contributing_data: -Just create a PR (Pull Request) corresponding to the `"Template for adding a new graph" `_. +Contributing data +----------------- + +New data — graphs, queries, grammars — enters the dataset through a single +`"CONTRIBUTE NEW DATA" issue `_: +you prepare one or more archives (one graph each), validate them, publish +them (e.g. to Google Drive), and open the issue with the links, the +validation output, the SHA256 of every archive, and the motivation. The +maintainers upload the archives and merge them into the dataset — no pull +request is needed. + +Contribution cases +^^^^^^^^^^^^^^^^^^ + +The issue template's "Contribution type" section lists the four contribution +cases — new graph(s) for an existing or a new category, with an existing or +a new grammar, and new queries for existing graph(s) — and what each +requires; its remaining sections carry every required field (statistics, +grammar definitions, category description). A set of graphs for a category +goes into one issue as several archives (one graph per archive). + +Validation +^^^^^^^^^^ + +Every archive must pass the structure checker before it is shared:: + + flpq-check-archive .tar.gz # full archive + flpq-check-archive .tar.gz --partial # partial archive + +The issue carries the pasted output of this command per archive, the +archive's SHA256, and the FLPQ_Data version used; the template has the +recipe — ``pip install flpq-data`` installs the checker. + +Consistency invariants +^^^^^^^^^^^^^^^^^^^^^^ + +- No graph without at least one query. +- No query without reference results (``results.mtx``). +- No item without description and motivation: every graph is described in + its archive ``README.md``, every query in ``queries/README.md``, and the + issue states why each new graph, grammar, or category is useful to other + users. + +Not accepted via this flow +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The issue template's "Not accepted via this flow" section lists the declined +cases: a grammar or query provided outside an archive (without reference +results), updates to an existing graph, a new representation of an existing +query, and more than one graph inside a single archive. .. _graph_file_structure: @@ -64,8 +112,8 @@ A **partial archive** provides new queries for an existing graph: it unpacks to a directory named after that graph and contains only ``queries/`` — the new query directories (same layout) plus a ``README.md`` fragment with one section per new query. It is validated with -``python utils/check_archive_structure.py .tar.gz --partial`` and -merged into the existing archive with :ref:`merge_archive`. +``flpq-check-archive .tar.gz --partial`` and merged into the existing +archive with :ref:`merge_archive`. The pre-migration graphs on the :ref:`old_graphs` page use the old CSV format instead. diff --git a/docs/reference/index.rst b/docs/reference/index.rst index fa70a98f..84761836 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -25,3 +25,4 @@ Reference dataset/index graphs/index queries/index + tools/index diff --git a/docs/reference/tools/index.rst b/docs/reference/tools/index.rst new file mode 100644 index 00000000..0de8fc56 --- /dev/null +++ b/docs/reference/tools/index.rst @@ -0,0 +1,82 @@ +.. _tools: + +.. currentmodule:: flpq_data + +***** +Tools +***** + +.. only:: html + + :Release: |release| + :Date: |today| + +.. automodule:: flpq_data.tools +.. autosummary:: + :toctree: generated/ + + MANDATORY_README_SECTIONS + QUERY_CLASSES + parse_readme_sections + stored_reverse_forward + unpack_single_dir + validate_archive + +.. _archive_structure: + +Archive structure checker +------------------------- + +``flpq-check-archive`` (installed with the package) validates that a graph +archive has the fixed self-contained structure documented in the "File +structure" section of the :ref:`graphs` page:: + + flpq-check-archive ARCHIVE.tar.gz + flpq-check-archive UNPACKED_DIR + flpq-check-archive PARTIAL.tar.gz --partial + flpq-check-archive --audit --prefix 6.0.0/graph/ \ + --access-key-id KEY_ID --secret-access-key SECRET + +A **partial archive** provides new queries for an existing graph: it contains +only ``queries/`` — the new query directories plus a ``README.md`` fragment +with their sections. With ``--partial`` the tool checks that skeleton, that +every representation file parses (the label and dimension checks are skipped +— there is no graph to check against), and that the fragment describes +exactly the query directories it contains. + +The checks: + +- **Skeleton.** A single top-level directory named after the archive, + containing exactly ``README.md``, ``graph/`` (non-empty, only ``.mtx`` + files), and ``queries/`` with its ``README.md`` and the three class + directories. Each class directory holds query directories only (flat query + files are rejected); each query directory contains at least one + representation file with a class extension (``.cnf``/``.rsm`` for CFPQ, + ``.re``/``.rsm`` for RPQ, ``.mcfg`` for MCFPQ) and exactly one + ``results.mtx``. +- **Graph.** Every MTX file parses as a Boolean pattern matrix, every edge + endpoint fits the declared dimensions, and all label matrices declare the + same dimensions. +- **Labels.** No forbidden label components (``bar``, ``rev``); and reversed + edges are not stored — neither ``