diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index af3353bf..87a5bd14 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -25,7 +25,7 @@ If applicable, add screenshots to help explain your problem. **Desktop (please complete the following information):** - OS: `[e.g. Ubuntu 20.04]` - - CFPQ_Data: `[e.g. 22]` + - FLPQ_Data: `[e.g. 22]` - Python: `[e.g. 3.8]` - Other important stuff versions diff --git a/.github/ISSUE_TEMPLATE/grammar-add-template.md b/.github/ISSUE_TEMPLATE/grammar-add-template.md index c2c2bd51..f649461b 100644 --- a/.github/ISSUE_TEMPLATE/grammar-add-template.md +++ b/.github/ISSUE_TEMPLATE/grammar-add-template.md @@ -13,7 +13,7 @@ assignees: rustam-azimov, vadyushkins | | | |-----------|------------------------------------------------------------------------| | Full Name | ```` | -| Version | ```` | +| Version | ```` | | Class | ```` | | Kind | ```` | | Origin | [link](````) | diff --git a/.github/ISSUE_TEMPLATE/graph-add-template.md b/.github/ISSUE_TEMPLATE/graph-add-template.md index cbc95800..2e0be260 100644 --- a/.github/ISSUE_TEMPLATE/graph-add-template.md +++ b/.github/ISSUE_TEMPLATE/graph-add-template.md @@ -1,4 +1,3 @@ ---- name: ADD NEW GRAPH about: Suggest new graph to this project title: "[NEW GRAPH] " @@ -12,13 +11,45 @@ assignees: rustam-azimov, vadyushkins ## Info | | | |---|---| -| Full Name | ```` | -| Version | ```` | -| Origin | [link](````) | +| Type | ```` | +| Full Name | ```` | +| Category | ``" for a new category (new graphs only)>`` | +| Version | ```` | +| Archive | [link](``.tar.gz, full or partial>``) | ## Data format -The archive must follow the standard layout described in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/CFPQ_Data/graphs/index.html): `.tar.gz` unpacks to `/{README.md, grammar/, graph/}`, where `graph/` holds one Boolean MatrixMarket pattern file per edge label. +The main way to provide data is a **Google Drive link** to an archive prepared per the structure-validation tool. Two kinds of archives exist (both documented in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/graphs/index.html)): + +- **Full archive** — a new graph: `.tar.gz` unpacks to a self-contained directory with `README.md`, `graph/` (one Boolean MatrixMarket pattern file per stored edge label), and `queries/` (one directory per query in `cfpq/`, `rpq/`, or `mcfpq/`; each query directory holds every representation of the query — `.cnf`/`.rsm`, `.re`/`.rsm` (regular only), or `.mcfg` — plus one `results.mtx` with the constrained reachability facts). +- **Partial archive** — new queries for an existing graph: `.tar.gz` unpacks to a directory containing only `queries/` — the new query directories (same layout) plus a `README.md` fragment with one `## /` section per new query. + +Validate the archive before sharing it: + +``` +python utils/check_archive_structure.py .tar.gz # full archive +python utils/check_archive_structure.py .tar.gz --partial # partial archive +``` + +A partial archive is merged into the existing graph archive by the maintainer (`python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`), re-validated, and uploaded. The query file formats — including the recursive state machine (`.rsm`) format — are documented on the [FLPQ design page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/flpq.html). + +## Description document + +**Full archive.** The archive's `README.md` must answer all of these questions (one `##` section each): +| | | +|---|---| +| What is this graph? | ```` | +| Source | ```` | +| Construction | ```` | +| Nodes | ```` | +| Edges and labels | ```` | +| Query classes | ```` | +| License | ```` | +| Caveats | ```` | + +**Partial archive.** Instead of a full description document, the `queries/README.md` fragment carries one `## /` section per new query, each describing the language (in words and/or in LaTeX) and its purpose on this graph. + +*New graphs only — a partial archive adds no nodes or edges.* ## Graph Statistics | Num Nodes | Num Edges | @@ -28,12 +59,20 @@ The archive must follow the standard layout described in the "File structure" se ## Edges Statistics | Edge Label | Num Edge Label | |---:|---:| -| ```` | ```` | +| ```` | ```` | List the stored labels only (no `_r` reversed-edge rows). A family of indexed labels (e.g. `load_0`, ..., `load_857`) is listed once with a placeholder subscript (e.g. `load_f`) and a note listing the index set. -## Canonical grammars +## Queries + +Canonical grammars are documented on the category page, not on the per-graph page. Pick the case that applies: + +1. **Graph for an existing category** — name the grammar(s) from the category's "Canonical grammars" section that apply to this graph; they become the table column(s) filled in for the new row. No new grammar text on the site. +2. **New grammar for an existing category** — provide the grammar below; it will be added to the category page and a new column to its table. +3. **Graph for a new category** — provide the canonical grammar(s) below; a new category page will be created with them. + +Every query you provide must ship inside the archive (a full archive ships all queries of the graph; a partial archive ships the new ones): one directory per query in `queries/cfpq/`, `queries/rpq/`, or `queries/mcfpq/`, holding every representation of the query (`.cnf`/`.rsm`, `.re`/`.rsm` — regular only, or `.mcfg`) plus one `results.mtx` with the constrained reachability facts (a Boolean pattern matrix with the graph's dimensions; an entry `(i, j)` means some path from node `i` to node `j` satisfies the query). Each query directory is described in a `## /` section of `queries/README.md`. A query may only use labels of this graph. - + - + diff --git a/.github/PULL_REQUEST_TEMPLATE/new_grammar.md b/.github/PULL_REQUEST_TEMPLATE/new_grammar.md index 060fc2a8..e44382c9 100644 --- a/.github/PULL_REQUEST_TEMPLATE/new_grammar.md +++ b/.github/PULL_REQUEST_TEMPLATE/new_grammar.md @@ -4,7 +4,7 @@ | | | |-----------|------------------------------------------------------------------------| | Full Name | ```` | -| Version | ```` | +| Version | ```` | | Class | ```` | | Kind | ```` | | Origin | [link](````) | diff --git a/.github/PULL_REQUEST_TEMPLATE/new_graph.md b/.github/PULL_REQUEST_TEMPLATE/new_graph.md index 3c65a224..1d5bc96d 100644 --- a/.github/PULL_REQUEST_TEMPLATE/new_graph.md +++ b/.github/PULL_REQUEST_TEMPLATE/new_graph.md @@ -3,13 +3,45 @@ ## Info | | | |---|---| -| Full Name | ```` | -| Version | ```` | -| Origin | [link](````) | +| Type | ```` | +| Full Name | ```` | +| Category | ``" for a new category (new graphs only)>`` | +| Version | ```` | +| Archive | [link](``.tar.gz, full or partial>``) | ## Data format -The archive must follow the standard layout described in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/CFPQ_Data/graphs/index.html): `.tar.gz` unpacks to `/{README.md, grammar/, graph/}`, where `graph/` holds one Boolean MatrixMarket pattern file per edge label. +The main way to provide data is a **Google Drive link** to an archive prepared per the structure-validation tool. Two kinds of archives exist (both documented in the "File structure" section of the [Graphs page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/graphs/index.html)): + +- **Full archive** — a new graph: `.tar.gz` unpacks to a self-contained directory with `README.md`, `graph/` (one Boolean MatrixMarket pattern file per stored edge label), and `queries/` (one directory per query in `cfpq/`, `rpq/`, or `mcfpq/`; each query directory holds every representation of the query — `.cnf`/`.rsm`, `.re`/`.rsm` (regular only), or `.mcfg` — plus one `results.mtx` with the constrained reachability facts). +- **Partial archive** — new queries for an existing graph: `.tar.gz` unpacks to a directory containing only `queries/` — the new query directories (same layout) plus a `README.md` fragment with one `## /` section per new query. + +Validate the archive before sharing it: + +``` +python utils/check_archive_structure.py .tar.gz # full archive +python utils/check_archive_structure.py .tar.gz --partial # partial archive +``` + +A partial archive is merged into the existing graph archive by the maintainer (`python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`), re-validated, and uploaded. The query file formats — including the recursive state machine (`.rsm`) format — are documented on the [FLPQ design page](https://formallanguageconstrainedpathquerying.github.io/FLPQ_Data/flpq.html). + +## Description document + +**Full archive.** The archive's `README.md` must answer all of these questions (one `##` section each): +| | | +|---|---| +| What is this graph? | ```` | +| Source | ```` | +| Construction | ```` | +| Nodes | ```` | +| Edges and labels | ```` | +| Query classes | ```` | +| License | ```` | +| Caveats | ```` | + +**Partial archive.** Instead of a full description document, the `queries/README.md` fragment carries one `## /` section per new query, each describing the language (in words and/or in LaTeX) and its purpose on this graph. + +*New graphs only — a partial archive adds no nodes or edges.* ## Graph Statistics | Num Nodes | Num Edges | @@ -19,12 +51,20 @@ The archive must follow the standard layout described in the "File structure" se ## Edges Statistics | Edge Label | Num Edge Label | |---:|---:| -| ```` | ```` | +| ```` | ```` | List the stored labels only (no `_r` reversed-edge rows). A family of indexed labels (e.g. `load_0`, ..., `load_857`) is listed once with a placeholder subscript (e.g. `load_f`) and a note listing the index set. -## Canonical grammars +## Queries + +Canonical grammars are documented on the category page, not on the per-graph page. Pick the case that applies: + +1. **Graph for an existing category** — name the grammar(s) from the category's "Canonical grammars" section that apply to this graph; they become the table column(s) filled in for the new row. No new grammar text on the site. +2. **New grammar for an existing category** — provide the grammar below; it will be added to the category page and a new column to its table. +3. **Graph for a new category** — provide the canonical grammar(s) below; a new category page will be created with them. + +Every query you provide must ship inside the archive (a full archive ships all queries of the graph; a partial archive ships the new ones): one directory per query in `queries/cfpq/`, `queries/rpq/`, or `queries/mcfpq/`, holding every representation of the query (`.cnf`/`.rsm`, `.re`/`.rsm` — regular only, or `.mcfg`) plus one `results.mtx` with the constrained reachability facts (a Boolean pattern matrix with the graph's dimensions; an entry `(i, j)` means some path from node `i` to node `j` satisfies the query). Each query directory is described in a `## /` section of `queries/README.md`. A query may only use labels of this graph. - + - + diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 1b87bb21..b32f86d4 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -2,6 +2,7 @@ version: 2 updates: - package-ecosystem: "pip" directory: "/" + target-branch: "dev" schedule: interval: "monthly" ignore: diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 2ef91bc4..aa83ab50 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -11,30 +11,22 @@ jobs: steps: - uses: actions/checkout@v7 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v7 + - name: Set up uv with Python ${{ matrix.python-version }} + uses: astral-sh/setup-uv@v10.1.0 with: python-version: ${{ matrix.python-version }} - - name: Install Poetry - run: | - pip install poetry - - - name: Install dependencies with Poetry - run: | - poetry install --no-interaction --no-ansi --with dev,test + # Why these flags: see "Development setup" in docs/developer.rst. + - name: Install dependencies with uv + run: uv sync --frozen --all-groups - name: Show installed packages - run: poetry show - - - name: Install packages - run: | - python -m pip install . - + run: uv tree - - name: Test CFPQ_Data with coverage - run: | - poetry run pytest --cov=cfpq_data --doctest-modules -vv -s cfpq_data tests + # Per-metric gate rationale: see "Test pipeline" in docs/developer.rst. + # term-missing keeps the uncovered lines in the CI log. + - name: Test FLPQ_Data with coverage (line and branch >= 95%) + run: uv run pytest --cov=flpq_data --cov-report=term-missing --cov-report=json && uv run python utils/check_coverage.py - name: Upload coverage to Codecov uses: codecov/codecov-action@v7 diff --git a/.github/workflows/deploy_docs.yml b/.github/workflows/deploy_docs.yml index 8b7babc1..eadc7e68 100644 --- a/.github/workflows/deploy_docs.yml +++ b/.github/workflows/deploy_docs.yml @@ -15,28 +15,20 @@ jobs: steps: - uses: actions/checkout@v7 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v7 + - name: Set up uv with Python ${{ matrix.python-version }} + uses: astral-sh/setup-uv@v10.1.0 with: python-version: ${{ matrix.python-version }} - - name: Install Poetry - run: | - pip install poetry - - - name: Install dependencies with Poetry - run: | - poetry install --no-interaction --no-ansi --with docs + # Why these flags: see "Development setup" in docs/developer.rst. + - name: Install dependencies with uv + run: uv sync --frozen --all-groups - name: Show installed packages - run: poetry show - - - name: Install packages - run: | - poetry run pip install . + run: uv tree - name: Build - run: poetry run make -C docs/ html + run: uv run make -C docs/ html - name: Deploy uses: JamesIves/github-pages-deploy-action@v4 diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 8da9e03f..176a534c 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -11,31 +11,22 @@ jobs: steps: - uses: actions/checkout@v7 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v7 + - name: Set up uv with Python ${{ matrix.python-version }} + uses: astral-sh/setup-uv@v10.1.0 with: python-version: ${{ matrix.python-version }} - - name: Install Poetry - run: | - pip install poetry - - - name: Install dependencies with Poetry - run: | - poetry install --no-interaction --no-ansi --with docs + # Why these flags: see "Development setup" in docs/developer.rst. + - name: Install dependencies with uv + run: uv sync --frozen --all-groups - name: Show installed packages - run: poetry show - - - name: Install packages - run: poetry run pip install . + run: uv tree - # No-warnings policy: the build runs with -W --keep-going (see - # docs/Makefile), so any warning — including an unresolved - # cross-reference under nitpicky — exits non-zero on its own. + # No-warnings policy: see "Docs build and deploy" in docs/developer.rst. - name: Build - run: poetry run make -C docs/ html + run: uv run make -C docs/ html - # The linkcheck builder exits non-zero on broken or timed-out links. + # Link check policy: see "Docs build and deploy" in docs/developer.rst. - name: Check links - run: poetry run sphinx-build -b linkcheck docs docs/_build/linkcheck + run: uv run sphinx-build -b linkcheck docs docs/_build/linkcheck diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index e3a80a63..6ae3087d 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -7,20 +7,28 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [ 3.9 ] + python-version: [ 3.11 ] steps: - uses: actions/checkout@v7 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v7 + - name: Set up uv with Python ${{ matrix.python-version }} + uses: astral-sh/setup-uv@v10.1.0 with: python-version: ${{ matrix.python-version }} - - name: Install pre-commit - run: | - python -m pip install pre-commit==3.6.2 - pre-commit install + # Why these flags: see "Development setup" in docs/developer.rst. + - name: Install dependencies with uv + run: uv sync --frozen --all-groups - name: Run pre-commit run: | - pre-commit run --all-files --color always --verbose --show-diff-on-failure + uv run pre-commit run --all-files --color always --verbose --show-diff-on-failure + + # The pre-commit pass already runs ruff and ty as hooks; these steps + # make the type checks explicit — Pyright is the stricter checker and + # only runs here. + - name: Type check with ty + run: uv run ty check + + - name: Type check with Pyright + run: uv run pyright diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index a7b23be3..b0ea9339 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -4,7 +4,10 @@ on: push: tags: ['v*'] # Packaging check for every PR targeting the release branch: build the - # sdist/wheel and publish to TestPyPI before a release tag is cut. + # sdist/wheel and publish to TestPyPI before a release tag is cut. Each PR + # run rewrites the version to .dev so every push publishes + # a distinct artifact (a fixed version would be skipped after the first + # upload). Tag pushes publish the committed version unchanged. pull_request: branches: [master] @@ -18,7 +21,7 @@ jobs: steps: - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: astral-sh/setup-uv@v10.1.0 with: python-version: '3.11' @@ -34,10 +37,27 @@ jobs: fi echo "Tag ${GITHUB_REF_NAME} matches package version ${ver}" - - name: Build distributions + # TestPyPI only: give this PR run a unique developmental version so + # repeated pushes produce new artifacts instead of being skipped. The + # committed sources on the runner are rewritten, never committed. A + # commit-hash suffix is impossible here (PEP 440 forbids local version + # identifiers on PyPI), so the SHA goes to the run summary instead. + - name: Set TestPyPI dev version + if: github.event_name == 'pull_request' run: | - python3 -m pip install --upgrade build - python3 -m build + set -euo pipefail + base="$(uv version --short)" + dev="${base}.dev${{ github.run_number }}" + python utils/set_dev_version.py "$dev" + { + echo "### TestPyPI pre-publish" + echo "" + echo "- version: \`${dev}\`" + echo "- commit: \`${{ github.sha }}\`" + } >> "$GITHUB_STEP_SUMMARY" + + - name: Build distributions + run: uv build - name: Upload dist uses: actions/upload-artifact@v7 @@ -62,8 +82,9 @@ jobs: repository-url: https://test.pypi.org/legacy/ # No `password`: uses OIDC Trusted Publishing configured on # TestPyPI for the pull_request subject claim. - # Concurrent PRs share the same package version and TestPyPI - # rejects re-uploads of existing files; skip instead of failing. + # Each run carries a unique .dev version, so this is a + # safety net rather than a workaround: TestPyPI rejects re-uploads + # of existing files, and a collision should stay green (not fail). skip-existing: true publish-pypi: diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 2cbd38dc..d4492e60 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -16,26 +16,17 @@ jobs: steps: - uses: actions/checkout@v7 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v7 + - name: Set up uv with Python ${{ matrix.python-version }} + uses: astral-sh/setup-uv@v10.1.0 with: python-version: ${{ matrix.python-version }} - - name: Install Poetry - run: | - pip install poetry - - - name: Install dependencies with Poetry - run: | - poetry install --no-interaction --no-ansi --with dev,test + # Why these flags: see "Development setup" in docs/developer.rst. + - name: Install dependencies with uv + run: uv sync --frozen --all-groups - name: Show installed packages - run: poetry show - - - name: Install packages - run: | - python -m pip install . + run: uv tree - - name: Test CFPQ_Data - run: | - poetry run pytest --doctest-modules -vv -s cfpq_data tests + - name: Test FLPQ_Data + run: uv run pytest diff --git a/.gitignore b/.gitignore index d14082df..5de93adc 100644 --- a/.gitignore +++ b/.gitignore @@ -1,9 +1,6 @@ # IDEA .idea -# Data -cfpq_data/data - # Temporal CFPQ reference-answer tool (local-only, not part of the package) temporal_cfpq/ @@ -49,6 +46,7 @@ htmlcov/ .tox/ .coverage .coverage.* +coverage.json .cache nosetests.xml coverage.xml @@ -57,6 +55,9 @@ coverage.xml .pytest_cache/ # Doctest files +bzip.txt +test.csv +test.mcfg test.txt test.xml @@ -80,16 +81,17 @@ instance/ docs/_build/ docs/*/generated/* docs/*/*/generated/* +docs/*/*/*/generated/* # PyBuilder target/ +# Node +node_modules/ + # Jupyter Notebook .ipynb_checkpoints -# pyenv -.python-version - # celery beat schedule file celerybeat-schedule diff --git a/.opencode/skills/add-grammar/SKILL.md b/.opencode/skills/add-grammar/SKILL.md index d55dd295..2bcb9d76 100644 --- a/.opencode/skills/add-grammar/SKILL.md +++ b/.opencode/skills/add-grammar/SKILL.md @@ -1,6 +1,6 @@ --- name: add-grammar -description: Use when adding a new grammar template to CFPQ_Data. Covers the generator/readwrite/converter module conventions and the PR/issue templates. +description: Use when adding a new grammar template to FLPQ_Data. Covers the generator/readwrite/converter module conventions and the PR/issue templates. --- # Add a grammar template @@ -17,22 +17,25 @@ Fill in every field in the triangle brackets (`<>`) of those templates. ## Package conventions -Grammar code lives in `cfpq_data/grammars/`: +Query code lives under `flpq_data/queries//` (`cfpq`, `rpq`, `mcfpq`): -- `generators/` — template factories (e.g. `dyck_grammar`, `c_alias_grammar`). -- `readwrite/` — (de)serialization (`cfg_*`, `cnf_*`, `rsa_*`, `regex_*`). -- `converters/` — conversions between formalisms (`cfg_from_cnf`, ...). -- `utils/` — helpers (e.g. `change_terminals_in_cfg`). +- `cfpq/generators/` — template factories (e.g. `dyck_grammar`, `c_alias_grammar`). +- `cfpq/readwrite/` — (de)serialization (`cfg_*`, `cnf_*`). +- `cfpq/converters/` — conversions between formalisms (`cfg_from_cnf`, ...). +- `cfpq/utils/` — helpers (e.g. `change_terminals_in_cfg`). +- `rpq/readwrite/` — `regex_*`, `rsa_*`. +- `mcfpq/readwrite/` — lark-based `mcfg_*`. Each new module must: 1. Define `__all__` listing its public functions. -2. Export it from the relevant `__init__.py` - (`generators/__init__.py`, `readwrite/__init__.py`, etc.). +2. Export it from the relevant `__init__.py` files up the chain + (e.g. `queries/cfpq/readwrite/__init__.py`, then `queries/cfpq/__init__.py`). 3. Provide numpydoc docstrings with `Examples` (these are run as doctests). -4. Add a mirroring test under `tests/grammars//test_*.py`. -5. Add the template name to `GRAMMAR_TEMPLATES` in - `cfpq_data/dataset/data.py` if it should be downloadable. +4. Add a mirroring test under `tests/queries///test_*.py`. + +Templates are not downloadable on their own — the query files they generate +ship inside each graph archive's `queries///` directory. Use `pyformlang` types (`CFG`, `CNF`, `RSA`, `Regex`) as the interchange format. See `.opencode/skills/run-tests` to verify. diff --git a/.opencode/skills/add-graph/SKILL.md b/.opencode/skills/add-graph/SKILL.md index 15a8b245..ff9d2576 100644 --- a/.opencode/skills/add-graph/SKILL.md +++ b/.opencode/skills/add-graph/SKILL.md @@ -1,6 +1,6 @@ --- name: add-graph -description: Use when adding a new graph to CFPQ_Data. Covers the mtx-per-label archive format, the download()/DATASET wiring, and the PR/issue templates. +description: Use when adding a new graph to FLPQ_Data. Covers the mtx-per-label archive format, the registry.json wiring, and the PR/issue templates. --- # Add a graph to the dataset @@ -18,28 +18,56 @@ Fill in every field in the triangle brackets (`<>`) of those templates. ## Data format The archive layout and the label conventions are documented once in the -"File structure" section of `docs/graphs/index.rst` — follow it exactly: -`.tar.gz` unpacks to `/{README.md, grammar/, graph/}`, where -`graph/` holds one Boolean MatrixMarket pattern file per edge label. +"File structure" section of `docs/graphs/index.rst` — follow it exactly. +Pack every query of the graph into the archive per the "Queries" section of +the PR template, then validate with +`python utils/check_archive_structure.py .tar.gz` (the upload tool +runs the same check and refuses invalid archives). + +New queries for an existing graph ship as a **partial archive** instead — +see the "Data format" section of the PR template; it is validated with +`--partial` and merged into the existing archive with +`python utils/merge_archive.py EXISTING.tar.gz PARTIAL.tar.gz -o .tar.gz`. ## Wiring a new graph into the code -1. Add the graph name to `DATASET` in `cfpq_data/dataset/data.py`. -2. The graph archive must be uploaded under the dataset URL +1. The graph archive must be uploaded under the dataset URL `https://cfpq-data.storage.yandexcloud.net/{VERSION[0]}.0.0/graph/.tar.gz`. -3. Loading happens via `download(name)` (which returns the graph directory) - and `graph_from_mtx_dir(path / "graph")` - (`cfpq_data/graphs/readwrite/mtx.py`). +2. Regenerate `flpq_data/dataset/registry.json` with + `python utils/generate_registry.py` (it picks up the new graph from S3 and + adds its row to the metadata API). +3. Loading happens via `graph_dir(name)` (which returns the self-contained + graph directory — graph plus queries) and `graph_from_mtx_dir(path / "graph")` + (`flpq_data/graphs/readwrite/mtx.py`). ## Documentation 1. Create the per-graph page `docs/graphs/data/.rst`, following the - structure of an existing page (description, statistics tables, canonical - grammars). Statistics conventions: the PR template is the source of truth. -2. Register it in `docs/graphs/index.rst` **twice**: a row in the section - table of its source area, and an entry in the matching captioned toctree - (the left navigation bar is generated from those toctrees). Keep both in - the same order as the table. + structure of an existing page (description, statistics tables). The page + carries **no** grammar section: canonical grammars are documented once on + the category page. Statistics conventions: the PR template is the source + of truth. +2. Handle the grammar case from the PR template's "Canonical grammars" + section: + - *existing category, existing grammar(s)* — nothing to add; the new table + row fills in the matching column(s); + - *new grammar for an existing category* — append the grammar to the + category page's "Canonical grammars" section and add a column to its + table; + - *new category* — create `docs/graphs/.rst` with the category + description, the graph table (one column per grammar), and the + "Canonical grammars" section. +3. Register it **twice**: a row in the per-category table of its source area + (`docs/graphs/.rst`) and an entry in the matching captioned + toctree of `docs/graphs/index.rst` (the left navigation bar is generated + from those toctrees). Keep both in the same order, and bump the category's + graph count in the Contents table of `docs/graphs/index.rst`. For a new + category, also add its page to the `docs/graphs/index.rst` toctree and a + row to the Contents table. + 4. Fill the new row's `Size (MB)` cell with the size the upload tool + reported, then run `python utils/archive_sizes.py --update` before + committing so every table row — including the new one — matches S3 (see + the "Archive sizes" section of `docs/utils.rst`). Note: a `data/*.rst` page missing from all toctrees produces only a Sphinx *warning*, not an error, so the docs quality gate will not catch it — check diff --git a/.opencode/skills/build-docs/SKILL.md b/.opencode/skills/build-docs/SKILL.md index 78ccfa71..626fdb0f 100644 --- a/.opencode/skills/build-docs/SKILL.md +++ b/.opencode/skills/build-docs/SKILL.md @@ -1,25 +1,27 @@ --- name: build-docs -description: Use when building the CFPQ_Data Sphinx documentation. Thin pointer to the "Docs build and deploy" section of docs/developer.rst and docs/README.md, which hold the model and the exact commands; keeps the agent-specific notes. +description: Use when building the FLPQ_Data Sphinx documentation. Thin pointer to the "Docs build and deploy" section of docs/developer.rst and docs/README.md, which hold the model and point at the CI workflow steps with the exact commands; keeps the agent-specific notes. --- # Build docs -The model (no-warnings policy, CI checks, gh-pages deployment) and the exact -local commands are documented in the "Docs build and deploy" section of -`docs/developer.rst`; the canonical local instructions live in -`docs/README.md` — do not duplicate them here. Read those before running -anything. This skill keeps only the agent-specific notes below. +The model (no-warnings policy, CI checks, gh-pages deployment) and where the +exact commands live (the CI workflow steps referenced by the section; the +local setup command in `docs/README.md`) are documented in the "Docs build +and deploy" section of `docs/developer.rst` — do not duplicate them here. +Read those before running anything. This skill keeps only the +agent-specific notes below. ## Notes - No-warnings policy (see the docs page): any Sphinx warning fails the build — fix it, do not suppress it (`suppress_warnings`) or weaken the flag. -- A config-only change (e.g. enabling `nitpicky`) does not re-resolve - references in documents whose content is unchanged — their doctrees are - reused from the cache, so unresolved-reference warnings stay hidden until - those pages are next edited. After changing reference-related config, do a - clean rebuild (`make clean && make html`) before trusting the warning log. +- A plain local rebuild is not CI-equivalent: stale doctrees and git-ignored + autosummary stubs can mask unresolved references (policy: "Fresh-state + verification" in docs/developer.rst). After adding or removing reference + targets, autosummary entries, or automodule directives — and before + trusting any green warning log for a push — run the fresh build (command in + docs/README.md). - API reference pages use `.. automodule::` / `.. autosummary::`; generated stub files land under `docs/*/generated/` and are git-ignored. - Add new public functions to the matching `docs/reference//...rst` diff --git a/.opencode/skills/code-review/SKILL.md b/.opencode/skills/code-review/SKILL.md index 411b8282..34b5aad6 100644 --- a/.opencode/skills/code-review/SKILL.md +++ b/.opencode/skills/code-review/SKILL.md @@ -20,8 +20,8 @@ findings. ## Checklist -- [ ] **Correctness** — every clause of the task (`tasks/tasks.md`) is - traceable to committed code. +- [ ] **Correctness** — every clause of the task issue (#N) is traceable to + committed code. - [ ] **Tests** — new code is covered; no test was weakened or skipped; the suite passes (see `run-tests`). - [ ] **Docs** — docs updated per the `documentation` skill; docstrings follow diff --git a/.opencode/skills/code-style/SKILL.md b/.opencode/skills/code-style/SKILL.md index 166d5066..096c7695 100644 --- a/.opencode/skills/code-style/SKILL.md +++ b/.opencode/skills/code-style/SKILL.md @@ -1,15 +1,17 @@ --- name: code-style -description: Use before committing to format and lint CFPQ_Data. Thin pointer to the "Pre-commit" section of docs/developer.rst, which holds the model and the exact commands; the hook list is defined in .pre-commit-config.yaml. +description: Use before committing to format and lint FLPQ_Data. Thin pointer to the "Quality checks" section of docs/developer.rst, which holds the model and points at the CI workflow step with the exact command; the hook list is defined in .pre-commit-config.yaml. --- # Code style -What pre-commit enforces, how CI runs it, and the exact local commands are -documented in the "Pre-commit" section of `docs/developer.rst` — the single -source of truth. Read that section before running anything. The hook list -lives in `.pre-commit-config.yaml`. +What the quality checks enforce (ruff lint/format, ty type check, hygiene +hooks), how CI runs them, and where the exact command lives (the CI workflow +step referenced by the section) are documented in the "Quality checks" +section of `docs/developer.rst`. Read that section before running anything. +The hook list lives in `.pre-commit-config.yaml`. ## Notes -- Dev deps are in `requirements/developer.txt` (`black`, `pre-commit`, `pytest`). +- Dev tools (`ruff`, `ty`, `pyright`, `pre-commit`) live in the `dev` + dependency group in `pyproject.toml`; run them via `uv run`. diff --git a/.opencode/skills/documentation/SKILL.md b/.opencode/skills/documentation/SKILL.md index a72f628a..06b85e19 100644 --- a/.opencode/skills/documentation/SKILL.md +++ b/.opencode/skills/documentation/SKILL.md @@ -6,14 +6,14 @@ description: Use when determining which docs to update for a code change. Maps s # Documentation Sphinx docs live in `docs/`; the API reference mirrors the package under -`docs/reference/` (`graphs/`, `grammars/`, `dataset/`). This skill is the +`docs/reference/` (`graphs/`, `queries/`, `dataset/`). This skill is the single source of truth for what docs to update when code changes. ## Mapping: source change -> doc action | Source change | Required doc action | |---|---| -| New public function/class | Add to the matching `autosummary` list in `docs/reference//.rst` (e.g. `graphs_generators.rst`, `grammars_readwrite.rst`); write a numpydoc docstring with `Examples` | +| New public function/class | Add to the matching `autosummary` list in the reference page (e.g. `docs/reference/graphs/graphs_generators.rst`, `docs/reference/queries/cfpq/cfpq_readwrite.rst`); write a numpydoc docstring with `Examples` | | Changed public function | Update its numpydoc docstring (params, returns, `Examples`) | | New module | Add a `*.rst` page and add it to the parent `index.rst` toctree | | Removed/renamed API | Update the autosummary list and any docstrings/links referencing it | @@ -24,6 +24,12 @@ single source of truth for what docs to update when code changes. - numpydoc style: `Parameters`, `Returns`, `Examples` sections. - `Examples` are run as **doctests** (`--doctest-modules`) — keep them self-contained and output-stable. +- Cross-references in docstrings (`:func:`, `:data:`, ...) must point at + targets that are actually documented (listed in an autosummary or on a + reference page); the build runs with `-W` (no-warnings), so a reference to + an undocumented constant or an external module (e.g. + `platformdirs.user_cache_dir`) fails it. Use double-backtick literals for + such names. Module-level `#:` comments are not docstrings and are safe. ## Completeness verification diff --git a/.opencode/skills/git-workflow/SKILL.md b/.opencode/skills/git-workflow/SKILL.md index a147870d..a8789218 100644 --- a/.opencode/skills/git-workflow/SKILL.md +++ b/.opencode/skills/git-workflow/SKILL.md @@ -19,6 +19,22 @@ forbidden). If the message mentions multiple subtask identifiers, STOP — split the changes into individual commits. One commit per completed atomic subtask. +**Issue-closing validation**: every completed task carries the closing +keyword for its own issue (`Closes #`) plus one per linked issue +it fully resolves (`Fixes #N` for defects, `Closes #N` otherwise) — all as +standalone lines in exactly one commit: the last subtask's commit. Verify on +the feature branch: + +```bash +git log dev..HEAD --format=%B | grep -cE '^(Fixes|Closes) #[0-9]+$' +``` + +must print `0` before the last subtask's commit and, afterwards, exactly one +line per fully resolved issue including the task's own (no more). A task that +only partially addresses a linked issue uses a bare `#N` reference (no +keyword). The model and closing timing live in the "Contribution guidelines" +section of `docs/developer.rst`. + ### Pre-commit checklist 1. Run commit gate: code-style, all tests, linters. diff --git a/.opencode/skills/planning/SKILL.md b/.opencode/skills/planning/SKILL.md index f51684e3..aecfd08e 100644 --- a/.opencode/skills/planning/SKILL.md +++ b/.opencode/skills/planning/SKILL.md @@ -78,7 +78,7 @@ Example: ``` ### S1: Add graph generator to CSV export -**Code:** New file `cfpq_data/graphs/generators/labeled_cycle_graph.py` +**Code:** New file `flpq_data/graphs/generators/labeled_cycle_graph.py` with a `labeled_cycle_graph` function **Tests:** New `tests/graphs/generators/test_labeled_cycle_graph.py` verifying node/edge counts and labels @@ -145,7 +145,8 @@ Requirements: ## Task Authoring Guidelines -When writing a new task for `tasks/tasks.md`, follow these rules: +When creating a new task issue (`gh issue create`, label `task`), follow +these rules (they apply to the issue body): - **Specify output format upfront**. If the task involves TeX or dot visualization, include the exact column layout, math mode conventions, and @@ -159,10 +160,10 @@ When writing a new task for `tasks/tasks.md`, follow these rules: - **Specify type genericity**. If a module must handle arbitrary types, state it explicitly (e.g., "generic over terminal and nonterminal types"). - **Specify reuse expectations**. If the task builds on existing infrastructure - (e.g., "reuse the labeled graph generators from `cfpq_data/graphs`"), name + (e.g., "reuse the labeled graph generators from `flpq_data/graphs`"), name the dependencies. This prevents reinvention. ## Task Completeness Verification See the `subtask-loop` skill — it is the single source of truth for verifying -task completion before marking a task `[done]` in `tasks/tasks.md`. +task completion before a task is considered done. diff --git a/.opencode/skills/quality-gates/SKILL.md b/.opencode/skills/quality-gates/SKILL.md index e865366f..1ce97010 100644 --- a/.opencode/skills/quality-gates/SKILL.md +++ b/.opencode/skills/quality-gates/SKILL.md @@ -1,6 +1,6 @@ --- name: quality-gates -description: Use before merging a task to dev. Defines the hard gate that must pass (tests + style + docs build) and how to interpret its result. References docs/developer.rst for the exact commands. +description: Use before merging a task to dev. Defines the hard gate that must pass (tests + style + type check + docs build) and how to interpret its result. References docs/developer.rst for the exact commands. --- # Quality Gates @@ -10,38 +10,46 @@ states: **PASS** or **BLOCKED**. There is no "pass with exceptions". ## What the gate is -The gate is the combination of the test suite, the style/lint checks, and -the docs build: +The gate is the combination of the test suite, the style/lint checks, the +type check, and the docs build: - **Tests** — see the "Test pipeline" section of `docs/developer.rst` for - the exact command. -- **Style/lint** — see the "Pre-commit" section of `docs/developer.rst` for - the exact command. + the exact command. The canonical command includes the coverage check: it + fails unless both line and branch coverage are at least 95%. +- **Style/lint** — see the "Quality checks" section of `docs/developer.rst` + for the exact command. +- **Type check** — `uv run ty check` must report no errors; CI additionally + runs Pyright (see the "Quality checks" section of `docs/developer.rst`). - **Docs build** — see the "Docs build and deploy" section of - `docs/developer.rst` (and `docs/README.md`) for the exact command. It must - exit 0. The build runs under the no-warnings policy (`-W --keep-going`, see + `docs/developer.rst` (and `docs/README.md`) for the exact command. Run it + fresh (the `fresh` target — a plain local rebuild is not CI-equivalent, see + the "Fresh-state verification" policy in the same section). It must exit 0. + The build runs under the no-warnings policy (`-W --keep-going`, see `docs/Makefile`): any warning — including an unresolved cross-reference under `nitpicky = True` — fails the build, so the exit code is sufficient. There are no tolerated warnings; fix them instead of suppressing. -- **Link check** — see the "Docs build and deploy" section of - `docs/developer.rst` (and `docs/README.md`) for the exact command. It must - report no broken or timed-out links (the builder exits non-zero on its - own; redirects are reported but do not fail the check). +- **Link check (CI only)** — not part of the local gate: it is network-bound + and slow locally (rate-limited retries). The "Check links" step of + `.github/workflows/docs.yml` must be green for the branch before merge. -`docs/developer.rst` is the single source of truth for the commands; this -skill only defines the gate semantics. +The CI workflows are the source of truth for the commands they run (see +the "CI as source of truth" section of `docs/developer.rst`); this skill +only defines the gate semantics. ## Procedure 1. Run the full test suite ("Test pipeline" section of `docs/developer.rst`). - It must show 0 failures and 0 skipped. -2. Run the full style/lint pass ("Pre-commit" section of + It must show 0 failures and 0 skipped, and the coverage check must pass + (line and branch each at least 95%). +2. Run the full style/lint pass ("Quality checks" section of `docs/developer.rst`). It must show no errors. -3. Build the docs ("Docs build and deploy" section of `docs/developer.rst`). - It must exit 0 (no-warnings policy: any warning fails the build). -4. Run the link check ("Docs build and deploy" section of - `docs/developer.rst`). It must report no broken or timed-out links. -5. Interpret the result: +3. Run the type check (`uv run ty check`). It must report no errors. +4. Build the docs fresh ("Docs build and deploy" section of + `docs/developer.rst`; command in `docs/README.md`). It must exit 0 + (no-warnings policy: any warning fails the build). +5. Confirm the branch's CI link check ("Check links" step of + `.github/workflows/docs.yml`) is green — it runs in CI only, not locally. +6. Interpret the result: - All clean → **PASS**. Proceed to merge (see `git-workflow`). - Any failure → **BLOCKED**. diff --git a/.opencode/skills/release/SKILL.md b/.opencode/skills/release/SKILL.md index 77f6e7cc..7f6ab96c 100644 --- a/.opencode/skills/release/SKILL.md +++ b/.opencode/skills/release/SKILL.md @@ -1,14 +1,16 @@ --- name: release -description: Use when cutting a CFPQ_Data release — bumping the version, finalizing the changelog, tagging, and publishing to PyPI. Covers the procedure and the automated publish workflow. +description: Use when cutting an FLPQ_Data release — bumping the version, finalizing the changelog, tagging, publishing to PyPI — or recovering from a failed publish. Covers the procedure, the automated publish workflow, and recovery. --- # Release -Cut a release of the `cfpq-data` package. The model, versioning policy, and -prerequisites are documented in `docs/release.rst` — do not duplicate them here. -Publishing is automated by `.github/workflows/publish.yml` (runs on a `v*` tag -push). +Cut a release of the `flpq-data` package. The model, versioning policy, +prerequisites, and packaging constraints are documented in `docs/release.rst` +— do not duplicate them here. Publishing is automated by +`.github/workflows/publish.yml`: on a `v*` tag push it builds, publishes to +PyPI, and creates the GitHub Release; on every PR targeting `master` it +pre-publishes to TestPyPI as a packaging check. ## Procedure @@ -23,11 +25,14 @@ push). 2. Verify the version guard passes: ```bash - poetry run pre-commit run check-version-sync --all-files + uv run pre-commit run check-version-sync --all-files ``` 3. Commit the bump (message: `chore: release X.Y.Z`). -4. Merge `dev` into `master` via a pull request (`master` is protected; do not - squash). This also deploys the docs. +4. Merge `dev` into `master` via a pull request (`master` is protected; do + not squash). This also deploys the docs. Wait for the PR's **Publish** + check (the TestPyPI pre-publish) to pass — it validates packaging for + exactly the code that will be released. The user reviews and merges the + PR manually; proceed only after they confirm the merge. 5. Tag the merged commit and push the tag: ```bash @@ -35,8 +40,53 @@ push). git tag -a vX.Y.Z -m "Release vX.Y.Z" origin/master git push origin vX.Y.Z ``` -6. The `publish` workflow builds, publishes to PyPI (Trusted Publishing), and - creates the GitHub Release. Verify on . +6. The `publish` workflow builds, publishes to PyPI (Trusted Publishing), + and creates the GitHub Release. Verify both: the package is live on + and the GitHub Release carries the + `[X.Y.Z]` changelog section with the dist artifacts as assets. + +## Recovery + +Use the `gh` CLI (installed and authenticated on this machine; if a new +machine is not, run `gh auth login` first). + +### Workflow failed before the PyPI publish + +Nothing reached PyPI, so the tag can simply be replaced: + +```bash +git push origin :refs/tags/vX.Y.Z # delete the tag +``` + +Fix the failure, merge the fix to `master` (PR + user merge), then re-tag +the new `origin/master` and push again. + +### Workflow failed after the PyPI publish (e.g. the GitHub Release step) + +PyPI already has the files for this version and rejects re-uploads — an +owner must delete the release on PyPI before the tag can be re-pushed (see +`docs/release.rst`). In the meantime, create the GitHub Release manually so +the release is complete: + +1. Extract the changelog section (same logic as the workflow): + + ```bash + awk -v v="X.Y.Z" 'index($0, "## [" v "]") == 1 {insec=1; next} /^## / && insec {exit} insec {print}' CHANGELOG.md > release_notes.md + ``` + +2. Create the release for the existing tag with the extracted notes: + + ```bash + gh release create vX.Y.Z --title "vX.Y.Z" --notes-file release_notes.md + ``` + +3. Upload the dist artifacts — download them from PyPI first so the release + assets are byte-identical to what was published + (`flpq_data-X.Y.Z-py3-none-any.whl`, `flpq_data-X.Y.Z.tar.gz`): + + ```bash + gh release upload vX.Y.Z flpq_data-X.Y.Z-py3-none-any.whl flpq_data-X.Y.Z.tar.gz + ``` ## Notes diff --git a/.opencode/skills/reusing/SKILL.md b/.opencode/skills/reusing/SKILL.md index 1fef8c0d..575646f7 100644 --- a/.opencode/skills/reusing/SKILL.md +++ b/.opencode/skills/reusing/SKILL.md @@ -13,17 +13,17 @@ the project's single-source-of-truth principle. Run through these before creating new material: -- [ ] **Package code** — search `cfpq_data/` for an existing function or type +- [ ] **Package code** — search `flpq_data/` for an existing function or type that already does (or nearly does) the job. Prefer generalizing the existing one over adding a near-copy. - [ ] **Graph types** — graphs are `networkx.MultiDiGraph`; use NetworkX built-ins (`nx.from_pandas_edgelist`, generators, `nx.MultiDiGraph`) and - the existing `cfpq_data.graphs` helpers instead of reimplementing. + the existing `flpq_data.graphs` helpers instead of reimplementing. - [ ] **Grammar types** — grammars are `pyformlang` `CFG` / `CNF` / `RSA` / - `Regex`; use pyformlang APIs and existing `cfpq_data.grammars` converters + `Regex`; use pyformlang APIs and existing `flpq_data.queries.cfpq` converters instead of reimplementing. -- [ ] **Generators** — check `cfpq_data/graphs/generators` and - `cfpq_data/grammars/generators` for an existing template before adding a +- [ ] **Generators** — check `flpq_data/graphs/generators` and + `flpq_data/queries/cfpq/generators` for an existing template before adding a new one. - [ ] **Readwrite** — check `readwrite` modules for an existing `*_from_text`/`*_to_text` pair before adding new serialization. diff --git a/.opencode/skills/run-tests/SKILL.md b/.opencode/skills/run-tests/SKILL.md index 0b9d0494..48100c89 100644 --- a/.opencode/skills/run-tests/SKILL.md +++ b/.opencode/skills/run-tests/SKILL.md @@ -1,14 +1,15 @@ --- name: run-tests -description: Use when running the CFPQ_Data test suite (pytest). Thin pointer to the "Test pipeline" section of docs/developer.rst, which holds the model and the exact commands; keeps the machine-specific bare-pytest pitfall. +description: Use when running the FLPQ_Data test suite (pytest). Thin pointer to the "Test pipeline" section of docs/developer.rst, which holds the model and points at the CI workflow step with the exact command; keeps the machine-specific pitfalls (bare pytest, partial venv after uv add, stale coverage.json gate). --- # Run tests -What the test pipeline is (doctest-enabled suite, CI matrix, coverage upload) -and the exact local commands are documented in the "Test pipeline" section of -`docs/developer.rst` — the single source of truth. Read that section before -running anything. This skill keeps only the machine-specific pitfall below. +What the test pipeline is (doctest-enabled suite, CI matrix, coverage gate) +and where the exact command lives (the CI workflow step referenced by the +section) are documented in the "Test pipeline" section of `docs/developer.rst`. +Read that section before running anything. This skill keeps only the +machine-specific pitfalls below. ## Do not run bare pytest @@ -16,12 +17,35 @@ Never run `pytest` or `python -m pytest` with the system interpreter. The system Python on this machine has networkx 2.6.2, while the project pins `networkx ^3.6.1` (`pyproject.toml`). The edge-count assertions in `tests/graphs/generators/test_fast_labeled_binomial_graph.py` and the doctest -in `cfpq_data/graphs/generators/fast_labeled_binomial_graph.py` depend on +in `flpq_data/graphs/generators/fast_labeled_binomial_graph.py` depend on networkx's version-specific RNG behavior and fail spuriously under 2.6.2 (91 != 85, 182 != 177, 722 != 711). If a bare-pytest run shows exactly those -failures, re-run under `poetry run` before investigating — the suite is green -in the Poetry environment (networkx 3.6.1). +failures, re-run under `uv run` before investigating — the suite is green in +the uv project environment (networkx 3.6.1). + +## Re-sync after `uv add` / `uv remove` + +`uv add ` syncs only part of the dependency set and can leave the +venv missing packages from the other groups (`test`, `docs`) — observed +2026-09-21: after `uv add lark`, the venv had no pytest, and `uv run pytest` +silently resolved to the system pytest 8.0.1 (networkx 2.6.2), producing +exactly the bare-pytest failure signature above. After any `uv add` or +`uv remove`, restore the full environment before running anything: + + uv sync --all-groups + +## The coverage gate reads a JSON file, not the live run + +`utils/check_coverage.py` reads `coverage.json` from disk — it does not run +pytest. If you invoke pytest with only `--cov-report=term` (or any report set +that omits `json`), the `coverage.json` left by the previous run is stale, and +the gate silently passes or fails on old numbers (observed 2026-10-05: a gate +run reported byte-identical percentages to the prior task because the JSON was +never regenerated). Always generate the JSON in the same command as the run — +use the canonical CI command from `docs/developer.rst`, which includes +`--cov-report=json`. ## Notes -- Test deps: `pytest`, `pytest-cov`, `codecov` (`requirements/tests.txt`). +- Test deps: `pytest`, `pytest-cov` (the `test` dependency group in + `pyproject.toml`). diff --git a/.opencode/skills/subtask-loop/SKILL.md b/.opencode/skills/subtask-loop/SKILL.md index 84a15ebd..19095b92 100644 --- a/.opencode/skills/subtask-loop/SKILL.md +++ b/.opencode/skills/subtask-loop/SKILL.md @@ -77,7 +77,21 @@ the operational procedure (validation, merge) lives in the skill. ### 7. Mark Completed -Mark the subtask as completed in `tasks/detailed_plan.md`. +Mark the subtask as completed in `tasks/detailed_plan.md` (`[done]` suffix + +commit hash on the subtask heading), then mirror the file onto the plan +comment of the task issue: find the comment by its `` +first line and PATCH its body to the current file content: + +```bash +REPO=$(gh repo view --json nameWithOwner -q .nameWithOwner) +CID=$(gh api "repos/$REPO/issues//comments" \ + --jq '[.[] | select(.body | startswith(""))] | last.id') +{ echo ''; cat tasks/detailed_plan.md; } > /tmp/plan_comment.md +gh api -X PATCH "repos/$REPO/issues/comments/$CID" \ + --input <(jq -Rs '{body: .}' /tmp/plan_comment.md) +``` + +The local file is the single source of truth; the comment is a pure mirror. If at any point in steps 1–7 you hit an unresolvable problem that prevents 100% completion, **STOP the cycle immediately** and follow the Blocked Work Protocol @@ -147,8 +161,7 @@ failing tests to make the suite green. Instead: 3. Ask the user for guidance: additional subtasks, algorithmic hints, descoping, or splitting the task. 4. **Transfer user guidance to the task** per the `user-guidance-transfer` - skill — append `**[USER GUIDANCE]**` annotation to the task in - `tasks/tasks.md`. + skill — post it verbatim as a comment on the task issue. 5. Append a `## Design Notes` section to `tasks/detailed_plan.md`. See the `planning` skill for the full template. Minimum required content: @@ -166,13 +179,16 @@ failing tests to make the suite green. Instead: goals. - **Skipped Tests**: list any tests skipped and the reason. - Commit this summary so the plan serves as a persistent design record for - future task refinement. + Commit this summary so the plan serves as a persistent design record for + future task refinement. +6. Post the block report (the `## Design Notes` content) as a comment on the + task issue, so the block is visible in the task record. ## Task Completion Verification This section is the **single source of truth** for what "done" means. Before -marking a task `[done]` in `tasks/tasks.md`, verify: +a task is considered done — its last subtask's commit carries `Closes #` +(the task's own issue) and is merged to `dev` — verify: - [ ] Every clause in the task description is traceable to implemented and committed code. @@ -181,7 +197,7 @@ marking a task `[done]` in `tasks/tasks.md`, verify: - [ ] All tests pass (0 failures, 0 skipped). - [ ] All quality gates pass (see the `quality-gates` skill). - [ ] If any of the above fails, the task is NOT done — it is blocked. Follow - the Blocked Work Protocol. Do NOT mark it `[done]` with known unresolved + the Blocked Work Protocol. Do NOT close it with known unresolved limitations. Partial completion is not completion. "All passing tests are for the parts I @@ -189,10 +205,11 @@ did" does not mean the task is done if other parts were reverted. ## Marking Complete -Mark the task as completed in `tasks/tasks.md` — **only prepend `[done] ` to the -existing task line. Never rewrite the task description.** The task text in -`tasks/tasks.md` is user-authored and immutable. +A task is complete when its last subtask's commit carries `Closes #` (the +task's own issue number) as a standalone line; GitHub closes the issue when +the release containing the task lands on `master`. Never edit the issue body +— it is user-authored and immutable. -The `[done]` tag means COMPLETE: every requirement met, every test passing, -every edge case handled. Never mark a task as `[done]` with known failures or -unresolved limitations. +Complete means COMPLETE: every requirement met, every test passing, every +edge case handled. Never close a task with known failures or unresolved +limitations. diff --git a/.opencode/skills/user-guidance-transfer/SKILL.md b/.opencode/skills/user-guidance-transfer/SKILL.md index ab986bea..26c281a8 100644 --- a/.opencode/skills/user-guidance-transfer/SKILL.md +++ b/.opencode/skills/user-guidance-transfer/SKILL.md @@ -1,6 +1,6 @@ --- name: user-guidance-transfer -description: Use when the user provides design guidance, descoping, algorithmic hints, or clarification during a task. Record it verbatim into tasks/tasks.md so it is never lost or paraphrased. +description: Use when the user provides design guidance, descoping, algorithmic hints, or clarification during a task. Record it verbatim as a comment on the task issue so it is never lost or paraphrased. --- # User Guidance Transfer @@ -11,21 +11,20 @@ verbatim. ## Procedure -1. In `tasks/tasks.md`, append a `**[USER GUIDANCE]**` annotation to the task - line it applies to. Never rewrite or delete the original task text — it is - user-authored and immutable. +1. Post the guidance as a comment on the task issue: + `gh issue comment --body '[USER GUIDANCE]: ""'`. + Never edit the issue body — it is user-authored and immutable. 2. Quote the user's guidance **verbatim**. Do not paraphrase, summarize, or interpret. 3. If the guidance is algorithmic and belongs in the persistent design record, also record it verbatim in the `### ` section of the `## Design Notes` block in `tasks/detailed_plan.md` (see the `planning` skill). -Example: +Example comment: ``` -- [ ] Task 3: Implement matrix-based CFPQ **[USER GUIDANCE]**: "Use CSR - representation; the matrix indices must be the same node order as the DFA - from task 2." +[USER GUIDANCE]: "Use CSR representation; the matrix indices must be the same +node order as the DFA from task 2." ``` Only transfer guidance the user actually gave; do not invent or pad it. diff --git a/.opencode/skills/workflow-management/SKILL.md b/.opencode/skills/workflow-management/SKILL.md index a32de092..ec1ae5b9 100644 --- a/.opencode/skills/workflow-management/SKILL.md +++ b/.opencode/skills/workflow-management/SKILL.md @@ -11,8 +11,9 @@ step. ## Core Rules -- Log all tasks to `tasks/tasks.md`. Use the description exactly as the user - provided it — minimal changes, only splitting and numbering. +- Log all tasks as GitHub issues with the `task` label (`gh issue create`). + Use the description exactly as the user provided it — minimal changes, only + splitting and numbering. The task ID is the issue number. - Do tasks strictly **one at a time**. Each task gets its own feature branch, its own detailed plan, and its own merge to `dev`. Never combine multiple tasks in a single feature branch. @@ -27,16 +28,23 @@ step. ## Working Loop -0. If the user requests multiple tasks at once, first create a global plan in - `tasks/global_plan.md` (see the `planning` skill) before proceeding. +0. If the user requests multiple tasks at once, first create one issue per + task, then create a global plan in `tasks/global_plan.md` referencing the + issue numbers (see the `planning` skill) before proceeding. 1. Ensure user-defined tasks, the global plan, and project architecture are aligned. -2. Choose exactly ONE task that is not yet done. +2. Choose exactly ONE open `task`-labeled issue that is not yet done: a task + is done when its subtask commits are on `dev` + (`git log dev --format=%s | grep -cE '\(-S[0-9]+\):'` > 0, where N is + the issue number). List candidates with + `gh issue list --label task --state open`. 3. Create a feature branch from `dev` for this single task (branching model: the "Contribution guidelines" section of `docs/developer.rst`; procedure: `git-workflow`). 4. Generate a detailed plan in `tasks/detailed_plan.md`, decomposing the task - into atomic subtasks (see `planning`). + into atomic subtasks (see `planning`), then post it as a comment on the + task issue; the first line of the comment is the marker + ``. 5. Load the `subtask-loop` skill, then execute each subtask using its cycle. 5a. Verify all subtasks are complete and unblocked. Check `tasks/detailed_plan.md`: @@ -61,7 +69,8 @@ step. your changes; fix every failure and re-run until PASS. Then merge the feature branch to `dev` (see `git-workflow`). Verify `git branch --show-current` is `dev`. -8. Mark the task `[done]` in `tasks/tasks.md` — see the Task Completeness - Verification in the `subtask-loop` skill (the single source of truth for - what "done" means). +8. Verify the last subtask's commit carries `Closes #` (the task's own + issue) as a standalone line — the issue closes when the release lands on + `master`. See the Task Completeness Verification in the `subtask-loop` + skill (the single source of truth for what "done" means). 9. Return to step 2. diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 6e45b4ae..a592c88b 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,15 +1,22 @@ repos: - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v4.3.0 + rev: v6.0.0 hooks: - id: check-yaml - id: end-of-file-fixer - id: trailing-whitespace - - id: requirements-txt-fixer - - repo: https://github.com/psf/black - rev: 22.10.0 + - repo: https://github.com/astral-sh/uv-pre-commit + # uv version. + rev: 0.12.15 hooks: - - id: black + - id: uv-lock + - repo: https://github.com/astral-sh/ruff-pre-commit + # Must match the ruff version locked in uv.lock. + rev: v0.16.7 + hooks: + - id: ruff-check + args: [--fix] + - id: ruff-format - repo: local hooks: - id: check-version-sync @@ -18,3 +25,9 @@ repos: language: system pass_filenames: false always_run: true + - id: ty-check + name: ty type check + entry: uv run ty check + language: system + pass_filenames: false + files: \.py$ diff --git a/.python-version b/.python-version new file mode 100644 index 00000000..2c073331 --- /dev/null +++ b/.python-version @@ -0,0 +1 @@ +3.11 diff --git a/AGENTS.md b/AGENTS.md index 662a4ddf..8b4af487 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,8 +1,10 @@ # AGENTS.md -CFPQ_Data is a Python package (`cfpq_data`, Apache-2.0) providing Graphs and -Grammars for experimental analysis of Context-Free Path Querying (CFPQ) -algorithms. This file is an entrypoint/TOC only — details live in the skills +FLPQ_Data is a Python package (`flpq_data`, Apache-2.0) providing Graphs and +Grammars for experimental analysis of Formal Language constrained Path +Querying (FLPQ) algorithms. It is both the dataset and the tools to support +and use it — extend, modify, validate. +This file is an entrypoint/TOC only — details live in the skills linked below. ## Start here @@ -15,23 +17,28 @@ gate → merge, and points to the other workflow skills below. ## Main Principles * Documentation is about "What" and "Why". Skills are about "How". -* Docs are the source of truth; skills are thin pointers. A skill references - the docs page for the model and keeps only agent-specific operational - details (commands, pitfalls, procedures) — it never re-describes what the - docs already say. When a topic gains a docs page, slim the overlapping - skills to pointers (see `release` → `docs/release.rst`). * This file is a short entry point for fast cold errors-free start. -* Only one source of truth. No duplicates. Each thing (in doth code and documentation) described exactly once. Use generalization (especially for code), cross-references, links, other similar techniques to avoid duplicates and reuse staff. -* Tools, not instructions. If you can do something with existing tool --- do it. No thinking, no long instructions, no manual analysis. You want to analyze code coverage? Just run coverage tool and analyze report. +* Only one source of truth. No duplicates. Each thing (in both code and documentation) described exactly once. Use generalization (especially for code), cross-references, links, other similar techniques to avoid duplicates and reuse staff. +* Source-of-truth hierarchy: code, scripts, CI configs > docs. If a fact can + be extracted from code or scripts (e.g., CI configs), it is not duplicated + in docs. Docs hold only what cannot be unambiguously reconstructed from + code: design decisions, non-trivial constraints. Skills stay thin pointers + to docs, code, or CI — they never re-describe them. +* Data archives are self-contained and must align with the checker + (`utils/check_archive_structure.py`) — the source of truth for any data + structure. +* Tools, not instructions. If you can do something with existing tool --- do it. No thinking, no long instructions, no manual analysis. You want to analyze code coverage? Just run coverage tool and analyze report. No workaround for regular tasks. If there is a tool for regular task it must be installed and configured appropriately. * Always learn, never forget — encode patterns before session ends ## Package layout -- `cfpq_data/config.py` — version, data directories. -- `cfpq_data/dataset/` — dataset download (`download`, `download_grammars`, - `download_benchmark`). -- `cfpq_data/graphs/` — `generators`, `readwrite`, `utils`. -- `cfpq_data/grammars/` — `generators`, `readwrite`, `converters`, `utils`. +- `flpq_data/config.py` — version. +- `flpq_data/dataset/` — graph registry + metadata API, machine-global cache + with lazy access (`graph_dir`), reachable-pair counts (`reachable_pairs`). +- `flpq_data/graphs/` — `generators`, `readwrite`, `converters`, `utils`. +- `flpq_data/queries/` — per-class query modules: `cfpq/` (`generators`, + `readwrite`, `converters`, `utils`), `rpq/` (`readwrite`), `mcfpq/` + (`readwrite`). - `tests/` mirrors the package structure (pytest + doctests). ## Skills diff --git a/CHANGELOG.md b/CHANGELOG.md index d424de1a..0ed9ee26 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,17 +1,232 @@ # Changelog -All notable changes to CFPQ_Data are documented in this file. +All notable changes to FLPQ_Data are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +## [6.0.0] - 2026-10-13 + ### Added +- Optimized WCNF grammar variants for the `java_points_to` and + `c_alias_analysis` categories, taken from "Optimization of the Context-Free + Language Reachability Matrix-Based Algorithm" (arXiv:2401.11029, + optimization 5): every archive of the two categories now carries + `java_points_to_muravev2024.cnf` / `c_alias_muravev2024.cnf` alongside the + original grammar. Each variant follows the label convention of its archive + (indexed `load_i` form or bare `load` family form) and was verified to + return identical reachable-pair counts to the original with FastMatrixCFPQ + on small graphs. +- The graph tables on the website now list the download size of every + archive in a `Size (MB)` column — one unit (MB) for all graphs, from the + smallest (0.002 MB) to the largest (112.65 MB) archive. +- `utils/upload_to_s3.py` reports the verified archive size (bytes and MB) + after each upload, and the new `utils/archive_sizes.py` checks or updates + the `Size (MB)` column of the docs graph tables against the stored + archives on Yandex S3. - The publish workflow now runs on every pull request targeting `master`: it builds the sdist and wheel and publishes them to TestPyPI (`skip-existing`), validating packaging before a release tag is cut. +- Ruff as the linting and formatting gate (rules `E`, `F`, `W`, `I` plus + `ruff format`), replacing black; the codebase was brought in line in a + one-time pass. +- Type checking with [ty](https://docs.astral.sh/ty/) for fast local checks + and Pyright in CI; the package, tests, and utility scripts are now fully + type-checked. +- A math-snippet guard in the test suite (`utils/check_math_snippets.py`): it + scans every docs page and rejects math that MathJax cannot render — an + unescaped underscore inside a text-mode command group (e.g. + `\textit{a_b}`) — which previously surfaced only as an error on the + deployed site while the Sphinx build stayed green. +- The MCFG readwrite module `flpq_data/queries/mcfpq/readwrite/mcfg.py`: a + data model for multiple context-free grammars, lark-based parsing of the + Datalog-like `.mcfg` format specified in `docs/flpq.rst`, semantic + validation (start symbol, dimension and rank), and `mcfg_from_text` / + `mcfg_to_text` with round-trip guarantees plus `mcfg_from_txt` / + `mcfg_to_txt` for file I/O. +- The `category` and `query_class` columns of `reachable_pairs.csv` and the + matching filters and row fields of the `reachable_pairs()` API: reference + counts are now keyed per graph category and query class (`cfpq`, `rpq`, + `mcfpq`). +- The transition-system description style of the `.rsm` format: a query may + be written as an explicit labelled transition system in addition to the + CFG (EBNF) style, and `rsa_from_text` accepts both. +- Partial graph archives: `utils/check_archive_structure.py --partial` + validates an archive that carries only new queries for an existing graph, + and `utils/merge_archive.py` merges such a partial archive into the + existing full one. +- The reachable-pairs table is republished, versioned, on the dataset object + storage (`6.0.0/reachable_pairs.csv`), exposed as `REACHABLE_PAIRS_URL` and + fetched into the local data cache by `download_reachable_pairs()`; once + downloaded, `reachable_pairs()` reads it instead of the bundled copy, so an + updated table is consumed without reinstalling the package. +- The per-class "Applicable graphs" lists of the new query sections: a new + `utils/applicable_graphs.py` renders each class's list from + `reachable_pairs.csv`, grouped by graph category and cross-linking the + shared per-graph pages, in check or `--update` mode. +- The end-user core rework design is documented in `docs/flpq.rst` (hub + #148): the core/maintenance boundary — `flpq_data/` is the end-user core + for dataset utilisation (benchmarks, statistical analysis), and dataset + maintenance (archive validation, upload, merge, registry generation) stays + in repo-only `utils/` scripts that are never installed with the wheel — + plus the per-graph registry, the machine-global versioned cache, the lazy + `graph_dir()` accessor, and the cache manipulation API. Implementation + follows in #150–#152. +- A per-graph `registry.json` bundled with the package (and to be published + to object storage as `/registry.json`): for every graph — its + category, node and edge counts, archive size in MB, sha256, and the query + list with representations. It is generated by the repo-only + `utils/generate_registry.py`. The network-free metadata API exposes it: + `graph_names()` (all names), `graph_info(name) -> GraphInfo`, and + `categories() -> dict[str, list[str]]` — the category of each graph + (#147). +- The machine-global cache API: `cache_root()` (the effective root — the + `FLPQ_DATA_CACHE` environment variable or the OS default from + `platformdirs`), `cached_versions()` (the dataset versions present in the + cache), and `clear_cache(keep=None)` (remove the cached version + directories, optionally keeping one). +- Streaming format conversion: `convert_graph(src, dst, *, src_format, + dst_format)` converts between `mtx` (a directory of MatrixMarket files), + `txt`, `rdf`, and the FastMatrixCFPQ `.g` text (write-only) — plus an + in-memory `nx.MultiDiGraph` as source — over a lazy edge stream without + materializing a graph, so converting the largest archive (`fs`, 3.6M edges) + from MTX to TXT or `.g` runs in ~0.1 GiB of RAM instead of ~4 GiB (#138). + The streaming primitives: `iter_edges_from_mtx_dir` / `mtx_dir_from_edges`, + `iter_edges_from_text` / `iter_edges_from_txt`, `iter_edges_from_rdf` / + `rdf_from_edges`, `iter_edges_from_graph`, and `g_text_from_edges`. + +### Changed + +- Filled 56 of the 60 previously "not available" reachable-pair reference + counts in `reachable_pairs.csv` (issue #130): the in-archive `results.mtx` + stubs of the affected graphs now carry the full Boolean reachability matrix, + computed with FastMatrixCFPQ using the language-equivalent low-memory + grammars where they exist (`c_alias_muravev2024.cnf`, + `java_points_to_muravev2024.cnf`, and the sparse uniprot grammar of issue + #157). 48 counts match independent reference values exactly; the 8 + `java_points_to` pairs with stale references were confirmed by the dataset + maintainer. The 4 pairs that exceed the 124 GB RAM compute host (libgdx / + name_resolution, unigraph_8/9/10) remain "not available"; per-pair run data + and the infeasibility rationale are recorded in + `utils/stub_results_record.json`. +- The `publish` workflow gives every pull request targeting `master` a unique + `X.Y.Z.dev` version on the runner before building, so each push + publishes a distinct TestPyPI pre-release instead of being skipped as an + existing file; the commit SHA is recorded in the run's step summary. The + committed version and the tag/PyPI release path are unchanged. +- **Breaking:** the package is renamed `cfpq_data` -> `flpq_data` (PyPI + distribution `flpq-data`) and `grammars/` is restructured into + `queries/{cfpq,rpq,mcfpq}/` — the module tree gains the same query-class + level as the site, per the FLPQ design in `docs/flpq.rst`. The flat + top-level API is unchanged apart from the renames below; import the new + names (`from flpq_data import *`). +- `download(name)` becomes `graph_dir(name)`, which is lazy: it returns the + local graph folder from a machine-global, per-user cache (the + `FLPQ_DATA_CACHE` environment variable or the OS default from + `platformdirs`) keyed by dataset version, and downloads only on a cache + miss — the archive is sha256-verified against the registry before it is + installed. `download_graph(name)` and `download(name)` remain as deprecated + aliases. The `DATASET` registry list is replaced by the `graph_names()` + function backed by the per-graph `registry.json`; `DATASET_URL` now serves + the `6.0.0/graph/` prefix. +- Development tooling migrated from Poetry to [uv](https://docs.astral.sh/uv/): + PEP 621 project metadata with hatchling as the build backend, PEP 735 + dependency groups (`dev`, `test`, `docs`) instead of Poetry groups and the + `setup.py` extras, a committed `uv.lock`, and all CI workflows on + `astral-sh/setup-uv`. +- The labeled graph generators type their `labels` parameter as + `Sequence[str]`, correctly accepting both a single label string and a + sequence of labels (previously annotated `List[str]` despite the string + default). +- The dataset on object storage moved to the 6.0.0 self-contained archive + layout: each graph archive carries its queries under + `queries///` (every representation plus one `results.mtx`) + instead of a separate top-level `grammar/` directory, and the website's + download links and `Size (MB)` tables point at the new prefix. +- The website's Dataset section is restructured into per-class query sections + (CFPQ, RPQ, MCFPQ): the flat Grammars section is replaced by one section per + class with its templates and an "Applicable graphs" list, and the "How to + add a new grammar?" pointer moves to the Dataset page. +- `utils/merge_archive.py` refuses to merge a partial archive whose new query + directory has no row in `reachable_pairs.csv` (same graph and query class, + with the query directory name as the stem of the row's grammar file), so the + registry cannot drift. +- **Breaking:** the RDF output of `graph_to_rdf` is now valid RDF 1.1 Turtle + — IRI nodes `urn:flpq:node:` and IRI predicates `urn:flpq:label: