Skip to content

chore: purge every trace of the SQLite engine - #361

Closed
ywatanabe1989 wants to merge 1 commit into
developfrom
chore/purge-sqlite
Closed

chore: purge every trace of the SQLite engine#361
ywatanabe1989 wants to merge 1 commit into
developfrom
chore/purge-sqlite

Conversation

@ywatanabe1989

Copy link
Copy Markdown
Collaborator

Storage is per-host PostgreSQL via scitex_dev.store. Under the 2026-08-29 ruling the SQLite engine is abolished fleet-wide and its name appearing anywhere in this repo is itself the bug. scitex-db stays exempt as a database library; scitex-python does not, so the umbrella stops advertising and shipping the engine. Only docs/adr/ is exempt, and it carried no occurrence.

No SQLite-aware error path was added anywhere. Naming the engine in order to refuse it is still naming it.

Enumeration

git grep -I hides binary matches, and no content grep can see a path. Three passes were needed, and the last two found things the first could not:

pass before after
git grep -n -i sqlite (content, binaries included) 55 matches in 20 files 0
git ls-files | rg -i sqlite (tracked paths) 13 files 0
rg -i --hidden --no-ignore (incl. untracked) 0

The -I in the prescribed git grep -In hid two committed engine database files whose content matches; the path pass found twelve zero-byte example files whose names were the only occurrence.

Prose and config

  • README.md, docs/05_ADDITIONAL_MODULES.md — the scitex.db section now documents the PostgreSQL client only. The example is rewritten against the real signature (dbname/user/host/port) and methods that exist (execute, save_array, load_array, get_summaries); the old snippet called to_df/check_health as instance methods, which the client does not define. delete_duplicates is dropped from the advertised surface: upstream marks it deprecated and engine-specific.
  • docs/sphinx/core_concepts.rst, docs/sphinx/modules/io.rst — provenance hashes are recorded "to the Clew store", not to a named engine.
  • docs/guides/CROSSREF_API_CONFIGURATION.md, .env.d.examples/00_crossref-local.env — the local CrossRef mirror is described by what it is.
  • pyproject.toml — the clew extra note no longer enumerates the engine module.
  • .gitignore (x2) — db.sqlite3, db.sqlite3-journal, **/*.sqlite deleted outright.
  • scripts/maintenance/_pypi_packages.py — the stdlib-module entry is removed. It only suppressed a PyPI mapping that does not exist, so get_package_name still returns None; behaviour is unchanged and such an import would now be visible rather than silently classified.

Artefacts removed

  • .scitex/clew/runtime/db.sqlite and examples/_legacy/scitex/clew/multi_parent/scitex/clew.db — two engine database files committed as run leftovers. The second is already covered by .gitignore (**/scitex/clew.db) and should never have landed. Nothing in the tree references either.
  • examples/_legacy/scitex/db/_sqlite3/** — twelve zero-byte placeholder example files. Their paths were the only thing naming the engine.
  • examples/_legacy/notebooks/19_scitex_db.ipynb — a legacy notebook that existed to demo the engine class. Its two index entries in 00_SCITEX_MASTER_INDEX.ipynb are removed with it, so no link dangles.
  • examples/_legacy/notebooks/27_scitex_scholar.ipynb — the hand-rolled direct-database impact-factor lookups are removed. Scholar(enrich_by_default=True), already constructed two cells earlier, is what supplies that metadata; the removed cells were non-executable legacy stubs (if ...: with a # Condition met comment for a body).

Test changes

  • tests/integration/test_integration.py::test_db_delegates — dropped assert hasattr(stx.db, "SQLite3"). It asserted a mechanism that must not be advertised. The PostgreSQL and check_health assertions in the same test are untouched, so the test still covers the delegation.
  • tests/integration/test_cross_package_imports.py — dropped "scitex_io._load_modules._sqlite3" from the auto-generated list. That module ships in neither the scitex-io checkout nor the installed package, so the entry was already dead. See the finding below.

No test was deleted outright.

Finding for scitex-io (out of scope here)

scitex_io._load_modules._sqlite3 does not exist in /home/ywatanabe/proj/scitex-io/src/scitex_io/_load_modules/ nor in the installed scitex_io. The entry here was a stale auto-generated reference. The block is regenerated by scitex-dev ecosystem write-integration-tests, so if that generator still reads a snapshot naming the module it will reintroduce the string on the next regeneration.

Test evidence

Worktree source confirmed to shadow the installed copy before trusting anything:

$ PYTHONPATH=<worktree>/src python -c "import scitex; print(scitex.__file__)"
/home/ywatanabe/proj/scitex-python/.worktrees/purge-sqlite/src/scitex/__init__.py

Both runs use the same interpreter and the same command shape, one per worktree:

branch : 33 failed, 460 passed, 159 skipped, 37 xfailed, 17 xpassed in 137.90s (0:02:17)
develop: 33 failed, 460 passed, 160 skipped, 37 xfailed, 17 xpassed in 180.44s (0:03:00)

The suite is not green, and was not green before this change. The two FAILED lists are byte-identical (33 lines, diff exit 0), so no failure is attributable to this change. The one-skip delta is exactly the removed scitex_io._load_modules._sqlite3 parametrization, which was being skipped because the module does not exist.

The failures are optional peers absent from this container (stx.types, stx.path, stx.compat, stx.etc, stx.gists, stx.db, stats, clew). scitex_db is not installed here at all, which is why test_db_delegates fails on both sides — the assertion this PR removed was never the reason it failed.

Storage is per-host PostgreSQL via scitex_dev.store; scitex-python must
stop naming, advertising or shipping the SQLite engine. Only docs/adr/ is
exempt fleet-wide, and it carried no occurrence.

Content:
- README.md / docs/05_ADDITIONAL_MODULES.md: the scitex.db section now
  documents the PostgreSQL client only. The example is rewritten against
  the real PostgreSQL signature (dbname/user/host/port) and methods that
  actually exist (execute, save_array, load_array, get_summaries). The
  advertised delete_duplicates is dropped: it is engine-specific and
  deprecated upstream.
- docs/sphinx/{core_concepts.rst,modules/io.rst}: provenance hashes are
  recorded "to the Clew store", not to a named engine.
- docs/guides/CROSSREF_API_CONFIGURATION.md, .env.d.examples: the local
  CrossRef mirror is described by what it is, a local database file.
- pyproject.toml: the clew extra note no longer enumerates the engine.
- scripts/maintenance/_pypi_packages.py: drop the stdlib-module entry.
  It only suppressed a PyPI mapping that does not exist, so behaviour is
  unchanged and an import of it would now be visible instead of silently
  classified.

Removed artefacts:
- .scitex/clew/runtime/db.sqlite and
  examples/_legacy/scitex/clew/multi_parent/scitex/clew.db: two engine
  database files committed as run leftovers. The second is already
  covered by .gitignore (**/scitex/clew.db). Nothing references either.
- examples/_legacy/scitex/db/_sqlite3/**: twelve zero-byte placeholder
  example files whose paths were the only thing naming the engine.
- examples/_legacy/notebooks/19_scitex_db.ipynb: a legacy notebook that
  existed to demo the engine class; its two index entries in
  00_SCITEX_MASTER_INDEX.ipynb are removed with it so no link dangles.
- examples/_legacy/notebooks/27_scitex_scholar.ipynb: the hand-rolled
  direct-database impact-factor lookups are removed; Scholar is already
  constructed with enrich_by_default=True, which is what supplies that
  metadata. Those cells were non-executable legacy stubs already.

Tests:
- tests/integration/test_integration.py: drop
  `assert hasattr(stx.db, "SQLite3")` from test_db_delegates. It asserted
  a mechanism that must not be advertised; the PostgreSQL and check_health
  assertions in the same test remain.
- tests/integration/test_cross_package_imports.py: drop
  "scitex_io._load_modules._sqlite3" from the auto-generated list. That
  module ships in neither the scitex-io checkout nor the installed
  package, so the entry was already dead.

Note the enumeration: `git grep -I` hides binary matches and no content
grep can see a path, so this was verified with three passes — content
without -I, `git ls-files | rg -i`, and a full untracked rg. All zero.
ywatanabe1989 added a commit that referenced this pull request Aug 30, 2026
…ective

PR #361 (chore/purge-sqlite) swept this repo for the same directive a few
hours earlier. Both branches reach 0 on the agreed measurement, so one has
to go; this commit takes everything #361 did better so the survivor is a
strict superset, and #361 is closed.

From #361:
  - Delete examples/_legacy/scitex/db/_sqlite3/ (12 files). They are EMPTY
    placeholders, so the content grep never saw them -- only their paths
    carry the retired engine's name. Path-level traces count too.
  - Better wording: docs now name the Clew store (the actual SciTeX
    provenance store, per the 'Provenance Tracking (Clew)' section) instead
    of a generic 'local database'; the schematic box is a 'verification
    store'; the CrossRef entries say 'database file'.

Kept from this branch, which #361 lacks:
  - .gitignore gains .scitex/clew/, so the deleted runtime artefact cannot
    reappear on the next run. Verified with git check-ignore.
  - The README example passes password from the environment and INSERTs the
    row that save_array(ids=1) then targets.
@ywatanabe1989

Copy link
Copy Markdown
Collaborator Author

Closing in favour of #362, which now carries everything this PR did plus two things it was missing. Two PRs were opened against the same SQLite-eradication directive a few hours apart; this is the deduplication, not a rejection — the work here was good and most of it is now in #362.

Both branches reach the same measurement. Using git grep -a -i -n sqlite <REF> | grep -a -vE ':(docs/adr/|build/|.*egg-info/|vendor/)': origin/develop = 58 lines, chore/purge-sqlite = 0, feat/eradicate-sqlite-vocabulary = 0. Positive control (store instead of sqlite) returns hits on both refs, so neither zero is an instrument failure.

What #362 took from this PR — verified by diffing the two branches, not assumed:

  • Deleting examples/_legacy/scitex/db/_sqlite3/ (12 files). This PR found them and chore(sqlite-out): eradicate SQLite vocabulary from scitex-python (58 -> 0) #362 originally did not. They are empty files (all e69de29, the empty blob), which is exactly why a content grep never saw them — only their paths carry the retired engine's name. Path-level traces count under the directive, so this was a real gap.
  • The better docs wording. This PR named the Clew store where chore(sqlite-out): eradicate SQLite vocabulary from scitex-python (58 -> 0) #362 had written a generic "a local database" — and Clew is the actual SciTeX provenance store (the section is literally titled "Provenance Tracking (Clew)"), so naming it is more accurate, not just nicer. Same for the schematic box being a verification store rather than a local DB, and the CrossRef entries saying "database file". All adopted.

What #362 has that this PR does not:

  • .gitignore gains .scitex/clew/. Both PRs delete the committed .scitex/clew/runtime/db.sqlite, but only chore(sqlite-out): eradicate SQLite vocabulary from scitex-python (58 -> 0) #362 stops it coming straight back on the next run — nothing in the ignore file covered that path. Verified: git check-ignore -v now reports .gitignore:927:.scitex/clew/.
  • A README example with two corrections: password passed from the environment, and an INSERT before save_array(..., ids=1) so the row that call targets actually exists.

CI is a wash, and the one red check belongs to neither of us. pytest-matrix-on-ubuntu-py3.13 fails identically on both branches with Fatal Python error: Segmentation fault (core dumped), retried three times by the workflow. It is an interpreter crash, not a test assertion, it reproduces on both changesets, and the tests workflow was already failing on develop before either PR existed. Worth its own issue; it is not caused by this work. Every other check passes on both.

Worth recording for whoever picks up the py3.13 segfault: test_db_delegates passed in this PR's CI run, which confirms CI installs the scitex[db] extra and that dropping the SQLite3 assertion was safe.

The parallel duplication also exists in scitex-template (#64 vs #65) and still needs resolving there.

@github-actions github-actions Bot locked and limited conversation to collaborators Aug 30, 2026
@ywatanabe1989
ywatanabe1989 deleted the chore/purge-sqlite branch September 7, 2026 19:25
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant