Skip to content

refactor: move tables into app/core/db - #83

Merged
dimslaev merged 5 commits into
refactor/backend-corefrom
refactor/backend-core-db
Oct 19, 2026
Merged

dimslaev merged 5 commits into
refactor/backend-corefrom
refactor/backend-core-db

Conversation

@dimslaev

Copy link
Copy Markdown
Owner

Step 2 of the backend architecture plan.

What moved

  • Every table and its stored enums now lives in app/core/db/, one module per table: publisher.py, article.py, tag.py, story.py, ledger.py, run.py, user.py, like.py, subscriber.py, event.py. UserBase, PublisherBase and ArticleBase moved with their tables.
  • Non-table shapes stay in each domain's models.py. app/newsletter/models.py and pipeline/models.py held only tables, so they are gone.
  • record_reject, CONTENT_CAP and CANDIDATE_CAP moved into ledger.py (was pipeline/rejects.py). record_reject now takes a LedgerItem TypedDict, so core does not import pipeline.types. The pipeline's Candidate still fits it.
  • kind_from_url moved into article.py (was pipeline/url_kind.py). like_counts_subquery moved into like.py (was in catalog/articles.py).
  • One engine: app/core/db/engine.py holds engine and get_session. app/core/db.py and pipeline/db.py are gone. deps.SessionDep uses get_session.
  • app/core/db/__init__.py imports every table module, so migrations/env.py imports app.core.db and needs no per-domain line.
  • Migrations moved from app/alembic to backend/migrations. The alembic config is [tool.alembic] in backend/pyproject.toml. alembic.ini is gone, and env.py sets up its own logging.
  • mypy and ruff excludes now name migrations. docs/backend.md points at the new paths. tests/pipeline/test_rejects.py moved to tests/core/test_ledger.py.

Behaviour changes

  • The pipeline (pipeline.run, pipeline.report, find_missing_tags) now connects through app.core.settings, as ADR 7's follow-up in the plan says. It needs PROJECT_NAME, POSTGRES_SERVER and POSTGRES_USER. The box already feeds the same .env to every unit.
  • The comments in deploy/agentique-pipeline.service and deploy/agentique-report.service changed. ExecStart did not change, so reinstalling the units is optional. Copy them when convenient.
  • prestart.sh still runs alembic upgrade head from backend/, which now reads pyproject.toml.

Bug fixes

fix: percent-encode database credentials in the dsn

PostgresDsn.build does not escape the user or password. A password with @, / or # produced a broken URL. The old pipeline engine quoted them, so merging the engines would have broken it. Both SQLALCHEMY_DATABASE_URI and MCP_DATABASE_URI now quote the user and password. Tests are in tests/core/test_settings.py.

fix: declare the indexes and column types the migrations built

alembic check failed on the base branch. The models were missing the three article indexes (score, published_at, the hnsw embedding index) and pipeline_run.publishers. They also declared VARCHAR, JSON or naive timestamps where the migrations built TEXT, JSONB and timestamptz (newsletter_subscriber, scored_url.url/created_at, pipeline_run.sources). The models now match the database, so no migration is needed.

Tests added

  • tests/core/test_db.py runs alembic's compare_metadata against the migrated test db, so CI now fails on schema drift. It also checks that every table lives in app.core.db, that the engine logs in with the settings credentials, that the request session closes afterwards, and that the routes and the pipeline share one engine.
  • tests/core/test_ledger.py adds Postgres-backed tests. A missing detail is stored as SQL NULL. Recording a URL again replaces its row. NUL bytes and an unparsable date still commit.

How I verified

  • On a local Postgres 16 with pgvector, alembic upgrade head on an empty db and then alembic check both pass. alembic check also passes on the prestart-migrated db. alembic revision --autogenerate produces an empty revision.
  • pytest: 603 passed. /search fails locally because the HF model download is blocked here. It passes in CI.
  • ruff check and format, mypy and ty as CI runs them, and cd backend && mypy app/core (strict, 0 errors) all pass. Strict mypy over app scripts reports 126 errors, which was 127 before. The banned-names hook passes.
  • typos only flags .claude/skills/curate/regression.md, which was already flagged before this PR.
  • scripts/generate-client.sh leaves frontend/src/client unchanged, and the OpenAPI JSON is identical to the base branch.

Generated by Claude Code

@claude

claude Bot commented Oct 18, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

* refactor: split backend domains into app/features

* refactor: type app/features for mypy strict

* fix: keep links on a shared platform's subdomains with the aggregator

* fix: drop a newsletter product whose picked link is past the search results

* test: cover approve and reject against the schema, and publisher resolution edges

* chore: point the end-of-file exclude at the moved email templates

* refactor: tidy comments and test helpers from the features split

* refactor: move mcp tools and scripts into app/entry (#85)

* refactor: expose what entry needs through each feature's service

* refactor: split the mcp tools into app/entry/mcp and fold the router into main

* refactor: move scripts and the timer entrypoints into app/entry/cli

* test: cover the mcp token gate end to end over http

* test: cover the cli entrypoints' targets and exit codes

* fix: fail the backup when pg_dump writes nothing

* refactor: make app/entry pass strict mypy

* fix: keep sentry off in development

* chore: name the seed module in restore-db's hint without backticks

* docs: say what tests-start.sh does with its argument

* test: return the tool result's error flag and text from the gate tests' helper

* fix: stop importing the email module from configuring the root logger

* fix: require the alembic that reads its config from pyproject

* test: check the collect run leaves the root logger alone

* docs: say where command-line entrypoints live

* refactor: name the newsletter service alike in every mcp tool

* test: colocate backend tests beside their modules (#86)

* test: colocate backend tests beside their modules

* test: cover the curation tools, sign-in refusals, publisher lookups and the known-URL filter

* fix: fail the daily report when its mail does not send

* test: pin that the report reaches the send, and clean up the filter's publisher

* ci: enforce backend import layers (#87)

* fix: run mypy and ty in pre-commit with the backend config

* ci: enforce backend import layers

* fix: name the exact claude-code-action tag in its pin comments

* ci: deploy only what the box runs (#88)

* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
@dimslaev
dimslaev merged commit 46ff955 into refactor/backend-core Oct 19, 2026
5 checks passed
@dimslaev
dimslaev deleted the refactor/backend-core-db branch October 19, 2026 22:43
dimslaev added a commit that referenced this pull request Oct 19, 2026
* refactor: move shared backend mechanism into app/core

* refactor: reword the check_link module docstring

* fix: read a feed date with no timezone as utc

* test: cover token rejection, the daily mail send and github lookups

* test: spell unparsable the way typos expects

* refactor: move tables into app/core/db (#83)

* fix: percent-encode database credentials in the dsn

* fix: declare the indexes and column types the migrations built

* refactor: move tables into app/core/db

* test: cover the schema against migrations, the engine and ledger writes

* refactor: split backend domains into app/features (#84)

* refactor: split backend domains into app/features

* refactor: type app/features for mypy strict

* fix: keep links on a shared platform's subdomains with the aggregator

* fix: drop a newsletter product whose picked link is past the search results

* test: cover approve and reject against the schema, and publisher resolution edges

* chore: point the end-of-file exclude at the moved email templates

* refactor: tidy comments and test helpers from the features split

* refactor: move mcp tools and scripts into app/entry (#85)

* refactor: expose what entry needs through each feature's service

* refactor: split the mcp tools into app/entry/mcp and fold the router into main

* refactor: move scripts and the timer entrypoints into app/entry/cli

* test: cover the mcp token gate end to end over http

* test: cover the cli entrypoints' targets and exit codes

* fix: fail the backup when pg_dump writes nothing

* refactor: make app/entry pass strict mypy

* fix: keep sentry off in development

* chore: name the seed module in restore-db's hint without backticks

* docs: say what tests-start.sh does with its argument

* test: return the tool result's error flag and text from the gate tests' helper

* fix: stop importing the email module from configuring the root logger

* fix: require the alembic that reads its config from pyproject

* test: check the collect run leaves the root logger alone

* docs: say where command-line entrypoints live

* refactor: name the newsletter service alike in every mcp tool

* test: colocate backend tests beside their modules (#86)

* test: colocate backend tests beside their modules

* test: cover the curation tools, sign-in refusals, publisher lookups and the known-URL filter

* fix: fail the daily report when its mail does not send

* test: pin that the report reaches the send, and clean up the filter's publisher

* ci: enforce backend import layers (#87)

* fix: run mypy and ty in pre-commit with the backend config

* ci: enforce backend import layers

* fix: name the exact claude-code-action tag in its pin comments

* ci: deploy only what the box runs (#88)

* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants