feat(release-tracking): add release-tracking system with MangaUpdates and Nyaa sources#11
Merged
Conversation
Deploying codex with
|
| Latest commit: |
4f7fd3b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://628c265f.codex-asm.pages.dev |
| Branch Preview URL: | https://release-tracker.codex-asm.pages.dev |
…s schema Introduce the schema and repository layer for tracked-series release discovery. Establishes the data model that downstream release-source plugins (Nyaa, MangaDex, Suwayomi, MangaUpdates) and the metadata-provider piggyback path will write announcements against. - migration: new m20260501_000067_create_release_tracking adds two tables. series_tracking is a 1:1 sidecar on series (FK cascade) carrying the tracked flag, status, latest known external chapter/volume, and per-series overrides. series_aliases stores matcher-oriented aliases with an alongside Unicode-normalized form. - entities: SeaORM models for both new tables, with reverse relations wired on the series entity (has_one tracking, has_many aliases) so cascades propagate cleanly. - repositories: SeriesTrackingRepository (upsert with Option<Option<T>> clear-vs-leave-alone semantics, status validation, list/count of tracked IDs) and SeriesAliasRepository (idempotent create, bulk_create with insert counting, find_by_normalized for cross-series matching, delete_by_source_for_series for refreshing metadata-derived aliases without touching manual ones). - normalization: Unicode-aware lowercase + strip non-alphanumeric + collapse whitespace, used to compare incoming release titles against stored aliases. External IDs reuse the existing series_external_ids table rather than a parallel structure. series_alternate_titles is intentionally untouched - it is purpose-built for labelled localized titles, which is the wrong shape for arbitrary matcher aliases. Includes unit and integration tests covering entity normalization, status validation, repo CRUD, idempotent inserts, and FK cascade on series delete.
Wires up the user-facing surface on top of the schema landed in dccba8c.
Admins can now flip a series to tracked, manage matcher aliases, and bulk-
mark via the existing series toolbar.
Backend:
- BackfillTrackingFromMetadata task seeds series_aliases from
series_metadata.title/title_sort and alternate titles. Idempotent on
re-run; per-series error isolation; never modifies the tracked flag.
- Five HTTP endpoints under /api/v1/series/{series_id}: GET/PATCH
/tracking and GET/POST/DELETE for /aliases. PATCH uses
Option<Option<T>> via a double_option serde helper so JSON null clears
a field while omitted leaves it alone. Idempotent alias create returns
200 instead of 201 on duplicate. New "Tracking" OpenAPI tag.
Frontend:
- TrackingPanel Mantine card on series detail (admin-only render) for
the tracked toggle, status, chapter/volume announce flags, latest
known chapter/volume, and the alias list with inline add/remove.
- useSeriesTracking and four sibling hooks plus a tracking.ts API
client.
- "Mark as Tracked" / "Mark as Untracked" entries added to the existing
bulk-selection toolbar; fans Promise.allSettled per-series PATCH calls
rather than introducing a bulk endpoint at this scale.
Tests added on both sides (backfill handler, tracking HTTP integration,
TrackingPanel component, and bulk-toolbar tracking actions). OpenAPI
spec and generated TypeScript types regenerated.
Establishes the dedup-keyed announcement ledger that release-source
plugins (and the in-core metadata-derived path) write into, plus the
read API that backs the inbox UI.
Schema additions:
- release_sources: one row per logical feed a plugin (or core)
exposes. Composite unique on (plugin_id, source_key) so a single
plugin can expose many sources (e.g., one per Nyaa uploader). The
literal "core" plugin_id is reserved for synthetic in-core sources
(metadata-derived path).
- release_ledger: dedup-keyed announcement table with FK cascade to
series and source. Two dedup paths enforced at the schema level:
(source_id, external_release_id) unique, plus a partial unique
index on info_hash where present so torrent sources collapse
cross-source duplicates. Partial index on state='announced'
optimizes the inbox hot path.
Repositories:
- ReleaseSourceRepository: CRUD, find_by_key, get_or_create (for
synthetic sources), list_enabled, and poll-status helpers
(record_poll_success clears errors and bumps last_polled_at;
record_poll_error preserves last_polled_at so users can see when
data last arrived).
- ReleaseLedgerRepository: idempotent record() that returns
RecordOutcome { row, deduped } so callers can tell insert from
dedup without re-querying. list_for_series and list_inbox/
count_inbox support combined filters (state, series, source,
language).
HTTP API:
- GET /series/{id}/releases — per-series, paginated, optional state
filter
- GET /releases — cross-series inbox with state/series/source/
language filters
- PATCH /releases/{id} plus convenience POST .../dismiss and
.../mark-acquired
- GET /release-sources, PATCH /release-sources/{id}
- POST /release-sources/{id}/poll-now is a deliberate stub returning
HTTP 501 with permission and 404 checks already wired, so the
task-queue swap-in is body-only
Permissions reuse the existing series surface (SeriesRead/Write for
ledger ops, PluginsManage for source admin). State changes emit
SeriesUpdated{fields:["releases"]} events for now; the dedicated
ReleaseAnnounced event lands with the UI work.
Adds repository unit tests covering cascade behavior both directions,
dedup priority, validation, and source state-tracking helpers, plus
HTTP integration tests for pagination, filters, state transitions,
error responses, and permission gating. OpenAPI spec and TypeScript
types regenerated.
…ability-gated reverse-RPC
Plugins can now declare a `release_source` capability and call a `releases/*`
reverse-RPC namespace. Dispatch is gated by a uniform permission layer that
also covers existing namespaces, closing the survey-identified gap where
manifests declared capabilities but the host didn't enforce them at dispatch.
Protocol additions:
- `ReleaseSourceCapability` (`kinds`, `requires_aliases`, `requires_external_ids`,
`can_announce_chapters/volumes`, `default_poll_interval_s`) plus
`ReleaseSourceKind` (`rss-uploader|rss-series|api-feed|metadata-feed`).
- Reverse-RPC method names: `releases/list_tracked`, `releases/record`,
`releases/source_state/{get,set}`. Plugin-side `releases/poll` defined for
the upcoming polling task.
Reverse-RPC handler (`releases_handler.rs`):
- `list_tracked` paginates tracked-series rows and only includes aliases /
external IDs the manifest asked for. Source naming `plugin:<name>` stripped
before matching against `requires_external_ids`.
- `record` resolves the per-series confidence threshold (or default 0.7),
validates and threshold-gates the candidate, and writes through the existing
ledger repo (idempotent on `(source_id, external_release_id)` and
`info_hash`).
- `source_state/get` returns `(etag, last_polled_at, last_error, last_error_at)`;
`source_state/set` accepts only `etag` (the rest are host-controlled).
- Every method asserts the source's `plugin_id` matches the calling plugin —
rejects with `AUTH_FAILED` otherwise.
Permission enforcement (`permissions.rs` + dispatcher in `rpc.rs`):
- Single mapping table `method → RequiredCapability` is the source of truth;
storage methods are explicitly `AlwaysAllowed` (per-(user, plugin) scoping
is the existing isolation, so no behavior change for storage callers).
- Dispatcher: `Denied → AUTH_FAILED`, `UnknownMethod → METHOD_NOT_FOUND`,
pre-init reverse-RPC also denied.
- `RpcClient` now holds a mutable `ReverseRpcContext` (storage handler +
releases handler + capability snapshot) behind `Arc<RwLock<…>>`.
`PluginHandle::start()` populates the capability snapshot post-`initialize`
and installs the releases handler when the manifest declares
`release_source`.
Matcher / candidate (`services/release/`):
- `ReleaseCandidate` mirrors the spec wire format. `evaluate(candidate,
threshold)` validates required fields, NaN/range, and a 1-hour clock-skew
grace on `observed_at`.
- `resolve_threshold(per_series_override)` falls back to 0.7 for
None/NaN/out-of-range overrides.
- `AcceptedCandidate::into_ledger_entry(source_id)` is the typed bridge into
the existing repository.
Tests added across the new modules; full lib suite still passes.
Wire periodic invocation for the release-tracking subsystem: scheduler
enqueues poll tasks; tasks call the plugin's releases/poll method, run
returned candidates through the matcher + threshold gate, and write
accepted candidates to the ledger.
- Add PollReleaseSource task variant + handler. Resolves source → plugin,
invokes releases/poll, applies per-series threshold overrides, persists
last_polled_at / etag / last_error. Plugins may also stream candidates
via the existing releases/record reverse-RPC; both paths land on the
same ledger and dedup naturally.
- Add ReleasePollRequest / ReleasePollResponse wire types and remove the
dead-code marker from the RELEASES_POLL method constant.
- Add scheduler integration: reconcile_release_sources() registers one
tokio-cron-scheduler job per enabled source row, called on startup and
from PATCH /release-sources/{id}. ±10% jitter at registration.
- Add interval-resolution helpers with a configurable global default
(release_tracking.default_poll_interval_s, default 86400 = daily).
Per-source override wins; values clamp to a 60s minimum.
- Add per-host backoff: plugin reports HTTP status via
ReleasePollResponse.upstream_status; handler records 429/503 signals
on a domain-keyed multiplier (doubling, cap 16x, 24h expiry); scheduler
consumes the multiplier at reconcile so a recently-throttled host
doesn't get re-polled immediately. State is shared between handler and
scheduler via TaskWorker::release_backoff().
- Wire the manual poll-now admin endpoint: returns 202 Accepted with the
task ID; 409 Conflict for disabled sources.
Tests added across the new modules; existing Phase 2 stub integration
tests replaced to reflect the new endpoint behavior.
Surface the delta between metadata-provider total chapter/volume counts (MangaBaka, AniList) and the highest locally classified chapter/volume as a passive UI signal. This is original-language publication data — it does not write to release_ledger or advance series_tracking.latest_known_*, which remain reserved for translation/scanlation feeds. Add upstreamChapterGap, upstreamVolumeGap, and upstreamGapProvider to SeriesDto and FullSeriesResponse. The fields are suppressed for untracked series, axis-disabled series (track_chapters / track_volumes false), and when the provider count is missing or local already meets/exceeds it. Compute the gap server-side in a pure helper so OPDS, Komga, and any future filter callers can reuse it without re-deriving. Round float math to one decimal place to suppress 0.0001-style noise; treat a missing local_max as zero so newly-tracked series surface the full upstream count rather than silently hiding it. Provider attribution falls back to a fixed priority order over series_external_ids (MangaBaka, AniList, MyAnimeList, MangaDex, Kitsu, ComicVine, OpenLibrary, then any other plugin source) because per-field provenance does not exist on series_metadata — every metadata-provider plugin merges into the same column. Manual / file-derived sources are not displayed because they don't supply upstream counts. Add SeriesTrackingRepository::get_for_series_ids so the list-series build path can fetch tracking rows for the page in a single query. Regenerate OpenAPI and TypeScript types so the frontend can consume the new fields without further plumbing. Includes unit and integration test coverage for the gap helper, the DTO build paths, and the new repository method.
…erence plumbing Adds the first real ReleaseSource plugin: a Node.js plugin that polls MangaUpdates per-series RSS feeds and announces new chapter / volume releases for tracked series. Also lands the supporting language-preference infrastructure that future release-source plugins will reuse. Backend: - New series_tracking.languages JSONB column with migration and TrackingUpdate / DTO / PATCH endpoint plumbing. - New release_tracking.default_languages server-wide setting (seeded to ["en"]). - New services::release::languages resolver: per-series → server default → hardcoded ["en"], with case-insensitive normalization and dedup. - releases/record reverse-RPC handler now advances series_tracking.latest_known_chapter / latest_known_volume on inserts, gated on the tracked flag, per-axis track_* flags, and the effective language list. Deduped rows do not move the high-water mark. SDK: - ReleaseSourceCapability and ReleaseSourceProvider types, plus a createReleaseSourcePlugin factory mirroring the existing metadata / sync / recommendation factories. - Generic HostRpcClient for non-storage reverse-RPC, exposed on InitializeParams.hostRpc. Uses a disjoint id range so it coexists with PluginStorage without coordination. Plugin (plugins/release-mangaupdates/): - Lightweight regex-based RSS parser that extracts chapter, volume, scanlation group, and ISO 639-1 language tag from MangaUpdates titles. Supports decimal chapters and volume-only bundles. Missing language tags fall back to a "unknown" sentinel (excluded by default). - Conditional-GET-capable fetcher with timeout and discriminated result shape (200 / 304 / error) so the host's per-host backoff sees real upstream status codes. - Language allowlist + admin-configured group blocklist. - Paginated releases/poll handler that streams candidates via releases/record. - User documentation covering setup, language preferences, group blocklist, configuration reference, and known limitations. Tests added across the resolver, the high-water-mark gate, the parser, fetcher, filter, and end-to-end pollSeries. Deferred (documented): - Per-(source, series) ETag state for true conditional GETs per series. Daily polls and small per-series feeds make this low priority. - Exposing the per-series language list on the releases/list_tracked payload so plugins can drop out-of-language candidates client-side before recording. Today the host's latest_known_* gate is the authoritative language enforcement; group blocklist is fully active. - Per-series tier of the group blocklist.
Add an EntityEvent::ReleaseAnnounced variant emitted on every non-deduped ledger insert from both the PollReleaseSource task handler and the releases/record reverse-RPC path. The event broadcaster is now plumbed through PluginManager → PluginHandle → ReleasesRequestHandler so plugins that stream candidates via reverse-RPC also produce SSE events. Build the user-facing surface on top of that: - Per-series Releases panel on series detail (grouped by chapter/volume, dismiss + mark-acquired actions, "open payload URL" link). - Four-variant Behind-by-N badges (translation/upstream × chapter/volume) inline next to the series header counts; translation badge navigates to the Releases section, upstream badge is informational only. - Top-level /releases inbox with state/language/series filters and a nav counter badge that bumps on incoming events and resets on visit. - SSE handler that fires deduped Mantine toasts and invalidates inbox, per-series ledger, tracking, and series queries. - Per-user notification preferences (language allowlist, plugin allowlist, per-series mute) backed by a Zustand store. - Source health admin page at /settings/release-tracking with per-row enable toggle, interval edit, last-poll status, and "Poll now". Tests cover the event emission paths, the Behind-by-N variants, the inbox filter and counter behavior, the per-series panel actions, the notification-preference store, and the source-admin page.
Adds the release-nyaa plugin as the first acquisition-pointer source, completing the three-signal release-tracking model: upstream gap (Phase 5), translation feed (MangaUpdates), and now where-to-acquire (Nyaa). Replaces a brittle external n8n flow with idempotent, alias-matched announcements. The plugin polls Nyaa user RSS feeds (and optional `q:<query>` search feeds for groups without an account) for an admin-configured uploader allowlist, parses titles into structured chapter/volume/format fields, and matches against tracked-series aliases via normalized exact match (0.95 confidence) or token+character bigram Sørensen-Dice fuzzy match floored at 0.85 ratio. Notable design choices: - One source row walks all uploader subscriptions (no admin endpoint exists yet for materializing one row per subscription); mirrors how release-mangaupdates polls all tracked series within one source row. - Normalization mirrors the host's normalize_alias Rust impl so exact matches between Nyaa titles and stored aliases are deterministic. - ETag is single-bucket on the source row; daily polls + small RSS bodies make per-subscription state slots a deferred optimization. - Title parser handles the user's mixed-format screenshot shapes (1r0n volume releases, v01-14 ranges, c126-142 ranges, decimal chapters, Digital/JXL hints). Also wires release-nyaa and release-mangaupdates (a Phase 6 wiring gap caught here) into docker-compose plugin dist mounts, the plugins-builder build/watch lists, and the GitHub Actions plugin matrices in both lint/test and per-plugin-binary build jobs. User docs covering uploader subscription syntax, alias-matching expectations, configuration reference, limitations, and risks land under docs/docs/plugins/release-nyaa.md. Plugin code ships with unit and end-to-end tests.
… plugin config UI
Notification preferences for the Releases inbox lived only in an in-memory
Zustand store, so language allowlists, plugin allowlists, and per-series
mutes were lost on page reload. Move them to durable storage and clean up
several adjacent rough edges in the plugin configuration UI.
Persistence:
- `release_tracking.notify_languages` and `release_tracking.notify_plugins`
are now seeded as server-wide settings (Array, default `[]`) under the
"Release Tracking" category. Admins manage them on the dedicated
/settings/release-tracking page; the generic ServerSettings page hides
the category to prevent two surfaces editing the same keys.
- Per-series mute moves to `user_preferences.release_tracking.muted_series_ids`
via the existing typed-preferences store (localStorage cache + debounced
server sync). The series detail "Releases" panel gains a per-series mute
toggle; the settings page shows the count and a "Clear all mutes" action.
- The SSE handler in useEntityEvents replaces its store-based shouldNotify
with a pure shouldNotifyRelease predicate that snapshots the latest
settings from React Query and the latest mute list from the user-prefs
store synchronously inside the callback. Bad JSON falls back to "no
filter" so corrupted values never silently bypass filtering.
- The release-announcements Zustand store is reduced to the unseen-count
badge counter; everything else is durable now.
Plugin configuration UI:
- Surface the `releaseSource` capability flag on PluginCapabilitiesDto so
the frontend can detect release-source plugins without parsing the
manifest JSON.
- Hide the Permissions / Scopes / Library Filter selectors on the Plugin
Config modal for plugins whose only capability is releaseSource,
userRecommendationProvider, or userReadSync. None of those flow through
the row-level RBAC gate, the scoped action UI, or the library filter,
so the empty selectors are misleading. Show an explanatory note instead.
- Drop the dead sync branch from getScopeData / getPermissionData and the
SYNC_SCOPES set; sync providers are gated by manifest capability only.
- Plugins without a manifest still see the "test connection to discover
capabilities" warning so first-time setup behavior is preserved.
Plugin marketplace + seed:
- Add release-mangaupdates and release-nyaa to the Official Plugins
carousel under a new "Releases" type (orange badge).
- Update seed-config.sample.yaml to drop the cargo-cult `metadata:read`
permission from non-metadata plugins (recommendations, sync, release-*);
it was decorative — those plugins are gated by manifest capability, not
RBAC. Comments inline explain why.
Adjacent cleanups:
- Replace the "Plugin sources" TagsInput on the Release Tracking settings
page with a MultiSelect populated from the registered-plugins list,
filtered to release-source plugins. Stale entries (in the allowlist but
not currently installed) render with a "(not installed)" suffix so admins
can see + remove them.
Tests cover the new shouldNotifyRelease predicate (mute, language allowlist
case-insensitivity, plugin allowlist, bad-JSON fallback), the no-permissions
branch on the Plugin Config modal for release/recommendation/sync plugins,
and the new dropdown behavior on the Release Tracking settings page.
Plugins implementing the release_source capability now declare their
desired source rows via releases/register_sources from onInitialize. The
host upserts each entry on (plugin_id, source_key), prunes rows the
plugin no longer declares, and reconciles the scheduler so new rows
start polling without a server restart. User-managed fields (enabled,
poll_interval_s) survive plugin restarts.
Threads the scheduler Arc through PluginManager -> PluginHandle ->
ReleasesRequestHandler so the handler can reconcile in-place. Adds
upsert / list_by_plugin / delete_by_plugin_excluding to the repo.
Extends ReleasePollRequest with sourceKey + config so plugins owning
multiple rows can dispatch directly.
feat(release-nyaa): one source row per uploader
Refactors the Nyaa plugin to materialize one release_sources row per
uploader subscription via releases/register_sources, with stable
kind:identifier source keys (user:tsuna69, query:luminousscans,
params:c=3_1&q=berserk). poll() now resolves a single subscription from
the host's per-poll config snapshot and fetches just that uploader's
feed, replacing the previous "one row, walk all subscriptions"
workaround. Each row gets its own ETag and last-error status.
Registration is deferred to a microtask + retries on METHOD_NOT_FOUND
to ride out the brief race where the host has not yet installed the
releases reverse-RPC handler after onInitialize returns.
feat(release-mangaupdates): auto-register single static source row
The plugin now registers one release_sources row ("MangaUpdates
Releases", kind=rss-series, sourceKey=default) on initialize so users
see a configurable row in Settings -> Release tracking out of the box.
No admin config is required.
…t-poll summary
Make release-tracking discoverable and actionable instead of an
empty-form setup wizard. Toggling a series to tracked now auto-seeds
matcher aliases, latest_known_*, and per-axis track_chapters/volumes
flags from data the system already has, so users don't have to manually
fill in the panel before notifications work. The Tracking panel
collapses to a one-line summary by default and is hidden entirely on
libraries with no covering release-source plugin.
Backend
- New seed_tracking_for_series service: Latin-script aliases from
series.name + metadata + alternate titles, latest_known_* from local
max chapter/volume, track_* inferred from book classification.
Aliases are append-only; tracking flags overwrite on re-seed.
- PATCH /series/{id}/tracking runs the seed on a false→true tracked
flip before applying the user's patch (explicit overrides win).
- New POST /series/bulk/track-for-releases and untrack-for-releases
endpoints with per-series outcome reporting (tracked / skipped /
errored). Mirrors the existing bulk-mark-as-read shape.
- New release_sources.last_summary column populated by the poll task
with a one-line outcome ("Fetched 12 items, recorded 1 (7 already in
ledger), dropped 4 below threshold", "Up to date — upstream returned
304", etc.). Exposed on the source DTO.
- New GET /release-sources/applicability endpoint (SeriesRead) returns
whether any enabled release-source plugin applies to a given library
and the friendly plugin display names.
- BackfillTrackingFromMetadata task delegates to the seed service so
there's one canonical implementation across the per-series PATCH,
bulk endpoints, and the maintenance task.
Frontend
- New useReleaseTrackingApplicability hook drives three UI gates: the
per-series Tracking panel + Releases tab, the bulk-selection menu
entries, and the global navigation entry.
- BulkSelectionToolbar gains "Track for releases" / "Don't track for
releases" entries (gated on applicability), replacing the previous
N-PATCH loop with single bulk endpoint calls.
- TrackingPanel renders as a compact one-line summary by default
(status, latest known marks, alias count); details are
collapsible. Auto-seeding eliminates the empty-form first-track UX.
- Release tracking settings table surfaces last_summary under the
per-row last-polled timestamp and as the OK badge tooltip, so users
can see why a poll returned no announcements without grepping logs.
- Tracking panel + Releases tab hidden on series whose library has no
applicable release-source plugin, eliminating dead-end UI.
Tests cover seeding (Latin filter, idempotency, axis inference,
overwrite-on-re-seed, manual-alias preservation), the new bulk
endpoints, last_summary string formatting, and the updated
TrackingPanel and BulkSelectionToolbar UX.
…led SSE event
Several gaps in the release-tracking polling pipeline surfaced once it was
exercised in real use:
- The host's releases/list_tracked filter only stripped `plugin:` prefixes
from external-ID source strings, but metadata plugins (MangaBaka,
OpenLibrary, etc.) write IDs with `api:<service>` per the SDK convention.
As a result, MangaUpdates received zero IDs and reported "Fetched 0
items" even on series that had been cross-referenced. Strip both
`api:` and `plugin:` namespaces in strip_external_id_namespace.
- ReleasePollResponse gained parsed/matched/recorded/deduped counters.
Streaming plugins (Nyaa, MangaUpdates) record candidates via the
releases/record reverse-RPC mid-poll and return an empty `candidates`
array, so the host's `last_summary` always read "Fetched 0 items"
regardless of activity. Both plugins now report counters; the host folds
them into PollReleaseSourceResult via fold_streaming_counters, with a
`matched - recorded` fallback for older plugins that omit `deduped`.
- Concurrent "Poll now" requests no longer stack. enqueue_poll_now checks
for an in-flight pending or processing task on the same source_id and
coalesces onto it; the handler returns status="already_running" so the
user sees the dedup explicitly. Backed by a generic
TaskRepository::find_pending_or_processing_by_param helper that works
on both SQLite and Postgres.
- Added a POST /api/v1/release-sources/{id}/reset endpoint (admin-only)
that drops every release_ledger row for the source and clears its
transient poll state (etag, last_polled_at, last_error, last_summary).
User-managed fields (enabled, pollIntervalS, displayName, config) are
preserved. Surfaced as a red restore icon in the Release tracking
settings table behind a confirm dialog.
- ReleaseLedgerEntryDto now carries series_title joined from the series
row, so the inbox UI renders human labels instead of sliced UUIDs. The
Nyaa plugin also surfaces the post-page URL (from <guid isPermaLink>)
as payloadUrl instead of the .torrent download URL.
- New EntityEvent::ReleaseSourcePolled emitted at the end of every poll
task run (success and error paths). The Release tracking settings page
subscribes to it and invalidates the sources query, so last_polled_at
/ last_summary / status badges refresh in real time without a manual
reload.
Also drops the unused `tracking_status` column from series_tracking and
its DTO/handler/UI surface; publication status is sourced from upstream
metadata, not stored on the tracking sidecar.
Tests cover external-ID prefix matching, streaming-counter folding,
poll-now dedup, source reset (with user-field preservation), and the
new SSE event variant.
…API calls
MangaUpdates exposes the same series ID in two interchangeable forms: an
internal numeric key (e.g. 15180124327) used by every `/v1/series/...`
API endpoint, and a base36 encoding of that key (e.g. 6z1uqw7) used only
in public URLs. Metadata sources like MangaBaka scrape the public URL
and store the slug, so the external ID the plugin received was usually
the slug — and the API rejects that form with a 405, breaking polls
silently.
The two representations are pure arithmetic conversions
(parseInt("6z1uqw7", 36) === 15180124327), so the fix doesn't need an
extra lookup or any host plumbing. The fetcher gains
`normalizeMangaUpdatesId`, called from `feedUrl`, which:
- passes all-digit input through unchanged (legacy numeric IDs keep
working without round-tripping through base36, which would produce a
different number for slugs that happen to be all digits),
- decodes anything matching the base36 alphabet,
- returns null for out-of-alphabet input so callers can surface a
config-style error.
Tests cover the slug-to-numeric decode, the all-digit pass-through, the
trim-and-decode path, and rejection of bad input.
Real-world Nyaa bundle titles (1r0n / danke-Empire / LuCaZ) mix volume ranges with bare-numeric chapter ranges and "extra" chapters separated by `,`, `+`, or `as`. The previous regex-pair parser dropped everything past the volume token, losing the chapter range and corrupting the series guess. Replace it with a tokenizer that anchors on the first chapter/volume token, then walks the release-info span aggregating spread tokens via min/max. Also split `Title A / Title B` into alias candidates so the matcher can score against either the JP or EN side, and capture the `(Omnibus Edition)` format hint.
…vive the filter
The MangaUpdates v1 RSS endpoint (`/v1/series/{id}/rss`) serves the
English release stream by design and ships titles without a `(xx)`
language tag. The parser was returning the `UNKNOWN_LANGUAGE` sentinel
for those, and the client-side filter then dropped every item before
recording — explaining why polls reported `Fetched N items, recorded 0`
even after the slug-to-numeric ID fix.
Default `language` to `"en"` in `parseTitle`. An explicit `(es)` /
`(id)` / etc. still wins when present, so multilingual feeds (or any
future API change) keep working. The host's per-series language list
remains the authoritative gate downstream — this only changes what the
plugin sends.
`UNKNOWN_LANGUAGE` stays exported for callers that want to surface "no
tag detected" explicitly, but the parser no longer produces it.
Tests updated to reflect the new default.
…URLs Some sources (Nyaa especially) carry two URLs per release: a human-readable landing page and the actual fetch URL (a .torrent, magnet link, or direct download). Previously the Nyaa plugin had to pick one for payloadUrl and discard the other, so the inbox row's external-link icon either pointed at the post page (no one-click acquire) or the .torrent (no readable context). Add a second URL channel to the release-source SDK: - payloadUrl keeps its meaning: the human-readable landing page. - mediaUrl is the optional direct-fetch URL. - mediaUrlKind classifies it: "torrent" | "magnet" | "direct" | "other", so the UI can render a kind-specific icon (download, magnet, cloud-down). The two fields travel together; the host validates the pair and rejects candidates that set one without the other. Both are nullable on the ledger, so single-URL sources (MangaUpdates) leave them empty and the extra icon hides. Nyaa now populates payloadUrl from <guid> (post permalink) and mediaUrl from <link> (.torrent) with kind="torrent". When the permalink is missing, falls back to the torrent as payloadUrl alone to avoid two icons pointing at the same URL. Schema: nullable media_url (VARCHAR 2048) + media_url_kind (VARCHAR 32) columns on release_ledger. Tests added for SDK validation, ledger persistence, Nyaa candidate construction (both branches), and inbox icon rendering.
…bility endpoint HTTP-layer integration tests for the endpoints introduced in e9b5682. Unit tests already covered the inner logic (seed pass, plugin manifest parsing); these tests pin the wiring. PATCH /api/v1/series/{id}/tracking - false→true transition runs the seed: aliases inserted, latest_known_* populated from local books, track_chapters/track_volumes inferred - explicit user values in the same PATCH win over seeded values - seed does not re-run when tracked is already true (otherwise user-deleted aliases would silently come back) POST /api/v1/series/bulk/track-for-releases (and untrack counterpart) - happy path flips tracked + seeds aliases for each series - already-tracked series report outcome=skipped, count toward already_in_state, do not re-seed - missing series report outcome=skipped with "not found" detail (one bad id must not poison the whole bulk request) - untrack preserves aliases — toggle is soft, not destructive - non-SeriesWrite caller gets 403 GET /api/v1/release-sources/applicability - false when only metadata-only plugins are enabled - true for global (empty library_ids) plugins, both with and without a libraryId query - library-scoped plugin matches its library, not others - aggregates display names across multiple release-source plugins - 401 when unauthenticated; disabled plugins drop out
The Release tracking settings page relied solely on the `release_source_polled` SSE event to refresh source rows after a "Poll now" finishes. Very fast polls can land before the event reaches the client, leaving `lastPolledAt`, `lastSummary`, and `lastError` stale until the user reloads. Apply the same belt-and-braces pattern used by Series exports: while any `poll_release_source` task is pending or running in `useTaskProgress`, refetch the sources query every 5s, and invalidate immediately when a poll task flips to a terminal state. SSE remains the fast path; this is the safety net.
…elete
Rebuild the Releases inbox filters and actions around what users actually
need: typing UUIDs into a Series filter or scrolling a flat language list
were both dead ends, and there was no way to undo a misclicked Acquired
or Dismiss without resetting the entire source.
Backend:
- GET /releases/facets returns the languages, libraries, and series
present in the ledger under the active filter set, with row counts
and library hydration. Each dimension is excluded from its own
facet (Solr-style) so dropdowns never collapse to the active
selection.
- GET /releases now accepts state=all (no state filter) and a
libraryId scope; LedgerInboxFilter gained the same fields.
- DELETE /releases/{id} hard-deletes a row and clears the owning
source's etag, so the next poll bypasses If-None-Match and the
upstream re-announces (the dedup logic re-records the row in
announced state once it can no longer find a matching id).
- POST /releases/bulk accepts {ids, action} for dismiss / mark
acquired / delete, all in single SQL statements. Bulk delete
also clears etags on the affected sources only, leaving every
other source's poll cache intact.
Frontend:
- Filter row replaced with four cascading Select inputs
(State / Library / Language / Series) populated from the new
facets endpoint; State picks up an "All" option, Series is
searchable and grouped by library with counts.
- Per-row action column gains a delete button; a sticky bulk
action bar appears once any rows are checkbox-selected, with
a confirmation modal for bulk delete that explains the
next-poll behavior.
- Bulk operations toast on success so 30-row deletes don't cost
30 toasts; per-row dismiss/mark-acquired stay silent (the row
vanishing is its own confirmation).
Tests on both sides cover the new filter shapes, facet self-
exclusion, etag-clear scoping, and the bulk + per-row UI flows.
…-poll dedup Inbox and the series-detail Releases panel were diverging in subtle ways (sort order, source label, missing actions, duplicated table JSX). Extract the shared surface and make both views identical where it matters. UI: - New shared components: ReleasesTable, ReleasesBulkActionBar, ReleasesBulkDeleteModal. Both pages compose them; row markup, action buttons, and bulk-delete confirm flow are now identical. - Series detail panel: collapsed by default, joins ReleaseSources for the display label (no more "source: 11111111…"), gains per-row delete and the full bulk action bar (Mark acquired / Dismiss / Delete + confirm). - Series detail layout: TrackingPanel and Releases panel moved below Custom Metadata so identifying data stays at the top. - Inbox + series panel rows render flat: each row carries its own Ch/Vol label, dropping the "blank cell on subsequent rows of the same chapter" grouping that made bulk selection ambiguous. Sort order: - Inbox: group every row of a series together (highest volume/chapter first), then break series ties by series.name ASC. Joins the series table so cross-series order is alphabetical instead of by UUID. Previously, a fresh poll batch split each series into "new" and "old" desc clusters by observed_at; now a series' chapter list reads as one contiguous descending sequence regardless of which poll surfaced each row. - Per-series ledger view (list_for_series) mirrors the same sort, so the series-detail panel matches the inbox. Backend: - Task dedup gains TaskType::dedup_params(): for task types whose identity lives in `params` (PollReleaseSource keyed on source_id), find_existing_task now filters by the JSON param in addition to task_type. Without this, two "Poll now" clicks for different sources were silently coalesced onto the first source's in-flight task. - ReleaseTrackingSettings tracks per-source poll/reset pending state so one row's spinner doesn't light up every other row. Tests added for: alphabetical cross-series sort, contiguous per-series chapters across observation batches, list_for_series chapter-desc ordering, panel bulk delete + confirm flow, per-source pending state.
…ed filter on series panel The "Hide dismissed / Show all states" anchor only switched between announced-only and all-states, which conflated two different things the user wanted to hide. For series with hundreds of chapters where most rows have been marked acquired, the default view was already clean (acquired is excluded by the `announced` filter), but the toggle labels suggested it was only about dismissed entries. Rename to a SegmentedControl with "New" (announced only, the default) and "All" (every state). "New" makes it explicit that acquired and dismissed are both hidden, which matches the use case where a long backlog of acquired chapters would otherwise drown out unhandled announcements. The cross-series Releases inbox page keeps its own richer state filter and is unchanged.
Plugin reverse-RPC handlers (notably releases/record) emitted entity events through a long-lived broadcaster captured at plugin init, bypassing the per-task recording broadcaster the worker creates in distributed deployments. As a result, release_announced events from MangaUpdates (and any future reverse-RPC emitter) never reached the web server's SSE stream when workers ran in a separate container, so users never saw notification toasts or the sidebar badge update. Reverse-RPCs now carry an optional parentRequestId, set by the plugin SDK via AsyncLocalStorage. The host's response reader routes them to the originating caller's task instead of dispatching itself, so the worker's recording broadcaster (set as a tokio task-local around handler.handle) propagates into the dispatcher. ReleasesRequestHandler reads the task-local at emit time, lands the event in tasks.result.emitted_events, and TaskListener replays it on the web server. Backwards compatible: plugins without parentRequestId fall back to dispatch-on-reader (the prior behavior). The plugin SDK auto-stamps the field via AsyncLocalStorage so plugin authors don't see it. Drops the now-unused with_event_broadcaster builders on PluginManager, PluginHandle, and ReleasesRequestHandler. Includes tests for the protocol field round-trip, the task-local helper, and the new no-broadcaster-in-scope path on releases/record.
… schedules Release-source polling cadence is now a cron expression end-to-end, matching the rest of the app (library scans, dedup, thumbnails, PDF cache cleanup). The previous `poll_interval_s` field forced a leaky abstraction: seconds were mapped to wall-clock-aligned cron strings inside the scheduler, jitter was applied at registration, and admins reasoned in two different units. Backend changes: - `release_sources.poll_interval_s INTEGER NOT NULL` becomes `release_sources.cron_schedule TEXT NULL`. NULL means "inherit the server-wide default" so a settings change propagates to every uncustomized row without per-row writes. - New seeded setting `release_tracking.default_cron_schedule` (default `0 0 * * *`) is the resolution fallback. Compile-time default kicks in only if the setting row is missing. - `services/release/schedule.rs` collapses to two helpers (`read_default_cron_schedule`, `resolve_cron_schedule`); seconds-based jitter, backoff multiplication, and the `secs_to_cron` approximator are gone. Per-host backoff stays in the polling task, where 429/503 already drives multiplier state. - Scheduler feeds the resolved cron straight to `tokio-cron-scheduler` via the existing `normalize_cron_expression` helper (5- or 6-field accepted) and only re-registers when the effective expression actually changes. - `default_poll_interval_s` removed from the plugin manifest protocol, the TS SDK, and the Nyaa + MangaUpdates plugins. Polling cadence is a host concern, not a plugin one; new rows simply inherit the server default. - API DTO exposes `cronSchedule` (raw, may be null) and `effectiveCronSchedule` (resolved). PATCH uses double-Option semantics so callers can clear the override with `null`. Frontend changes: - `ReleaseTrackingSettings` row replaces the seconds NumberInput with the shared `<CronInput>`. When inheriting, the row shows the resolved default in human form with `(Default)` plus an "Override" affordance; when overridden, the editor renders inline with a "Reset to default" link. Empty input clears the override. Tests updated across repository, scheduler, plugin handler, integration suites, and the affected frontend specs.
The `release_tracking.default_cron_schedule` setting was seeded but had no UI: ServerSettings hides the "Release Tracking" category in favor of the dedicated page, and that page only handled notification filters and per-source rows. Admins had no way to change the server-wide default. Add a "Default schedule" card to the Release Tracking settings page that reads/writes the setting via the existing settings API and uses the shared `<CronInput>`. Saving invalidates the source list query so every inheriting row's "(Default)" label refreshes immediately. Also fix two related bugs surfaced while testing: - `ReleaseSourceDto.cron_schedule` was annotated with `skip_serializing_if = "Option::is_none"`, so inheriting rows arrived on the wire as `undefined` rather than `null`. Drop the attribute so the field is always present, eliminating the omit-vs-null ambiguity for clients. - `ReleaseSourceRow` checked `cronSchedule !== null`, which was true for `undefined`, so the editor opened pre-filled on every inheriting row. Switch to a `Boolean(cronSchedule)` check that handles `null`, `undefined`, and empty strings uniformly.
…s/chapters Add an `ignored` ledger state distinct from `dismissed` (a user decision) that ingestion applies automatically when a release directly matches a volume or chapter the user already owns. Direct matches only: a release for "Vol 1" matches an owned whole vol 1 but not a "Ch 5 of vol 1" book, and a release for "Ch 12" never matches based on owned volume metadata (chapter→volume mapping is unreliable upstream). When no book in the series carries volume metadata, fall back to volumes_owned_count for volume-only releases. Compute the initial state in the poll handler and the reverse-RPC record handler, with a per-poll cache to avoid N+1 owned-key lookups, and skip the release_announced SSE emit when a row lands as anything other than `announced` so notifications and inbox stay quiet. Add `ignore` and `reset` bulk actions to /api/v1/releases/bulk; reset returns any state to `announced`, giving users a universal undo for mistaken dismiss/acquire/ignore. The series panel's "New" filter already filters to `announced` and the cross-series inbox defaults to the same, so the new state is hidden from both surfaces by default and visible under "All" or the new "Ignored" filter option. Includes unit tests for the predicate, repository tests for the owned-keys query, ledger tests for initial_state, and API tests for the new bulk actions.
The plugin config schema's `type` field was a narrow union ("number" |
"string" | "boolean") that the host never enforced — `admin_config` is
forwarded to plugins as opaque JSON. The narrow union just rejected
valid manifests without buying any safety.
Widen `ConfigField.type` to a free-form `string` and `default`/`example`
to a `JsonValue` alias (mirrors the Rust `serde_json::Value`). Add the
`JsonValue` alias to the SDK's public exports.
Adopt the new shape in release-nyaa: `uploaders` is now declared as
`type: "string-array"` with a JSON-array default. The parser accepts
`string[]` (preferred) or the legacy comma-separated string, so
existing stored configs and CLI/env paths keep working without a
migration. Tests cover the array path, empty array, non-string
filtering, and the no-comma-split contract for arrays.
Update the Rust DTO doc comment to reflect that `field_type` is a
documentation hint, not a wire constraint.
Loosen buildQuery's parameter type to `object` so the typed ReleaseInboxParams / SeriesReleaseListParams / ReleaseFacetsParams interfaces can be passed without requiring an index signature. Switch MediaUrlIcon's KIND_META lookup to a ternary so the meta fallback isn't polluted by `kind` itself when it's an empty string, keeping the destructured label/Icon access well-typed.
…orker, build SDK first Three independent changes bundled because they all surfaced while updating Playwright scenarios for features added since v1.10.1. Screenshot scenarios: - Add MangaUpdates plugin install to the plugin-store flow so the release-tracking scenario has a working source. - New scenarios for releases, series-detail extras, and per-library scheduled jobs covering tracking panel, bulk metadata edit modal, series actions menu, reset confirmation, and job editor. - Extend settings scenario with release-tracking, plugin-storage, exports, and integrations pages. - Expose the screenshots backend on host port 8081 for debugging. PDF worker (web): - Bundle the PDF.js worker locally via Vite's `new URL(..., import.meta.url)` instead of fetching from `//unpkg.com`. The protocol-relative URL resolved to http:// when the page was served over plain HTTP and Chromium does not follow cross-origin redirects on dynamic imports, breaking the reader. Reverts to the original pattern that was lost in an unrelated PDFium commit. Plugin build order (Makefile): - Build sdk-typescript first; every other plugin imports its compiled dist, and esbuild fails fast on missing exports if the SDK dist is stale. Previously alphabetical iteration built release-mangaupdates and release-nyaa before the SDK they depend on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end release tracking so users can subscribe to series, ingest upstream releases from plugin-provided sources, and triage them through a unified UI. Ships two official source plugins (MangaUpdates RSS, Nyaa uploader feeds) on top of a new
release_sourceplugin capability.Backend
series_tracking,series_aliases, and release-ledger schema (migrations 72–79) with repos, HTTP API, and DTOs.releases/register_sourcesreverse-RPC for plugin-driven source registration.release_sourceplugin capability with capability-gated reverse-RPC; reverse-RPC events now replay through a task-local broadcaster.ReleaseAnnouncedevent,mediaUrl/mediaUrlKindfields for direct fetch URLs, upstream-publication gap on the series DTO, and persisted notification preferences.Plugins
release-mangaupdates: RSS fetcher with language preference plumbing, base36 → numeric ID decoding, and language-default fallback so items survive filtering.release-nyaa: uploader-feed source with bundle tokenization and alias-form title splitting.Frontend
/releases) with facets, bulk actions, per-row delete, and a New/All segmented filter on the series panel.