diff --git a/CHANGELOG.md b/CHANGELOG.md index b68539a..de3e8cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,53 @@ may change what the API or the configuration means, and says so here. is made with wants some 4 GB at its last step, more than a Docker VM shared with other containers may have. A build from source takes the same variable; CI and the published image are unchanged. +- A **Reading** panel on the profile page: whether the posters of works for + adults are shown as they are or veiled, kept with the account + (`reader.adultPosters`) so the choice follows it from one browser to the + next. The switch in the bar's *Language and reading* menu is the same + choice for whoever is signed in, and stays this browser's for a visitor. +- Turning the WebP or the AVIF copies off, while some are kept, asks whether + to delete them too. `DELETE /api/v1/media/variants/{format}` deletes every + copy of a format once it is off, in the background, which the Media page + follows (`purging` in `GET /api/v1/media/status`). + +### Changed + +- The WebP copies are off until `media.webp` is set: the pages are given the + pictures themselves — with an AVIF `` over them where the night's + copies reach as wide — and an address of a WebP copy answers the copy kept + or sends to the original, making none. A server that wants the lighter + pages turns them on, on the Media page. `media.variants` prepares the + placeholders, and the first copies only while the WebP copies are on; + `GET /api/v1/auth/options` says whether they are (`webp`). +- The front page no longer offers an administrator to write a programmer's + note there: the administration's page writes it, as do the pages of a + selection, a genre, a network, a year and a decade. +- Radarr's search leaves out the films it could not add, those without a + TMDB id that were not entered by hand. A `tvdb:` term from Sonarr resolves + every id this server gives, those made of a TMDB id or a Fankai one too. +- An administrator's own page among the accounts no longer offers to reset + their password, which the server refuses there, and points to their account + page instead. Closing every session of their own there asks first, since it + ends this one too. + +### Fixed + +- A work entered by hand reaches Sonarr and Radarr. It is given a number of + its own, from 1,000,000,000 up, worked out from its catalogue id so that it + never changes: Sonarr's searches list it under that number and its lookups + answer it, Radarr takes it as the film's `TmdbId`, and the change + notifications and the seeding find it by it. Fankai's numbers end at + 999,999,999. Plex was given such a work whole already: an agent that also + lists Plex's own provider lets Plex guess it by its title, which + [docs/plex.md](docs/plex.md) explains. +- Signing out, or changing one's password, shows a visitor at once: the bar + kept the account's *Admin* until the page was reloaded. So does closing + one's own sessions, or untying one's own account from the identity + provider, from the administration. A session that ends somewhere else — it + runs out, or is closed from another browser or by another administrator — + is noticed at the next request it is refused, or when who is signed in is + next asked: the interface then forgets the account and shows a visitor. ## [0.10.0] — 2026-10-25 diff --git a/docs/editing.md b/docs/editing.md index 13f84f0..848b062 100644 --- a/docs/editing.md +++ b/docs/editing.md @@ -450,6 +450,14 @@ edited. Its identifiers are what Sonarr and Radarr find it by — TheTVDB's for for a film — and where its links lead. An identifier that another entry already claims is refused with a `409`. +Without those identifiers it is found all the same. Sonarr and Radarr find it by its title +and keep it under a number this server makes of its id here, the same for good — see +[Works entered by hand](sonarr-radarr.md#works-entered-by-hand). Plex matches it by its title +and year, under the catalogue's own id; Plex's own provider, when the agent lists it beside +this one, can only guess at it by its title, and lays what it guessed over it — see +[A work entered by hand](plex.md#a-work-entered-by-hand). An identifier the work does have +elsewhere is worth giving it: Plex's provider is then given the work itself. + ## Duplicates Two records of one work turn up in a catalogue now and then: a film entered by hand before a diff --git a/docs/media.md b/docs/media.md index 6eae129..ab581ff 100644 --- a/docs/media.md +++ b/docs/media.md @@ -35,9 +35,10 @@ default: nothing is kept until it is asked for. pictures as they were fetched — JPEG and PNG, whole — at the copies' addresses when `AMS_PUBLIC_URL` is set, and at the providers' when it is not, since only this server's own pages can follow a path. This server's - own pages are given smaller copies instead: see below. A bucket's media - go through this server, or, with **Sent to the bucket's own address**, a - reader is redirected to a presigned address good for an hour. + own pages are given the same pictures, or smaller copies once those are + turned on: see below. A bucket's media go through this server, or, with + **Sent to the bucket's own address**, a reader is redirected to a + presigned address good for an hour. - **By hand.** From a work's page, a picture or a theme can be uploaded — read for what it is, never for what it is called; SVG is refused, since a document that can carry a script must not be served from here — and a @@ -66,10 +67,10 @@ default: nothing is kept until it is asked for. A poster kept is two to three thousand pixels tall and a background up to four thousand wide: what a client that keeps one picture wants, and ten -times what a card or a page draws. So this server's own pages are given -copies, made of each picture at the widths it is drawn at, once and twice -over for a dense screen, never wider than the picture and never wider than -1920 pixels: +times what a card or a page draws. So this server can give its own pages +WebP copies instead, made of each picture at the widths it is drawn at, +once and twice over for a dense screen, never wider than the picture and +never wider than 1920 pixels: | A picture drawn as | Widths | |---|---| @@ -80,6 +81,14 @@ over for a dense screen, never wider than the picture and never wider than | an episode's still | 320, 640 | | a photograph of the cast | 120, 240 | +They are off until somebody turns them on — **WebP copies for the pages** +on the Media page, the setting `media.webp` — and until then the pages are +given the pictures themselves, whole, with the AVIF copies over them where +the night made some (see [AVIF](#avif)): no WebP copy is made. A page +learns which with the site's other options (`webp` in +`GET /api/v1/auth/options`) — the catalogue's pages before they draw +anything — and asks for the right files from its first picture. + - **WebP.** Lossy, at quality 80, through libwebp's encoder — a poster's colours converted with its sharper method, so that thin lettering keeps its edges, and a transparency kept exact beside the colours; lossless @@ -92,13 +101,22 @@ over for a dense screen, never wider than the picture and never wider than the original. - **Made once, when first asked for.** A copy is named by its picture's hash and its width — `/media/-480.webp` — and kept beside the - picture in the store, in a directory of its own. The copy a page asks for - first is made as the picture is stored: a poster's 480, a background's - 1280. Any other is made the first time a page asks for it — a few hundred - milliseconds for a large background — and served from the store after. - A width no page asks for is refused, so a picture has a dozen copies at - the most. The thumbnails earlier versions made are still served; none is - made any more. + picture in the store, in a directory of its own. While the copies are + on, the copy a page asks for first is made as the picture is stored: a + poster's 480, a background's 1280. Any other is made the first time a + page asks for it — a few hundred milliseconds for a large background — + and served from the store after. A width no page asks for is refused, so + a picture has a dozen copies at the most. The thumbnails earlier versions + made are still served; none is made any more. +- **Turned on.** The pages ask for the copies at once, and the pictures + kept already have theirs made as they are asked for: a catalogue's first + pages after the switch are a little slower to fill. +- **Turned off.** The pages are given the pictures themselves again, and + no copy is made. A page opened before, which still asks for one, is + given the copy where it is kept, and sent to the picture where it is not, + for that time only. The copies made stay in the store, ready should they + be turned back on, until they are deleted: see + [Deleting the copies of a format](#deleting-the-copies-of-a-format). - **Read safely.** A picture is decoded in Rust, never by libwebp: a picture past 16,384 pixels a side or 100 million pixels is not decoded at all, nor one whose pixels would take more than 256 MB, and a decoder @@ -121,14 +139,15 @@ proportions before a byte of it is there. ### Preparing what was kept before -Pictures stored before this version have neither a placeholder nor their -first copy. Their copies are made all the same, as pages ask for them; the -task **Prepare the pictures** (`media.variants`, on **Tasks** and on the -Media page, which says how many are left) works out their placeholders and -makes their first copies, a few at a time. It runs only when somebody runs -it — never at start: a catalogue's pictures are hours of decoding on a small -machine. It can be stopped; each picture is noted as it is done, so the -next run takes up where the last one stopped. +Pictures stored before this version have no placeholder, and no first +copy: a page draws them without one, and — while the WebP copies are on — +has their copies made as it asks for them. The task **Prepare the +pictures** (`media.variants`, on **Tasks** and on the Media page, which +says how many are left) works out their placeholders and, while the WebP +copies are on, makes their first copies, a few at a time. It runs only +when somebody runs it — never at start: a catalogue's pictures are hours of +decoding on a small machine. It can be stopped; each picture is noted as it +is done, so the next run takes up where the last one stopped. ### AVIF @@ -144,6 +163,13 @@ background's 1920 takes about a second and a half at night. No browser is asked what it reads: the addresses are fixed, and every browser the interface supports reads both. +With the WebP copies off, a page offers the same AVIF copies over the +picture itself, by the same rule: where they reach as wide as the role the +picture is drawn in, a browser takes the AVIF of the width it needs rather +than the whole picture, and where they do not, the picture. An AVIF whose +file has gone is answered with the WebP of its width while those are on, +and with the picture while they are off. + Measured on fifty posters from TMDB, on one core of an Apple M-series processor, at quality 68 and rav1e's speed 6: @@ -158,9 +184,36 @@ feature; NASM assembles it on x86_64); a build from source has it with `cargo build --release --features avif-asm`. Making them takes about 170 MB of memory at its peak. +### Deleting the copies of a format + +Turning the WebP or the AVIF copies off on the Media page while some are +kept asks what becomes of them, and says how many there are and what they +weigh. **Cancel** changes nothing. **Turn off and keep them** turns them +off and keeps the copies: a WebP copy is ready should they be turned back +on, and an AVIF copy is still given to the pages, though no other is made. +**Turn off and delete them** turns them off and deletes every copy of that +format, to have the space back. With none kept, the switch simply turns +off. + +The deletion runs in the background, a few hundred copies at a time — each +file taken from the store, then its row — and the Media page says +**Deleting copies now** until it is over. Turned on again meanwhile, the +setting stops it where it is. The pictures themselves are never touched, +nor the squares Plex is given. + +Through the API, an administrator's: `DELETE /api/v1/media/variants/webp` +(or `avif`), once that format's setting is off. It answers at once, `202`, +with how many copies there were and what they weighed (`copies`, `bytes`), +and the store's status says `purging` while it runs +(`GET /api/v1/media/status`). It is refused with `409` while the setting +is still on, while the AVIF copies are being made, or while another +deletion runs, and with `503` when no store is configured. Of several +instances, the one asked may not have heard yet that the setting is off: +the Media page asks once more a second later. + The providers' terms still apply to what is kept: TMDB, TheTVDB and Fanart.tv each say what may be done with their pictures. The switches of the **Media** page — store, how a bucket's copies are served, the cast's -photographs, the themes, the AVIF copies — are settings: see [Media](settings.md#media) in -the settings reference. +photographs, the themes, the WebP copies, the AVIF copies — are settings: see +[Media](settings.md#media) in the settings reference. diff --git a/docs/plex.md b/docs/plex.md index 01cff67..ef6649d 100644 --- a/docs/plex.md +++ b/docs/plex.md @@ -129,7 +129,8 @@ were fetched, JPEG and PNG, never the WebP copies this server's own pages are gi A rating key names a work by the identifier it is best known by, so that it survives the work being fetched again, or a new database: a film by TMDB's id, then IMDb's; a series by TheTVDB's, then TMDB's, then Fankai's; either by the -catalogue's own id when it has none of those, as an entry made by hand may. +catalogue's own id when it has none of those, as an entry made by hand may (see +[A work entered by hand](#a-work-entered-by-hand)). ``` movie-tmdb-603 show-tvdb-81189 @@ -172,6 +173,36 @@ not see. A season is matched by its series and its number; an episode by its series and its season and number, or by the day it aired. +### A work entered by hand + +A work [entered by hand](editing.md#entering-a-work-by-hand) is matched by its +title and year like any work the catalogue holds and, given none of the ids a +[rating key](#rating-keys) is made of, keyed by the catalogue's own id: +`show-id-01a17789-…`, `movie-id-…`, and its seasons' and episodes' keys after +it. Plex Media Server takes them like any other — on 1.43.5, matched when the +library is scanned and through *Fix Match*, with its record, its seasons and +episodes and its pictures. Its `Guid` lists the ids it does have elsewhere, and +nothing when it has none. + +With nothing to map it by, Plex's own provider cannot know it. An agent that +lists *Plex Series* or *Plex Movie* beside this provider — Plex suggests it for +the cast — has Plex look such a work up in its own catalogue by title and year +alone, and take what it finds: in one test a series entered by hand came back as +a talk show of a title close to its own, whose cast, studio, country, poster and +background were laid over it — and its title too, with Plex's provider listed +first. The item is linked to the work guessed, and an app that follows the link +asks Plex Media Server for a work its library does not hold: its log then says +`Invalid library metadata ID <24 hexadecimal digits> passed.`, the id being +Plex's own for the guess, never one this server hands out. Nothing in this +server's answer stops the guess — Plex maps +another provider's work by IMDb's, TMDB's and TheTVDB's ids alone, and a work +entered by hand has none of them to give. + +So give the work the ids it has elsewhere when it has some, and Plex's provider +is given that very work; or keep Plex's own provider out of the agent of a +library that holds works entered by hand. Once it is out, a refresh of the item +in Plex puts this provider's record back. + ## What Plex is given - **Language.** `X-Plex-Language` (a header, else a query parameter), then the diff --git a/docs/settings.md b/docs/settings.md index 023f715..4b91156 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -630,6 +630,7 @@ On the **Media** page, and read only when a store is configured (`AMS_MEDIA_STOR | [`media.serve`](#mediaserve) | `proxy` or `redirect` | `proxy` | server | | [`media.people`](#mediapeople) | `true` or `false` | `true` | server | | [`media.audio`](#mediaaudio) | `true` or `false` | `true` | server | +| [`media.webp`](#mediawebp) | `true` or `false` | unset | server | | [`media.avif`](#mediaavif) | `true` or `false` | unset | server | ### `media.store` @@ -650,12 +651,20 @@ The cast's photographs too: one per person, many per work. The themes: a Fan-Kai's, a few megabytes each. +### `media.webp` + +Make the WebP copies the pages ask for, at the widths they ask for them — the first as a +picture is stored, any other the first time a page asks — and give the pages those rather +than the pictures themselves. Off until it is set, and the pages are given the originals, +whole; the copies made while it was on are kept until an administrator deletes them. See +[The copies the pages are given](media.md#the-copies-the-pages-are-given). + ### `media.avif` -Make an AVIF copy of every poster, at 480 and 960 pixels, and of every background, at 1280, -each night at 03:00 UTC — the task `media.avif`, also run by hand from **Tasks**. A third -lighter than the WebP the interface is given otherwise, and a second or two of a processor -each to make: off until it is set. See [AVIF](media.md#avif). +Make an AVIF copy of every poster, at 480 and 960 pixels, and of every background, at 1280 +and 1920, each night at 03:00 UTC — the task `media.avif`, also run by hand from **Tasks**. +A third lighter than a WebP copy of the same width, and a second or two of a processor each +to make: off until it is set. See [AVIF](media.md#avif). ## Cache diff --git a/docs/sonarr-radarr.md b/docs/sonarr-radarr.md index 2e85b38..9dec471 100644 --- a/docs/sonarr-radarr.md +++ b/docs/sonarr-radarr.md @@ -49,7 +49,8 @@ reading: the same series, with what v5 reads besides. use (`tvdbId` and `metadataId`). - **Who numbers it.** `metadataSource` is `tvdb`, or `tmdb` for a series TMDB numbers — one TheTVDB does not list, or one it stopped answering for (see - [Who numbers a series](#who-numbers-a-series)) — or `fankai`. + [Who numbers a series](#who-numbers-a-series)) — or `fankai`, or `local` for a + series [entered by hand](#works-entered-by-hand) that nothing elsewhere names. The language is three letters, as Sonarr writes it (`eng`, `fra`). The series is answered in it when the catalogue holds it so; otherwise in the caller's own @@ -133,6 +134,43 @@ Radarr is sent, on `/v1/movie/{id}`, `/v1/movie/imdb/{id}` and `/v1/movie/bulk`: `/v1/movie/changed` names the films changed here too — fetched again, a lock put on, changed or lifted — with or without TMDB. +### Works entered by hand + +A work [entered by hand](editing.md#entering-a-work-by-hand) is found by the ids +it was given elsewhere, as any other: TheTVDB's for a series — or a number made +of TMDB's or Fankai's, as for a series TheTVDB does not list — and TMDB's for a +film. One given none of them is handed an id of this server's own, and Sonarr +and Radarr find it by its title, add it and refresh it like any other work: + +- **The id** is a number from 1 000 000 000 to 2 147 483 647, made of the work's + id in this catalogue: the same at every answer, after a restart, on every + instance, whatever is edited on the work — and above every number TheTVDB's + series, TMDB's films and series, and this server's other ranges fill. Sonarr + keeps it as the series' TheTVDB id, Radarr as the film's TMDB id. Both refuse + an id of nought or below for a work they add, which is why it is not negative + as an episode's own is (see [Orders](#orders)). `tvdb:` or `tmdb:` and the + number find the work too. +- **Asked for by it**, on either edition of Sonarr's surface and on Radarr's + (`/v1/movie/{id}`, `/v1/movie/bulk`), the work is answered from the catalogue + alone, without asking a source, which none has: switched off or not, and + merged into another or not — then with the work it went into, as + [After a merge](#after-a-merge) says. A number whose work was deleted is a + `404` at once, as nothing will answer for it again; Sonarr and Radarr then take + it for deleted. +- Sonarr v5 is sent `metadataSource` `local`. A film entered by hand is named + under it among the films changed (`/v1/movie/changed`) once one of its fields + is edited, as any film is once a lock is put on it, so that Radarr's scheduled + refresh reads it again; with [Tell them of a change](#telling-them-of-a-change) + on, either is told under this number. +- Given an id elsewhere later — its TheTVDB entry, say — the work is handed that + one from then on, and still answered by the number to the client that added it + by it. + +Two works of one kind entered by hand could make one number: the one entered +first keeps it, and the other is left out of the searches rather than offered +under a number that would add the first. A work entered by hand now is given an +id here whose number no other holds. + ## Connecting your Sonarr and Radarr Open **Sonarr & Radarr** in the administration (`/admin/arr`; administrators @@ -194,7 +232,8 @@ is asked to read the work again at once, after: client has the answer already. The work is found there by the id it is served under — TheTVDB's, or the synthetic -id that a TMDB-only series or a Fan-Kai is handed, for Sonarr; TMDB's for Radarr — +id that a TMDB-only series or a Fan-Kai is handed, for Sonarr; TMDB's for Radarr; +the number a [work entered by hand](#works-entered-by-hand) is handed, for either — through `/api/v3/series?tvdbId=` or `/api/v3/movie?tmdbId=`. One command then asks for every work found, up to a hundred at a time: `RefreshSeries` with `seriesIds`, `RefreshMovie` with `movieIds` (`POST /api/v3/command`). @@ -242,6 +281,11 @@ imports what is missing the way the import page does: or one this server made for a TMDB-only series or a Fan-Kai — then by IMDb's; - from Radarr (`/api/v3/movie`): by TMDB's id, then by IMDb's. +A work held there under the number a [work entered by hand](#works-entered-by-hand) +is handed is looked for in this catalogue alone, since no source has it: *already +held here*, or *held there under no id to look it up by* once the catalogue no +longer holds it. + It starts by hand, from the panel *Fill the catalogue from their library* of the Sonarr & Radarr page or from the tasks on the **Jobs** page (`/admin/jobs`), and only while one of the two is connected. It imports one work after another, with half a second between two diff --git a/frontend/e2e/adult-posters.spec.ts b/frontend/e2e/adult-posters.spec.ts new file mode 100644 index 0000000..898a753 --- /dev/null +++ b/frontend/e2e/adult-posters.spec.ts @@ -0,0 +1,142 @@ +import { expect, test, type Page } from '@playwright/test' +import { scan } from './support/axe' + +/** + * The posters of works for adults, as an account chooses them on its own + * page: veiled until it asks, then shown — in every browser it signs in on, + * not only the one it asked in — and veiled again from the bar's reading + * menu, which holds the same choice. + * + * The administrator's own choice, which the desk alone turns: both projects + * run at once, and would turn the one account's choice under each other. + * What the account kept before is put back afterwards, whatever happened. + */ + +const USERNAME = process.env.AMS_E2E_USER +const PASSWORD = process.env.AMS_E2E_PASSWORD + +/** Where the account keeps the choice (`adultPosters.ts`). */ +const PREFERENCE = '/api/v1/account/preferences/reader.adultPosters' + +async function signIn(page: Page) { + await page.goto('/login') + await page.getByLabel(/username|identifiant/i).fill(USERNAME!) + await page.getByLabel(/password|mot de passe/i).fill(PASSWORD!) + await page.getByRole('button', { name: /sign in|se connecter/i }).click() + await page.waitForURL('**/admin') +} + +interface Work { + id: string + isAdult?: boolean + contentRating?: string | null + images?: { coverType: string }[] +} + +/** + * Whether the interface veils a work's poster for a reader who has not asked + * to see it, as `forAdults` decides: marked for adults, or rated eighteen or + * more, or in one of the few ratings said in letters. + */ +function forAdults(work: Work): boolean { + const rating = work.contentRating?.trim().toUpperCase() ?? '' + const age = Number(/\d+/.exec(rating)?.[0] ?? Number.NaN) + return work.isAdult === true || ['NC-17', 'X', 'XXX', 'AO', 'ADULTS ONLY'].includes(rating) || (age >= 18 && age <= 21) +} + +/** What the account keeps of the choice, if anything. */ +async function kept(page: Page): Promise { + const answer = await page.request.get('/api/v1/account/preferences') + return ((await answer.json()) as { values: Record }).values['reader.adultPosters'] +} + +/** The choice as the page sends it, a moment after it is made. */ +function sent(page: Page) { + return page.waitForResponse( + (response) => response.url().endsWith(PREFERENCE) && response.request().method() === 'PUT', + ) +} + +/** A work's plate: its poster, and the veil over it, which is a button. */ +function plate(page: Page) { + const header = page.locator('main article > header') + return { + poster: header.getByRole('img', { name: /^(poster for|affiche de) /i }), + veil: header.getByRole('button', { name: /for adults|pour adultes/i }), + } +} + +test.describe('the posters of works for adults', () => { + test.skip(!USERNAME || !PASSWORD, 'needs AMS_E2E_USER and AMS_E2E_PASSWORD') + + test('are shown once the account asks on its own page, in every browser, and veiled again from the menu', async ({ + page, + browser, + isMobile, + }) => { + test.skip(isMobile, 'the choice is the account’s: the desk alone turns it') + test.slow() + await signIn(page) + + const listed = await page.request.get('/api/v1/items?includeAdult=true&limit=100') + const { items } = (await listed.json()) as { items: Work[] } + const work = items.find((item) => forAdults(item) && item.images?.some((image) => image.coverType === 'poster')) + test.skip(!work, 'the catalogue holds no work for adults with a poster') + + const before = await kept(page) + try { + // Whatever the account chose before is set aside, and this browser has + // chosen nothing: the poster is veiled. + expect((await page.request.delete(PREFERENCE)).ok()).toBe(true) + await page.goto(`/work/${work!.id}`) + await expect(plate(page).poster).toBeVisible() + await expect(plate(page).veil).toBeVisible() + + // Asked for on the account's page, by the switch its words name. + await page.goto('/account') + const choice = page.getByRole('switch', { + name: /^(show the posters of works for adults|montrer les affiches des œuvres pour adultes)$/i, + }) + await expect(choice).toHaveAttribute('aria-checked', 'false') + await page.waitForLoadState('networkidle') + await scan(page, 'the account page') + const on = sent(page) + await choice.click() + await expect(choice).toHaveAttribute('aria-checked', 'true') + expect((await on).ok()).toBe(true) + expect(await kept(page)).toBe(true) + + await page.goto(`/work/${work!.id}`) + await expect(plate(page).poster).toBeVisible() + await expect(plate(page).veil).toHaveCount(0) + + // Another browser, which kept nothing of it: the account brings it. + const elsewhere = await browser.newContext() + try { + const other = await elsewhere.newPage() + await signIn(other) + await other.goto(`/work/${work!.id}`) + await expect(plate(other).poster).toBeVisible() + await expect(plate(other).veil).toHaveCount(0) + } finally { + await elsewhere.close() + } + + // The bar's reading menu holds the same choice: turned off there, the + // poster is veiled at once, and the account keeps that too. + await page.getByRole('button', { name: /^(language and reading|langue et lecture)/i }).click() + const menu = page.getByRole('switch', { name: /^(show adult posters|montrer les affiches pour adultes)$/i }) + await expect(menu).toHaveAttribute('aria-checked', 'true') + const off = sent(page) + await menu.click() + await expect(menu).toHaveAttribute('aria-checked', 'false') + await expect(plate(page).veil).toBeVisible() + expect((await off).ok()).toBe(true) + expect(await kept(page)).toBe(false) + } finally { + // The account's choice as it was found. + if (before === undefined) await page.request.delete(PREFERENCE) + else await page.request.put(PREFERENCE, { data: { value: before } }) + } + }) +}) diff --git a/frontend/e2e/editorial.serial.spec.ts b/frontend/e2e/editorial.serial.spec.ts index 8e75abe..045430c 100644 --- a/frontend/e2e/editorial.serial.spec.ts +++ b/frontend/e2e/editorial.serial.spec.ts @@ -5,7 +5,8 @@ import { expect, test, type Page } from '@playwright/test' * The programmer's note: written in place by whoever maintains the catalogue, * in English and in French; read by a visitor in theirs, the other standing * in until it is written; listed in the administration, written again and - * taken away there. + * taken away there. The front page offers none to write: a note on its work + * of the day is begun in the administration, and read under the plate. * * Serial: a note belongs to the catalogue, and every page of the genre reads * it — the other specs' genre pages among them. Each test takes its notes @@ -129,4 +130,36 @@ test.describe('the programmer’s note', () => { await page.getByRole('dialog').getByRole('button', { name: 'Take it away' }).click() await expect(page.getByText('The last year of the century.')).toHaveCount(0) }) + + test('is not offered on the front page, and one begun in the administration is read there', async ({ page }) => { + await page.addInitScript(() => localStorage.setItem('ams.lang', 'en')) + await signIn(page) + + // The work of the day, by the way to its page; who is looking is known by + // then — the bar offers the administration — and nothing under the plate + // offers a note to write. + await page.goto('/') + const open = page.getByRole('link', { name: 'See the details' }) + await expect(open).toBeVisible() + await expect(page.getByRole('banner').getByRole('link', { name: 'Admin', exact: true })).toBeVisible() + await expect(page.getByRole('button', { name: 'Write a programmer’s note' })).toHaveCount(0) + const id = (await open.getAttribute('href'))?.replace(/^\/work\//, '') + expect(id).toBeTruthy() + + try { + await page.goto('/admin/editorial') + await page.getByRole('button', { name: 'Write a note' }).click() + const form = page.locator('form').filter({ has: page.getByRole('button', { name: 'Save the note' }) }) + await form.getByLabel('About').selectOption('work') + await form.getByLabel('Which one').fill(id!) + await form.getByLabel('The note', { exact: true }).fill('Why this one, *today*.') + await form.getByRole('button', { name: 'Save the note' }).click() + await expect(page.getByText('Why this one, today.')).toBeVisible() + + await page.goto('/') + await expect(page.getByRole('complementary', { name: 'Programmer’s note' }).locator('em')).toHaveText('today') + } finally { + await page.request.delete(`/api/v1/editorial/work/${encodeURIComponent(id!)}/en`) + } + }) }) diff --git a/frontend/e2e/media.serial.spec.ts b/frontend/e2e/media.serial.spec.ts index 519a81d..979e98c 100644 --- a/frontend/e2e/media.serial.spec.ts +++ b/frontend/e2e/media.serial.spec.ts @@ -1,7 +1,7 @@ -import { deflateSync } from 'node:zlib' - import { expect, test, type Page } from '@playwright/test' +import { png } from './support/png' + /** * The media kept: a picture put on a work by hand is stored and served from * this server, a work's pictures are read from the copies kept, and a copy @@ -23,39 +23,6 @@ async function signIn(page: Page) { await page.waitForURL('**/admin') } -/** A red 600×900 PNG, made here: nothing to fetch. */ -function png(width: number, height: number): Buffer { - const crc = (buf: Buffer) => { - let c = ~0 - for (const byte of buf) { - c ^= byte - for (let k = 0; k < 8; k += 1) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1 - } - return (~c >>> 0) - } - const chunk = (type: string, data: Buffer) => { - const length = Buffer.alloc(4) - length.writeUInt32BE(data.length) - const body = Buffer.concat([Buffer.from(type, 'ascii'), data]) - const sum = Buffer.alloc(4) - sum.writeUInt32BE(crc(body)) - return Buffer.concat([length, body, sum]) - } - const header = Buffer.alloc(13) - header.writeUInt32BE(width, 0) - header.writeUInt32BE(height, 4) - header[8] = 8 - header[9] = 2 - const row = Buffer.concat([Buffer.from([0]), Buffer.alloc(width * 3, 0).map((_, i) => (i % 3 === 0 ? 180 : 40))]) - const raw = Buffer.concat(Array.from({ length: height }, () => row)) - return Buffer.concat([ - Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), - chunk('IHDR', header), - chunk('IDAT', deflateSync(raw)), - chunk('IEND', Buffer.alloc(0)), - ]) -} - test.describe('the media kept', () => { test.skip(!USERNAME || !PASSWORD, 'needs AMS_E2E_USER and AMS_E2E_PASSWORD') diff --git a/frontend/e2e/pages.spec.ts b/frontend/e2e/pages.spec.ts index 799c0dc..70945ad 100644 --- a/frontend/e2e/pages.spec.ts +++ b/frontend/e2e/pages.spec.ts @@ -542,8 +542,13 @@ test.describe('related works', () => { await page.goto(`/work/${related!.id}`) await expect(page.getByText(/^(related works|œuvres liées)$/i)).toBeVisible() + // Looked for on the shelf of related works alone: an anime and the manga + // it was drawn from often share a title, and the work's own seasons, + // higher on the page, carry it too. The shelf keeps the order the work + // gives, so the first card of that name on it is the first relation's. const first = related!.relations[0]! - const card = page.getByRole('link', { name: first.title.slice(0, 24) }).first() + const shelf = page.getByRole('list', { name: /^(related works|œuvres liées)$/i }) + const card = shelf.getByRole('link', { name: first.title.slice(0, 24) }).first() await expect(card).toBeVisible() if (first.workId) { await expect(card).toHaveAttribute('href', `/work/${first.workId}`) diff --git a/frontend/e2e/session-end.spec.ts b/frontend/e2e/session-end.spec.ts new file mode 100644 index 0000000..74f79b3 --- /dev/null +++ b/frontend/e2e/session-end.spec.ts @@ -0,0 +1,179 @@ +import { expect, test as base, type APIRequestContext, type Locator, type Page } from '@playwright/test' + +/** + * A session that ends other than by the way out, with the page told at once + * and never loaded again: an administrator closing every session of their + * own from their own page, the password they cannot give themselves there, + * and a session closed from somewhere else, heard of at the next thing the + * server refuses (`sessionCaches`). + * + * Each test ends the sessions of an administrator of its own, opened for it + * and deleted after it whatever happens — in the teardown of the fixture + * that opened it, which runs even when the test runs out of time. Never the + * shared administrator's: the other specs are signed in with it meanwhile. + * That one signs in here only in a request context of its own, to open, + * delete and close from outside, and signs out of that alone. + */ + +const USERNAME = process.env.AMS_E2E_USER +const PASSWORD = process.env.AMS_E2E_PASSWORD + +interface Account { + id: string + username: string + password: string +} + +const test = base.extend<{ administrator: APIRequestContext; account: Account }>({ + administrator: async ({ request }, use) => { + const signedIn = await request.post('/api/v1/auth/login', { data: { username: USERNAME, password: PASSWORD } }) + expect(signedIn.ok()).toBe(true) + await use(request) + await request.post('/api/v1/auth/logout') + }, + account: async ({ administrator }, use, info) => { + const username = `e2e-end-${info.project.name}-${Date.now().toString(36)}` + const created = await administrator.post('/api/v1/users', { data: { username, role: 'admin' } }) + expect(created.status()).toBe(201) + const { user, password } = (await created.json()) as { user: { id: string }; password: string } + await use({ id: user.id, username, password }) + expect((await administrator.delete(`/api/v1/users/${user.id}`)).status()).toBe(204) + }, +}) + +/** The account, signed in in this page's browser, which shares its cookies with `page.request`. */ +async function signIn(page: Page, account: Account) { + const signedIn = await page.request.post('/api/v1/auth/login', { + data: { username: account.username, password: account.password }, + }) + expect(signedIn.ok()).toBe(true) +} + +/** Mark the page as it is now: one loaded again has lost the mark. */ +async function mark(page: Page) { + await page.evaluate(() => { + ;(window as Window & { sameVisit?: boolean }).sameVisit = true + }) +} + +function sameVisit(page: Page) { + return page.evaluate(() => (window as Window & { sameVisit?: boolean }).sameVisit === true) +} + +/** + * Where the catalogue keeps the way in and the way out: its bar, from `md` + * up; on a phone, the sheet along the foot, opened for the purpose. + */ +async function ways(page: Page, isMobile: boolean): Promise { + if (!isMobile) return page.getByRole('banner') + await page.getByRole('button', { name: /^(more|plus)$/i }).click() + return page.getByRole('dialog', { name: /^(more|plus)$/i }) +} + +const ADMIN = /^(admin|administration)$/i +const WAY_IN = /^(sign in|se connecter)$/i +const WAY_OUT = /^(sign out|se déconnecter)$/i + +test.describe('a session ended other than by the way out', () => { + test.skip(!USERNAME || !PASSWORD, 'needs AMS_E2E_USER and AMS_E2E_PASSWORD') + // Two sign-ins and an account opened before each, and several pages after. + test.slow() + + test('one’s own page gives no new password, and leads to the account page that changes it', async ({ + page, + account, + }) => { + await signIn(page, account) + // The server refuses it: a session left open must not be enough to lock + // its owner out. The page offered it all the same. + const reset = await page.request.post(`/api/v1/users/${account.id}/password`, { data: {} }) + expect(reset.status()).toBe(409) + + await page.goto(`/admin/users/${account.id}`) + await expect(page.getByRole('heading', { level: 1, name: account.username })).toBeVisible() + await expect(page.getByRole('button', { name: /^(set the password|définir le mot de passe)$/i })).toHaveCount(0) + const password = page.getByRole('region', { name: /^(password|mot de passe)$/i }) + await expect(password).toContainText(/account page|page de compte/i) + + await password.getByRole('link', { name: /^(my account|mon compte)$/i }).click() + await page.waitForURL('**/admin/account') + await expect(page.getByLabel(/^(current password|mot de passe actuel)$/i)).toBeVisible() + }) + + test('closing every session of one’s own, from one’s own page, signs out here at once', async ({ + page, + account, + isMobile, + }) => { + await signIn(page, account) + await page.goto(`/admin/users/${account.id}`) + await expect(page.getByRole('heading', { level: 1, name: account.username })).toBeVisible() + await mark(page) + + // The catalogue a page further on, to come forward to once signed out — + // by the administration's own way back, which its side bar holds from + // `lg` up. + const bar = page.getByRole('banner') + if (!isMobile) { + await page.getByRole('link', { name: /^(back to the catalogue|retour au catalogue)$/i }).click() + await page.waitForURL((url) => url.pathname === '/') + await expect(bar.getByRole('link', { name: ADMIN })).toBeVisible() + await page.goBack() + await page.waitForURL(`**/admin/users/${account.id}`) + } + + // Asked first, saying this one goes with the rest. + await page.getByRole('button', { name: /^(close every session|fermer toutes les sessions)$/i }).click() + const asked = page.getByRole('dialog', { name: /^(close every session of yours\?|fermer toutes vos sessions \?)$/i }) + await expect(asked).toContainText(/this one included|celle-ci comprise/i) + await asked.getByRole('button', { name: /^(close them all|toutes les fermer)$/i }).click() + + await page.waitForURL('**/login') + expect(await sameVisit(page)).toBe(true) + // Closed on the server too, this one with the rest. + expect((await page.request.get('/api/v1/account')).status()).toBe(401) + + if (!isMobile) { + await page.goForward() + await page.waitForURL((url) => url.pathname === '/') + await expect(bar.getByRole('link', { name: WAY_IN })).toBeVisible() + await expect(bar.getByRole('link', { name: ADMIN })).toHaveCount(0) + await expect(bar.getByRole('button', { name: WAY_OUT })).toHaveCount(0) + expect(await sameVisit(page)).toBe(true) + } + }) + + test('a session closed from somewhere else is heard of at the next refusal, and the bar shows a visitor', async ({ + page, + administrator, + account, + isMobile, + }) => { + await signIn(page, account) + await page.goto('/') + const before = await ways(page, isMobile) + const admin = before.getByRole('link', { name: ADMIN }) + await expect(admin).toBeVisible() + await expect(before.getByRole('button', { name: WAY_OUT })).toBeVisible() + await mark(page) + + // Closed by an administrator from another browser: nothing tells this page. + expect((await administrator.delete(`/api/v1/users/${account.id}/sessions`)).status()).toBe(204) + + // The administration, from the catalogue: what it asks first is refused, + // who is looking is asked again, and the answer is nobody. The bar's + // answer, five minutes fresh, used to keep the page in the administration + // with every panel refused. + await admin.click() + await page.waitForURL('**/login') + expect(await sameVisit(page)).toBe(true) + + await page.goBack() + await page.waitForURL((url) => url.pathname === '/') + const after = await ways(page, isMobile) + await expect(after.getByRole('link', { name: WAY_IN })).toBeVisible() + await expect(after.getByRole('link', { name: ADMIN })).toHaveCount(0) + await expect(after.getByRole('button', { name: WAY_OUT })).toHaveCount(0) + expect(await sameVisit(page)).toBe(true) + }) +}) diff --git a/frontend/e2e/sign-out.spec.ts b/frontend/e2e/sign-out.spec.ts new file mode 100644 index 0000000..c7243fd --- /dev/null +++ b/frontend/e2e/sign-out.spec.ts @@ -0,0 +1,82 @@ +import { expect, test, type Page } from '@playwright/test' + +/** + * Signing out of the catalogue's bar, and back in, with the page never loaded + * again: the bar says who is here at once — a visitor's way in the moment the + * session is over, the administrator's way to the administration once they + * are back. + * + * The cache used to be emptied on the way out without a word to whoever read + * it, and the bar, which is told who is looking from above every page, kept + * "Admin" and the way out until a reload (`endSession`). + * + * Signing out ends this page's own session and no other: every spec signs in + * on its own. + */ + +const USERNAME = process.env.AMS_E2E_USER +const PASSWORD = process.env.AMS_E2E_PASSWORD + +/** Through the sign-in page the browser is on, to the administration. */ +async function signIn(page: Page) { + await page.getByLabel(/username|identifiant/i).fill(USERNAME!) + await page.getByLabel(/password|mot de passe/i).fill(PASSWORD!) + await page.getByRole('button', { name: /sign in|se connecter/i }).click() + await page.waitForURL('**/admin') +} + +/** Whether this is still the page that was marked: one loaded again has lost the mark. */ +function sameVisit(page: Page) { + return page.evaluate(() => (window as Window & { sameVisit?: boolean }).sameVisit === true) +} + +test.describe('signing out', () => { + test.skip(!USERNAME || !PASSWORD, 'needs AMS_E2E_USER and AMS_E2E_PASSWORD') + + test('the bar offers a visitor’s way in at once, and the administration again once signed back in', async ({ + page, + isMobile, + }) => { + test.skip(isMobile, 'the bar holds the way out from `md` up; on a phone it is in the sheet along the foot') + await page.goto('/login') + await signIn(page) + + await page.goto('/') + const bar = page.getByRole('banner') + const admin = bar.getByRole('link', { name: /^(admin|administration)$/i }) + const wayIn = bar.getByRole('link', { name: /^(sign in|se connecter)$/i }) + const wayOut = bar.getByRole('button', { name: /^(sign out|se déconnecter)$/i }) + await expect(admin).toBeVisible() + await expect(wayOut).toBeVisible() + await page.evaluate(() => { + ;(window as Window & { sameVisit?: boolean }).sameVisit = true + }) + + // A page further on by a tab, so that there is one to come back to: the + // sign-in page's own ways back to the catalogue load the page anew. + await bar.getByRole('link', { name: /^(schedule|programme)$/i }).click() + await page.waitForURL('**/calendar') + + await wayOut.click() + await page.waitForURL('**/login') + await expect(page.getByRole('heading', { level: 1, name: /^(sign in|connexion)$/i })).toBeVisible() + await page.goBack() + await page.waitForURL((url) => url.pathname === '/') + await expect(wayIn).toBeVisible() + await expect(admin).toHaveCount(0) + await expect(wayOut).toHaveCount(0) + expect(await sameVisit(page)).toBe(true) + + // In again from the bar, and back to the catalogue by the administration's + // own way: the administrator's bar, not the visitor's it was just now. + await wayIn.click() + await page.waitForURL('**/login') + await signIn(page) + await page.getByRole('link', { name: /^(back to the catalogue|retour au catalogue)$/i }).click() + await page.waitForURL((url) => url.pathname === '/') + await expect(admin).toBeVisible() + await expect(wayOut).toBeVisible() + await expect(wayIn).toHaveCount(0) + expect(await sameVisit(page)).toBe(true) + }) +}) diff --git a/frontend/e2e/support/png.ts b/frontend/e2e/support/png.ts new file mode 100644 index 0000000..f7f3704 --- /dev/null +++ b/frontend/e2e/support/png.ts @@ -0,0 +1,42 @@ +/** + * A picture made here, for a spec that uploads one: nothing to fetch. + * + * The store files a picture by its content, so a spec gives its picture a + * size of the run's own — a run that failed before taking its picture away + * would otherwise leave the next one sharing the file, and its copies. + */ + +import { deflateSync } from 'node:zlib' + +/** A red PNG of the size asked. */ +export function png(width: number, height: number): Buffer { + const crc = (buf: Buffer) => { + let c = ~0 + for (const byte of buf) { + c ^= byte + for (let k = 0; k < 8; k += 1) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1 + } + return (~c >>> 0) + } + const chunk = (type: string, data: Buffer) => { + const length = Buffer.alloc(4) + length.writeUInt32BE(data.length) + const body = Buffer.concat([Buffer.from(type, 'ascii'), data]) + const sum = Buffer.alloc(4) + sum.writeUInt32BE(crc(body)) + return Buffer.concat([length, body, sum]) + } + const header = Buffer.alloc(13) + header.writeUInt32BE(width, 0) + header.writeUInt32BE(height, 4) + header[8] = 8 + header[9] = 2 + const row = Buffer.concat([Buffer.from([0]), Buffer.alloc(width * 3, 0).map((_, i) => (i % 3 === 0 ? 180 : 40))]) + const raw = Buffer.concat(Array.from({ length: height }, () => row)) + return Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + chunk('IHDR', header), + chunk('IDAT', deflateSync(raw)), + chunk('IEND', Buffer.alloc(0)), + ]) +} diff --git a/frontend/e2e/webp.serial.spec.ts b/frontend/e2e/webp.serial.spec.ts new file mode 100644 index 0000000..97b917a --- /dev/null +++ b/frontend/e2e/webp.serial.spec.ts @@ -0,0 +1,207 @@ +import { expect, test, type APIRequestContext, type Page } from '@playwright/test' + +import { scan } from './support/axe' +import { png } from './support/png' +import { escaped, pattern } from './support/regex' + +/** + * The copies of the pictures kept that the pages are given: none until the + * WebP copies are turned on, the pictures themselves meanwhile; on, the + * pages ask for them; turned off while some are kept, the Media page asks + * what becomes of them, and keeps them or deletes them as it is told. + * + * Needs a store, as the media spec does: without one the test says so and + * skips. Server-wide, because the setting is the whole server's and a + * deletion takes every WebP copy kept. It puts the setting back as it found + * it, and takes its picture away. + */ + +const USERNAME = process.env.AMS_E2E_USER +const PASSWORD = process.env.AMS_E2E_PASSWORD + +test.describe.configure({ mode: 'serial' }) + +async function signIn(page: Page) { + await page.goto('/login') + await page.getByLabel(/username|identifiant/i).fill(USERNAME!) + await page.getByLabel(/password|mot de passe/i).fill(PASSWORD!) + await page.getByRole('button', { name: /sign in|se connecter/i }).click() + await page.waitForURL('**/admin') +} + +interface Status { + backend: string + webp: boolean + purging: boolean + variants: { webp: number } +} + +async function status(request: APIRequestContext): Promise { + const answer = await request.get('/api/v1/media/status') + expect(answer.ok()).toBe(true) + return (await answer.json()) as Status +} + +/** The server's own value of a setting, or nothing where it is not set. */ +async function settingNow(request: APIRequestContext, key: string): Promise { + const answer = await request.get('/api/v1/settings/server/-') + expect(answer.ok()).toBe(true) + return ((await answer.json()) as { key: string; value: string }[]).find((setting) => setting.key === key)?.value +} + +/** A setting given a value, or — `null` — not set at all, as it starts. */ +async function setting(request: APIRequestContext, key: string, value: string | null) { + const answer = await request.put('/api/v1/settings/server/-', { data: { key, value } }) + expect(answer.ok(), `${key}=${value}`).toBe(true) +} + +/** A film whose poster nobody chose or added by hand: one added now leads it. */ +async function plainFilm(request: APIRequestContext): Promise<{ id: string; title: string } | undefined> { + const list = (await (await request.get('/api/v1/items?kind=movie&limit=40')).json()) as { items: { id: string }[] } + for (const { id } of list.items) { + const work = (await (await request.get(`/api/v1/items/${id}`)).json()) as { + title: string + primaryImages?: { poster?: string } + images?: { coverType: string; isManual: boolean }[] + } + if (work.primaryImages?.poster) continue + if ((work.images ?? []).some((image) => image.coverType === 'poster' && image.isManual)) continue + return { id, title: work.title } + } + return undefined +} + +test.describe('the copies the pages are given', () => { + test.skip(!USERNAME || !PASSWORD, 'needs AMS_E2E_USER and AMS_E2E_PASSWORD') + + test('are the pictures themselves until the WebP copies are on, and are kept or deleted when they go off', async ({ + page, + browser, + }) => { + test.slow() + await signIn(page) + test.skip((await status(page.request)).backend === 'off', 'no media store is configured on this server') + + const film = await plainFilm(page.request) + test.skip(!film, 'no film whose poster nobody chose') + const { id } = film! + + const was = await settingNow(page.request, 'media.webp') + let assetId: string | undefined + try { + await setting(page.request, 'media.webp', 'false') + + // A poster of this run's own, put on the film by hand: it leads it. + const uploaded = await page.request.post(`/api/v1/items/${id}/media`, { + multipart: { + file: { name: 'poster.png', mimeType: 'image/png', buffer: png(600, 900 + (Date.now() % 89)) }, + coverType: 'poster', + }, + }) + expect(uploaded.status()).toBe(201) + const sent = (await uploaded.json()) as { assetId: string; url: string } + assetId = sent.assetId + const { url } = sent + expect(url).toMatch(/^\/media\/[0-9a-f]{64}\.png$/) + const stem = url.replace(/\.png$/, '') + const copy = pattern(`${escaped(stem)}-\\d+\\.webp$`) + + // Off: a visitor's page draws the picture itself, at no other width. + const before = await browser.newContext() + try { + const them = await before.newPage() + const asked: string[] = [] + them.on('request', (request) => { + if (copy.test(new URL(request.url()).pathname)) asked.push(request.url()) + }) + await them.goto(`/work/${id}`) + const poster = them.locator(`img[src^="${stem}"]`).first() + await expect(poster).toHaveAttribute('src', url) + expect(await poster.getAttribute('srcset')).toBeNull() + await expect.poll(() => poster.evaluate((img) => (img as HTMLImageElement).naturalWidth)).toBe(600) + expect(asked, 'no WebP copy is asked for while they are off').toEqual([]) + } finally { + await before.close() + } + + // On: a page opened now asks for the copies, from its first picture — + // never for the picture itself — and each is made as it is asked for. + await setting(page.request, 'media.webp', 'true') + const after = await browser.newContext() + try { + const them = await after.newPage() + const whole: string[] = [] + them.on('request', (request) => { + if (new URL(request.url()).pathname === url) whole.push(request.url()) + }) + const made = them.waitForResponse((response) => copy.test(new URL(response.url()).pathname)) + await them.goto(`/work/${id}`) + const answer = await made + expect(answer.status()).toBe(200) + expect(answer.headers()['content-type']).toBe('image/webp') + const poster = them.locator(`img[src^="${stem}"]`).first() + await expect(poster).toHaveAttribute('src', `${stem}-480.webp`) + await expect(poster).toHaveAttribute('srcset', pattern(`${escaped(stem)}-480\\.webp 480w`)) + expect(whole, 'the picture itself is not asked for').toEqual([]) + } finally { + await after.close() + } + await expect.poll(async () => (await status(page.request)).variants.webp).toBeGreaterThan(0) + + // Off again from the Media page, with copies kept: it asks first. + await page.goto('/admin/media') + const webp = page.getByRole('switch', { name: /^(webp copies for the pages|copies webp pour les pages)$/i }) + const dialog = page.getByRole('dialog', { name: /turn off the webp copies|désactiver les copies webp/i }) + await expect(webp).toBeChecked() + await webp.click() + await expect(dialog).toBeVisible() + await expect(dialog).toContainText(/webp cop(y|ies) (is|are) kept|copies? webp (est|sont) stockées?/i) + await scan(page, 'the Media page, asking what becomes of the WebP copies') + + // Cancelled: nothing changes. + await dialog.getByRole('button', { name: /^(cancel|annuler)$/i }).click() + await expect(dialog).toBeHidden() + await expect(webp).toBeChecked() + expect((await status(page.request)).webp).toBe(true) + + // Turned off, the copies kept. + const kept = (await status(page.request)).variants.webp + await webp.click() + await dialog.getByRole('button', { name: /turn off and keep|désactiver et (la|les) garder/i }).click() + await expect(dialog).toBeHidden() + await expect(webp).not.toBeChecked() + const off = await status(page.request) + expect(off.webp).toBe(false) + expect(off.variants.webp).toBe(kept) + + // On again — no question asked — then off, the copies deleted: every + // one of them, in the background, the picture itself kept. + await webp.click() + await expect(webp).toBeChecked() + expect((await status(page.request)).webp).toBe(true) + await webp.click() + await expect(dialog).toBeVisible() + await dialog.getByRole('button', { name: /turn off and delete|désactiver et (la|les) supprimer/i }).click() + await expect(dialog).toBeHidden() + await expect(page.getByRole('status').filter({ hasText: /^(deleting|suppression de) .*webp/i })).toBeVisible() + await expect(webp).not.toBeChecked() + await expect + .poll( + async () => { + const now = await status(page.request) + return !now.purging && now.variants.webp === 0 + }, + { timeout: 30_000 }, + ) + .toBe(true) + expect((await page.request.get(url)).status()).toBe(200) + // Asked for anyway, a copy no longer kept is the picture, for now. + const gone = await page.request.get(`${stem}-480.webp`, { maxRedirects: 0 }) + expect(gone.status()).toBe(302) + expect(gone.headers()['location']).toMatch(pattern(`${escaped(url)}$`)) + } finally { + if (assetId) await page.request.delete(`/api/v1/items/${id}/media/${assetId}`) + await setting(page.request, 'media.webp', was ?? null) + } + }) +}) diff --git a/frontend/openapi.json b/frontend/openapi.json index cb3d9df..c79fba4 100644 --- a/frontend/openapi.json +++ b/frontend/openapi.json @@ -7196,6 +7196,50 @@ } } }, + "/api/v1/media/variants/{format}": { + "delete": { + "tags": [ + "Catalogue" + ], + "summary": "Delete every copy of a format: the WebP copies once `media.webp` is off,\nthe AVIF copies once `media.avif` is — the space they took, back.", + "description": "Answered at once, with what there was, and done in the background: each\ncopy's file taken from the store, then its row; the status says\n`purging` until it is over. The pictures themselves are not touched,\nnor the squares Plex is given. Turned on again meanwhile, the setting\nstops it where it is.", + "operationId": "media_delete_variants", + "parameters": [ + { + "name": "format", + "in": "path", + "description": "The format whose copies are deleted", + "required": true, + "schema": { + "$ref": "#/components/schemas/MediaVariantFormat" + } + } + ], + "responses": { + "202": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/VariantsDeleted" + } + } + } + }, + "400": { + "description": "Not a format whose copies are deleted: `webp` or `avif`" + }, + "403": { + "description": "The caller is not an administrator" + }, + "409": { + "description": "The format's copies are still made — its setting is on — or the AVIF copies are being made, or copies are being deleted already" + }, + "503": { + "description": "No media store is configured" + } + } + } + }, "/api/v1/media/works": { "get": { "tags": [ @@ -9725,12 +9769,13 @@ "Radarr compatibility" ], "summary": "A movie by TMDB id, in Radarr's format.", + "description": "The id may also be the one a film entered by hand that TMDB does not name\nis handed in TMDB's stead, from 1 000 000 000: such a film is answered\nfrom the catalogue alone.", "operationId": "radarr_by_tmdb", "parameters": [ { "name": "tmdb_id", "in": "path", - "description": "TMDB movie id", + "description": "TMDB movie id, or the one a film entered by hand is handed", "required": true, "schema": { "type": "integer", @@ -9885,7 +9930,7 @@ "Sonarr compatibility" ], "summary": "A series by TVDB id, in Skyhook's format.", - "description": "The id may also be one this server synthesised for a work TMDB indexes and\nTVDB does not; the response echoes back whichever id was asked for, because\nSonarr has already stored it.\n\nA series the catalogue holds is answered from the catalogue, as it was,\nwhen the sources do not answer. An id nobody holds is a 404 only when every\nsource that could resolve it answered that there is no such series; when\none did not answer — a timeout, a 5xx, a refused connection — it is a 502,\nnever a 404: Sonarr takes a series for deleted on a 404, and keeps what it\nhas on anything else.", + "description": "The id may also be one this server synthesised for a work TMDB indexes and\nTVDB does not, or for a series entered by hand that nothing elsewhere\nnames; the response echoes back whichever id was asked for, because Sonarr\nhas already stored it.\n\nA series the catalogue holds is answered from the catalogue, as it was,\nwhen the sources do not answer. An id nobody holds is a 404 only when every\nsource that could resolve it answered that there is no such series; when\none did not answer — a timeout, a 5xx, a refused connection — it is a 502,\nnever a 404: Sonarr takes a series for deleted on a 404, and keeps what it\nhas on anything else.", "operationId": "sonarr_show", "parameters": [ { @@ -11093,7 +11138,8 @@ "required": [ "site", "registration", - "passwordLogin" + "passwordLogin", + "webp" ], "properties": { "site": { @@ -11117,6 +11163,10 @@ "type": "null" } ] + }, + "webp": { + "type": "boolean", + "description": "Whether the pages are given WebP copies of the pictures this server\nkeeps (`media.webp`), rather than the pictures themselves: said here,\nwhere every page reads it before it draws one." } } }, @@ -17376,10 +17426,12 @@ "counts", "variants", "unprepared", + "webp", "avif", "fetching", "sweeping", - "preparing" + "preparing", + "purging" ], "properties": { "backend": { @@ -17415,6 +17467,10 @@ "format": "int64", "description": "The pictures kept that this version has not prepared — no\nplaceholder, no first copy — which the task `media.variants` does." }, + "webp": { + "type": "boolean", + "description": "Whether WebP copies are made and given to the pages (`media.webp`);\noff, the pages are given the pictures themselves." + }, "avif": { "type": "boolean", "description": "Whether the night's task makes AVIF copies (`media.avif`)." @@ -17429,9 +17485,21 @@ "preparing": { "type": "boolean", "description": "Whether the pictures are being prepared, or their AVIF copies made." + }, + "purging": { + "type": "boolean", + "description": "Whether every copy of a format is being deleted." } } }, + "MediaVariantFormat": { + "type": "string", + "description": "A format the copies of the pictures are written as, whose copies can be\ndeleted once they are no longer made.", + "enum": [ + "webp", + "avif" + ] + }, "MediumStatus": { "type": "string", "description": "What is kept of one address a work points at: where its asset stands,\nor that it has none.", @@ -22235,7 +22303,7 @@ }, "metadataSource": { "type": "string", - "description": "Who numbers the episodes: `tvdb`; `tmdb` for a series TMDB numbers —\none TheTVDB does not list, under an id this server made for it (see\n[`ids`]), or one TheTVDB stopped answering for; `fankai` for a\nFan-Kai." + "description": "Who numbers the episodes: `tvdb`; `tmdb` for a series TMDB numbers —\none TheTVDB does not list, under an id this server made for it (see\n[`ids`]), or one TheTVDB stopped answering for; `fankai` for a\nFan-Kai; `local` for a series entered by hand that nothing elsewhere\nnames." } } }, @@ -23593,6 +23661,30 @@ } } }, + "VariantsDeleted": { + "type": "object", + "description": "What there was of a format's copies as their deletion began.", + "required": [ + "format", + "copies", + "bytes" + ], + "properties": { + "format": { + "$ref": "#/components/schemas/MediaVariantFormat" + }, + "copies": { + "type": "integer", + "format": "int64", + "description": "How many copies there were." + }, + "bytes": { + "type": "integer", + "format": "int64", + "description": "What they weighed, in bytes." + } + } + }, "WantedWork": { "type": "object", "description": "One work wanted, by its id at a source.", diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index c5532da..4a36edd 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -5,9 +5,11 @@ import { BrowserRouter, Navigate, Route, Routes, useLocation, type NavigateOptio import { PageBoundary } from './components/PageBoundary' import { PublicShell } from './components/PublicShell' import { Spinner } from './components/ui' +import { useAdultPostersFromAccount } from './lib/adultPosters' import { ApiError, api } from './lib/api' import { useAuthOptions } from './lib/hooks' import { I18nProvider } from './lib/i18n' +import { sessionCaches } from './lib/session' import type { Me } from './lib/types' import { Browse } from './routes/Browse' import { Home } from './routes/Home' @@ -75,7 +77,10 @@ const Styleguide = ? named(() => import('./routes/Styleguide'), 'Styleguide') : null -const queryClient = new QueryClient({ +const queryClient: QueryClient = new QueryClient({ + // What is refused for want of a credential, read or written, may be the + // session ending somewhere else (`sessionCaches`). + ...sessionCaches(() => queryClient), defaultOptions: { queries: { staleTime: 30_000, @@ -128,73 +133,87 @@ function Router() { options.data?.site === 'private' && me.error instanceof ApiError && me.error.isUnauthorized return ( - }> - - : locked ? : }> - } /> - } /> - } /> - } /> - } /> - {/* One page for a person, by their record or by TMDB's id, as the - pages were addressed before there were records. */} - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - {/* Said, not redirected: see `NotFound`. */} - } /> - - - } /> - } /> - {Styleguide ? } /> : null} - - {/* The shell is the guard: it asks who is signed in once, and sends - anyone who may not write to the door before a screen mounts. */} - }> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - } /> - - - + <> + + }> + + : locked ? : }> + } /> + } /> + } /> + } /> + } /> + {/* One page for a person, by their record or by TMDB's id, as the + pages were addressed before there were records. */} + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + {/* Said, not redirected: see `NotFound`. */} + } /> + + + } /> + } /> + {Styleguide ? } /> : null} + + {/* The shell is the guard: it asks who is signed in once, and sends + anyone who may not write to the door before a screen mounts. */} + }> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + + + + ) } +/** + * The choice of the account signed in about the posters of works for adults, + * put where every card reads it, on either side of the site. A component of + * its own, drawing nothing: the router, holding the hook, would draw every + * page again each time any preference of the account changed. + */ +function AdultPostersFromAccount() { + useAdultPostersFromAccount() + return null +} + /** To the sign-in page, and back here once through it. */ function ToSignIn() { const location = useLocation() diff --git a/frontend/src/components/ProgrammersNote.tsx b/frontend/src/components/ProgrammersNote.tsx index c400ccf..b25811e 100644 --- a/frontend/src/components/ProgrammersNote.tsx +++ b/frontend/src/components/ProgrammersNote.tsx @@ -8,7 +8,9 @@ * last; the server says which, and the text carries that language for a * screen reader. Whoever may edit the catalogue writes it where it is read: * the note opens as a field in place, with what it will look like beneath it, - * and is saved in the language of the interface they are using. + * and is saved in the language of the interface they are using. The front + * page offers none to write where none is (`offer`): its work changes every + * day, and a note on it is begun in the administration's page of notes. */ import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query' @@ -83,11 +85,14 @@ export function ProgrammersNote({ scope, id, note: given, + offer = true, className, }: { scope: EditorialScope id: string note?: EditorialNote | null + /** Whether whoever may edit is offered a note to write where none is written. */ + offer?: boolean className?: string }) { const { t, lang, locale } = useI18n() @@ -117,8 +122,9 @@ export function ProgrammersNote({ } if (!shown) { - // A visitor is shown nothing; whoever may edit is offered a note to write. - return editor ? ( + // A visitor is shown nothing; whoever may edit is offered a note to + // write, where the page offers one. + return editor && offer ? (

{t.reader.adultHint}

diff --git a/frontend/src/components/SignOut.tsx b/frontend/src/components/SignOut.tsx index 55881b3..7c28d41 100644 --- a/frontend/src/components/SignOut.tsx +++ b/frontend/src/components/SignOut.tsx @@ -14,7 +14,7 @@ import { useNavigate } from 'react-router' import { api } from '../lib/api' import { cn } from '../lib/cn' import { useI18n } from '../lib/i18n' -import { forget } from '../lib/recent' +import { endSession } from '../lib/session' import { Glyph } from './ui' export function SignOut({ className, iconOnly }: { className?: string; iconOnly?: boolean }) { @@ -31,16 +31,15 @@ export function SignOut({ className, iconOnly }: { className?: string; iconOnly? onClick={async () => { // Whatever the server says, the session is over here: everything cached // was answered for somebody who is now gone, and so was the shelf of - // the works they opened. A server that cannot be reached is caught - // rather than left to reject — signing out still works, and an - // unhandled rejection would be the only trace of it. + // the works they opened (`endSession`). A server that cannot be + // reached is caught rather than left to reject — signing out still + // works, and an unhandled rejection would be the only trace of it. try { await api.post('/auth/logout') } catch { // Nothing to tell them: they are being signed out either way. } finally { - forget() - queryClient.clear() + endSession(queryClient) navigate('/login', { replace: true }) } }} diff --git a/frontend/src/components/account/ChangePassword.tsx b/frontend/src/components/account/ChangePassword.tsx index 05fc4e2..a042a91 100644 --- a/frontend/src/components/account/ChangePassword.tsx +++ b/frontend/src/components/account/ChangePassword.tsx @@ -12,7 +12,7 @@ import { useNavigate } from 'react-router' import { ApiError, api } from '../../lib/api' import { useI18n } from '../../lib/i18n' -import { forget } from '../../lib/recent' +import { endSession } from '../../lib/session' import { Button, FormField, Glyph, Input, Spinner } from '../ui' export function ChangePassword() { @@ -30,8 +30,7 @@ export function ChangePassword() { onSuccess: () => { // Signed out everywhere, this browser too: what it kept of the person // who was here goes with the rest. - forget() - queryClient.clear() + endSession(queryClient) navigate('/login', { replace: true }) }, }) diff --git a/frontend/src/components/media.tsx b/frontend/src/components/media.tsx index bd0493d..e7af0de 100644 --- a/frontend/src/components/media.tsx +++ b/frontend/src/components/media.tsx @@ -10,7 +10,7 @@ import { cn } from '../lib/cn' import * as fmt from '../lib/format' import { workQuery } from '../lib/hooks' import { useI18n } from '../lib/i18n' -import { hasSources, headlineRating, lookOf, poster, sized, type ImageRole, type Picture } from '../lib/media' +import { hasSources, headlineRating, lookOf, poster, sized, useWebpOffered, type ImageRole, type Picture } from '../lib/media' import { forAdults, unveil, useAdultPostersShown, useUnveiled } from '../lib/prefs' import { hasAlpha, thumbHashToDataURL } from '../lib/thumbhash' import { plainClick, useTransitionNavigate } from '../lib/transitions' @@ -172,12 +172,15 @@ export function placeholderStyle(picture: Picture | undefined, role: ImageRole): /** * An image, fetched at the size it is drawn at. * - * A copy this server keeps comes at the widths of its role, as WebP — under - * a `` offering AVIF first where the night made those — with its - * proportions given to the box and, until it arrives, its colour and its - * sketch under it (see `placeholderStyle`). What the server says of a - * picture comes with the work it is in, and is found by its address; a - * caller that holds it may pass it as `picture`. + * A picture this server keeps comes at the widths of its role, as WebP + * while the server makes those copies, and whole otherwise — either way + * under a `` offering AVIF first where the night made those — with + * its proportions given to the box and, until it arrives, its colour and + * its sketch under it (see `placeholderStyle`). Whether the WebP copies are + * on is the server's to say, and every picture is drawn again when it + * changes (see `useWebpOffered`). What the server says of a picture comes + * with the work it is in, and is found by its address; a caller that holds + * it may pass it as `picture`. * * Lazy unless told otherwise, because most of these are below the fold. A hero * backdrop is the exception and says so with `eager` — with `fetchPriority` @@ -214,7 +217,8 @@ export function Artwork({ picture?: Picture } & Omit, 'src' | 'srcSet' | 'sizes' | 'alt'>) { const look = picture ?? lookOf(url) - const image = sized(url, role, look) + const webp = useWebpOffered() + const image = sized(url, role, look, webp) // Each remembered by address, so a card that moves on to another work // starts afresh rather than wearing the last one's failure, or its load. const [retried, setRetried] = useState() @@ -225,7 +229,11 @@ export function Artwork({ return <>{fallback} } - const original = retried === url || image.src === image.original + // The original alone: once a copy has failed, and where nothing else is + // offered — an address with no copies, or a picture kept whose WebP + // copies are off and none of whose AVIF copies reaches as wide. + const original = retried === url || (image.src === image.original && !image.avif) + const src = original ? image.original : image.src const under = loaded === url ? undefined : placeholderStyle(look, role) const img = ( {alt} { diff --git a/frontend/src/lib/adultPosters.test.ts b/frontend/src/lib/adultPosters.test.ts new file mode 100644 index 0000000..448f8a4 --- /dev/null +++ b/frontend/src/lib/adultPosters.test.ts @@ -0,0 +1,22 @@ +import { describe, expect, it } from 'vitest' + +import { ADULT_POSTERS_PREFERENCE, readChoice } from './adultPosters' + +describe('the choice of the posters for adults, kept with the account', () => { + it('is kept under a name the server takes', () => { + // `clean_preference_key`, in src/api/native/account.rs: words of + // letters, digits and dashes joined by dots, the first starting with a + // small letter, 64 characters at most. A name it refuses would be a + // choice never kept, and sent again on every page. + expect(ADULT_POSTERS_PREFERENCE.length).toBeLessThanOrEqual(64) + expect(ADULT_POSTERS_PREFERENCE).toMatch(/^[a-z][A-Za-z\d-]*(?:\.[A-Za-z\d-]+)*$/) + }) + + it('reads as shown or veiled, and anything else as no choice at all', () => { + expect(readChoice(true)).toBe(true) + expect(readChoice(false)).toBe(false) + for (const raw of ['shown', 'true', 1, 0, null, undefined, {}, []]) { + expect(readChoice(raw), JSON.stringify(raw)).toBeUndefined() + } + }) +}) diff --git a/frontend/src/lib/adultPosters.ts b/frontend/src/lib/adultPosters.ts new file mode 100644 index 0000000..465bfd6 --- /dev/null +++ b/frontend/src/lib/adultPosters.ts @@ -0,0 +1,71 @@ +/** + * Whether the posters of works for adults are shown as they are, chosen by + * whoever is signed in — on their own page (`/account`) or in the bar's + * reading menu — and kept with their account, so the choice follows them + * from one browser to the next. A visitor's is kept in this browser + * (`prefs.ts`), and so is the choice of an account that never made one. + * + * Every card reads the store in `prefs.ts`, not the account: the account's + * choice is put there once, from the top of the page + * (`useAdultPostersFromAccount`), so a card costs no more than it did. + */ + +import { useCallback, useEffect } from 'react' + +import { useMe } from './hooks' +import { usePreference } from './preferences' +import { adoptAdultPosters, showAdultPosters, useAdultPostersShown } from './prefs' + +/** The account's preference, under a name the server takes (`clean_preference_key`). */ +export const ADULT_POSTERS_PREFERENCE = 'reader.adultPosters' + +/** What the account keeps, read as a choice: shown or veiled. Anything else is none. */ +export function readChoice(raw: unknown): boolean | undefined { + return typeof raw === 'boolean' ? raw : undefined +} + +/** + * Put the choice of the account signed in where every card reads it — and + * take it away when nobody is, or the account never chose, so that this + * browser's counts. Mounted once, above everything that shows a poster. + * + * Signing out, or changing one's password, takes the account's answers back + * (`endSession`) and says so to whoever reads them, this among them: nobody + * is signed in from then on, so the account's choice goes, and whoever signs + * in next is read afresh — their choice is the one put here, not the choice + * of the account that left. + */ +export function useAdultPostersFromAccount() { + const account = useMe().data?.user?.id + const [kept] = usePreference(ADULT_POSTERS_PREFERENCE, undefined, readChoice) + const chosen = account ? kept : undefined + + useEffect(() => { + adoptAdultPosters(chosen) + }, [chosen]) +} + +/** + * Whether the posters of works for adults are shown here, and the way to + * choose: for whoever is signed in, kept with their account — taking at + * once, and sent a moment after — and for a visitor, in this browser. + */ +export function useAdultPosters(): [boolean, (shown: boolean) => void] { + const shown = useAdultPostersShown() + const account = useMe().data?.user?.id + const [, keep] = usePreference(ADULT_POSTERS_PREFERENCE, undefined, readChoice) + + const choose = useCallback( + (next: boolean) => { + if (!account) { + showAdultPosters(next) + return + } + adoptAdultPosters(next) + keep(next) + }, + [account, keep], + ) + + return [shown, choose] +} diff --git a/frontend/src/lib/api.gen.ts b/frontend/src/lib/api.gen.ts index 44f924f..de83c70 100644 --- a/frontend/src/lib/api.gen.ts +++ b/frontend/src/lib/api.gen.ts @@ -2958,6 +2958,31 @@ export interface paths { patch?: never; trace?: never; }; + "/api/v1/media/variants/{format}": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Delete every copy of a format: the WebP copies once `media.webp` is off, + * the AVIF copies once `media.avif` is — the space they took, back. + * @description Answered at once, with what there was, and done in the background: each + * copy's file taken from the store, then its row; the status says + * `purging` until it is over. The pictures themselves are not touched, + * nor the squares Plex is given. Turned on again meanwhile, the setting + * stops it where it is. + */ + delete: operations["media_delete_variants"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/media/works": { parameters: { query?: never; @@ -3922,7 +3947,12 @@ export interface paths { path?: never; cookie?: never; }; - /** A movie by TMDB id, in Radarr's format. */ + /** + * A movie by TMDB id, in Radarr's format. + * @description The id may also be the one a film entered by hand that TMDB does not name + * is handed in TMDB's stead, from 1 000 000 000: such a film is answered + * from the catalogue alone. + */ get: operations["radarr_by_tmdb"]; put?: never; post?: never; @@ -4007,8 +4037,9 @@ export interface paths { /** * A series by TVDB id, in Skyhook's format. * @description The id may also be one this server synthesised for a work TMDB indexes and - * TVDB does not; the response echoes back whichever id was asked for, because - * Sonarr has already stored it. + * TVDB does not, or for a series entered by hand that nothing elsewhere + * names; the response echoes back whichever id was asked for, because Sonarr + * has already stored it. * * A series the catalogue holds is answered from the catalogue, as it was, * when the sources do not answer. An id nobody holds is a 404 only when every @@ -4494,6 +4525,12 @@ export interface components { /** @description Whether the password form is offered to everyone. */ passwordLogin: boolean; oidc?: components["schemas"]["OidcButton"] | null; + /** + * @description Whether the pages are given WebP copies of the pictures this server + * keeps (`media.webp`), rather than the pictures themselves: said here, + * where every page reads it before it draws one. + */ + webp: boolean; }; Authority: { subject: string; @@ -6906,6 +6943,11 @@ export interface components { * placeholder, no first copy — which the task `media.variants` does. */ unprepared: number; + /** + * @description Whether WebP copies are made and given to the pages (`media.webp`); + * off, the pages are given the pictures themselves. + */ + webp: boolean; /** @description Whether the night's task makes AVIF copies (`media.avif`). */ avif: boolean; /** @description Whether a fetch or a sweep is under way. */ @@ -6913,7 +6955,15 @@ export interface components { sweeping: boolean; /** @description Whether the pictures are being prepared, or their AVIF copies made. */ preparing: boolean; + /** @description Whether every copy of a format is being deleted. */ + purging: boolean; }; + /** + * @description A format the copies of the pictures are written as, whose copies can be + * deleted once they are no longer made. + * @enum {string} + */ + MediaVariantFormat: "webp" | "avif"; /** * @description What is kept of one address a work points at: where its asset stands, * or that it has none. @@ -8690,7 +8740,8 @@ export interface components { * @description Who numbers the episodes: `tvdb`; `tmdb` for a series TMDB numbers — * one TheTVDB does not list, under an id this server made for it (see * [`ids`]), or one TheTVDB stopped answering for; `fankai` for a - * Fan-Kai. + * Fan-Kai; `local` for a series entered by hand that nothing elsewhere + * names. */ metadataSource: string; }; @@ -9191,6 +9242,20 @@ export interface components { /** Format: int64 */ jpegBytes: number; }; + /** @description What there was of a format's copies as their deletion began. */ + VariantsDeleted: { + format: components["schemas"]["MediaVariantFormat"]; + /** + * Format: int64 + * @description How many copies there were. + */ + copies: number; + /** + * Format: int64 + * @description What they weighed, in bytes. + */ + bytes: number; + }; /** @description One work wanted, by its id at a source. */ WantedWork: { source: components["schemas"]["IdSource"]; @@ -16146,6 +16211,56 @@ export interface operations { }; }; }; + media_delete_variants: { + parameters: { + query?: never; + header?: never; + path: { + /** @description The format whose copies are deleted */ + format: components["schemas"]["MediaVariantFormat"]; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + 202: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["VariantsDeleted"]; + }; + }; + /** @description Not a format whose copies are deleted: `webp` or `avif` */ + 400: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The caller is not an administrator */ + 403: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The format's copies are still made — its setting is on — or the AVIF copies are being made, or copies are being deleted already */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description No media store is configured */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; media_by_work: { parameters: { query?: { @@ -18296,7 +18411,7 @@ export interface operations { query?: never; header?: never; path: { - /** @description TMDB movie id */ + /** @description TMDB movie id, or the one a film entered by hand is handed */ tmdb_id: number; }; cookie?: never; diff --git a/frontend/src/lib/hooks.ts b/frontend/src/lib/hooks.ts index c847a7b..dcd394a 100644 --- a/frontend/src/lib/hooks.ts +++ b/frontend/src/lib/hooks.ts @@ -12,6 +12,7 @@ import { NavigationType, useLocation, useNavigationType } from 'react-router' import { api, query, segment } from './api' import { useI18n } from './i18n' +import { offerWebp } from './media' import type { AuthOptions, Collections, CuratedLists, MediaItem, Me, Orders, SearchHits } from './types' /** @@ -116,11 +117,20 @@ export function useMe() { /** * Whether the site is open, and whether one may sign up: readable before * anybody signs in, which is when it is needed. + * + * It says too whether the pages are given WebP copies of the pictures kept, + * which every picture is told before the answer is (see `offerWebp`): the + * public pages wait for it, and ask for the right files from their first + * picture. */ export function useAuthOptions() { return useQuery({ queryKey: ['auth', 'options'], - queryFn: () => api.get('/auth/options'), + queryFn: async () => { + const options = await api.get('/auth/options') + offerWebp(options.webp) + return options + }, retry: false, staleTime: 5 * 60_000, }) diff --git a/frontend/src/lib/lang/en.ts b/frontend/src/lib/lang/en.ts index faf00f7..dd4efd0 100644 --- a/frontend/src/lib/lang/en.ts +++ b/frontend/src/lib/lang/en.ts @@ -843,6 +843,9 @@ export const en = { editor: 'Also corrects the catalogue: fields, imports, refreshes, lists.', admin: 'Also settles everything else: accounts, access, keys, settings.', } as Record, + reading: 'Reading', + adultPosters: 'Show the posters of works for adults', + adultPostersHint: 'Off, they stay blurred until pressed. The choice follows your account from one browser to the next.', password: 'Password', passwordNone: 'This account signs in through its identity provider and has no password here.', sessions: 'Sessions', @@ -961,7 +964,11 @@ export const en = { resetBody: 'Their sessions are closed. Leave the box empty to have one generated.', resetConfirm: 'Set the password', resetDone: 'Their new password, shown once:', + resetOwn: 'Your own password is changed on your account page, which asks for the current one first.', closeSessions: 'Close every session', + closeOwnTitle: 'Close every session of yours?', + closeOwnBody: 'Every session of yours is ended, this one included, so you will be asked to sign in again.', + closeOwnConfirm: 'Close them all', noSessions: 'Not signed in anywhere', noKeys: 'No key', since: (when: string) => `member since ${when}`, @@ -971,6 +978,7 @@ export const en = { unlink: 'Untie from the identity provider', unlinkTitle: 'Untie this account from the identity provider?', unlinkBody: 'It signs in through the provider again only once tied anew — on its next visit, if tying by e-mail is on. Without a password of its own it cannot sign in at all until given one.', + unlinkOwn: 'This is your account: its sessions close with the tie, this one included, so you will be asked to sign in again, with your password.', }, accessPage: 'Opening & APIs', oidc: { @@ -1704,9 +1712,30 @@ export const en = { peopleHint: 'One per person, and a work names dozens: the largest share of what is kept.', audio: 'The themes', audioHint: 'A Fan-Kai’s theme music, a few megabytes each.', + webp: 'WebP copies for the pages', + webpHint: + 'Each picture copied at the widths a page draws it, as it is stored or as a page first asks: a card’s copy weighs a tenth of the picture. Off, the pages are given the pictures themselves, whole.', avif: 'AVIF copies at night', avifHint: - 'An AVIF copy of every poster and background, made each night at 03:00 UTC: a third lighter than the WebP the pages are given otherwise, and a second or two of a processor each to make.', + 'An AVIF copy of every poster and background, made each night at 03:00 UTC and given to the pages wherever it exists: a third lighter than a WebP copy of the same width, and a second or two of a processor each to make.', + turnOff: { + title: (format: string) => `Turn off the ${format} copies`, + webp: (n: number, count: string, size: string) => + plural(n, { + one: `One WebP copy is kept, weighing ${size}. Delete it too? Kept, it is used again should the copies be turned back on; until then the pages are given the pictures themselves.`, + other: `${count} WebP copies are kept, weighing ${size}. Delete them too? Kept, they are used again should the copies be turned back on; until then the pages are given the pictures themselves.`, + }), + avif: (n: number, count: string, size: string) => + plural(n, { + one: `One AVIF copy is kept, weighing ${size}. Delete it too? Kept, it is still given to the pages, and no other is made.`, + other: `${count} AVIF copies are kept, weighing ${size}. Delete them too? Kept, they are still given to the pages, and no more are made.`, + }), + keep: (n: number) => plural(n, { one: 'Turn off and keep it', other: 'Turn off and keep them' }), + delete: (n: number) => plural(n, { one: 'Turn off and delete it', other: 'Turn off and delete them' }), + kept: (why: string) => `Turned off, and the copies are kept: ${why}`, + deleting: (n: number, count: string, format: string, size: string) => + plural(n, { one: `Deleting the ${format} copy (${size})…`, other: `Deleting ${count} ${format} copies (${size})…` }), + }, copies: 'Copies for the pages', copiesLine: (webp: string, avif: string, squares: string) => `WebP: ${webp} · AVIF: ${avif} · squares for Plex: ${squares}`, copiesOf: (n: string, size: string) => `${n} (${size})`, @@ -1715,9 +1744,15 @@ export const en = { one: 'One picture kept has no placeholder and no first copy yet. Prepare it now, or a page has its copies made as it asks for them.', other: `${n} pictures kept have no placeholder and no first copy yet. Prepare them now, or a page has its copies made as it asks for them.`, }), + unpreparedWhole: (n: number) => + plural(n, { + one: 'One picture kept has no placeholder yet. Prepare it now, or a page draws it with none until it arrives.', + other: `${n} pictures kept have no placeholder yet. Prepare them now, or a page draws them with none until they arrive.`, + }), allPrepared: 'Every picture kept is prepared.', prepareNow: 'Prepare the pictures', preparing: 'Preparing the pictures now', + purging: 'Deleting copies now', serve: 'How a bucket’s media are served', serveProxy: 'Through this server', serveRedirect: 'Sent to the bucket’s own address', @@ -3398,8 +3433,8 @@ export const en = { 'export.nfo': 'A .nfo document for every work, for a library Plex, Jellyfin or Kodi reads.', 'media.store': 'Every picture and theme the catalogue points at that is not kept yet, fetched now and kept, a few at a time. It can be stopped; the rest are fetched as they come up.', 'media.sweep': 'The media nobody points at any more — a work removed, a picture a provider dropped, an upload taken off — forgotten and deleted.', - 'media.variants': 'Every picture kept before this version — or before there were copies at all — prepared as one kept now is: the colour and the sketch a page draws while it loads worked out, and the copy its kind is first asked for made, a poster’s 480 pixels wide and a background’s 1280. A few at a time; it can be stopped, and the next run takes up where it stopped. Never run on its own: until it is, a page is given the copies it asks for, made as it asks.', - 'media.avif': 'An AVIF copy of every poster, 480 and 960 pixels wide, and of every background, 1280 — a third lighter than the WebP the interface is given otherwise, and a second or two of a processor each to make. At night, once “AVIF copies at night” is on, on the Media page. It can be stopped.', + 'media.variants': 'Every picture kept before this version — or before there were copies at all — prepared as one kept now is: the colour and the sketch a page draws while it loads worked out, and, while the WebP copies are on, the copy its kind is first asked for made, a poster’s 480 pixels wide and a background’s 1280. A few at a time; it can be stopped, and the next run takes up where it stopped. Never run on its own: until it is, a page draws those pictures with no placeholder.', + 'media.avif': 'An AVIF copy of every poster, 480 and 960 pixels wide, and of every background, 1280 and 1920 — a third lighter than a WebP copy of the same width, and a second or two of a processor each to make. At night, once “AVIF copies at night” is on, on the Media page. It can be stopped.', 'tls.renew': 'The certificate the clients’ door shows, looked at daily and issued anew by this server’s authority when fewer than thirty days remain — loaded without a restart.', 'update.check': 'GitHub asked for this project’s latest release, a minute after the start and then daily — one anonymous request, and only with the check turned on in the settings. A newer one is said on the dashboard, and told to the webhook once.', 'arr.seed': 'Every series your Sonarr holds and every film your Radarr holds, imported here when it is not yet, one after the other — before pointing them at this server, or again later for what came since. It can be stopped.', diff --git a/frontend/src/lib/lang/fr.ts b/frontend/src/lib/lang/fr.ts index c255a5a..cd6e1a8 100644 --- a/frontend/src/lib/lang/fr.ts +++ b/frontend/src/lib/lang/fr.ts @@ -832,6 +832,9 @@ export const fr: Dict = { editor: 'Corrige aussi le catalogue : champs, imports, rafraîchissements, listes.', admin: 'Règle aussi tout le reste : comptes, accès, clés, réglages.', }, + reading: 'Lecture', + adultPosters: 'Montrer les affiches des œuvres pour adultes', + adultPostersHint: 'Désactivé, elles restent floutées jusqu’à ce qu’on les touche. Le choix suit votre compte d’un navigateur à l’autre.', password: 'Mot de passe', passwordNone: 'Ce compte se connecte par son fournisseur d’identité et n’a pas de mot de passe ici.', sessions: 'Sessions', @@ -950,7 +953,11 @@ export const fr: Dict = { resetBody: 'Ses sessions sont fermées. Laissez vide pour en générer un.', resetConfirm: 'Définir le mot de passe', resetDone: 'Son nouveau mot de passe, montré une fois :', + resetOwn: 'Votre propre mot de passe se change sur votre page de compte, qui demande d’abord l’actuel.', closeSessions: 'Fermer toutes les sessions', + closeOwnTitle: 'Fermer toutes vos sessions ?', + closeOwnBody: 'Toutes vos sessions sont fermées, celle-ci comprise : il faudra vous reconnecter.', + closeOwnConfirm: 'Toutes les fermer', noSessions: 'Connecté nulle part', noKeys: 'Aucune clé', since: (when: string) => `membre depuis le ${when}`, @@ -960,6 +967,7 @@ export const fr: Dict = { unlink: 'Délier du fournisseur d’identité', unlinkTitle: 'Délier ce compte du fournisseur d’identité ?', unlinkBody: 'Il ne se reconnecte par le fournisseur qu’une fois lié à nouveau — à son prochain passage si la liaison par e-mail est active. Sans mot de passe à lui, il ne peut plus se connecter du tout avant d’en recevoir un.', + unlinkOwn: 'C’est votre compte : ses sessions se ferment avec le lien, celle-ci comprise. Il faudra vous reconnecter, avec votre mot de passe.', }, accessPage: 'Ouverture & API', oidc: { @@ -1695,9 +1703,30 @@ export const fr: Dict = { peopleHint: 'Une par personne, et une fiche en nomme des dizaines : la plus grosse part de ce qui est stocké.', audio: 'Les musiques', audioHint: 'La musique de générique d’un Fan-Kai, quelques mégaoctets chacune.', + webp: 'Copies WebP pour les pages', + webpHint: + 'Chaque image copiée aux largeurs où une page la dessine, à son stockage ou à la première demande d’une page\u202f: la copie d’une carte pèse le dixième de l’image. Désactivé, les pages reçoivent les images elles-mêmes, entières.', avif: 'Copies AVIF la nuit', avifHint: - 'Une copie AVIF de chaque affiche et de chaque fond, faite chaque nuit à 3 h UTC : un tiers plus légère que le WebP que les pages reçoivent sinon, et une seconde ou deux de processeur à faire chacune.', + 'Une copie AVIF de chaque affiche et de chaque fond, faite chaque nuit à 3 h UTC et donnée aux pages partout où elle existe\u202f: un tiers plus légère qu’une copie WebP de même largeur, et une seconde ou deux de processeur à faire chacune.', + turnOff: { + title: (format: string) => `Désactiver les copies ${format}`, + webp: (n: number, count: string, size: string) => + plural(n, { + one: `Une copie WebP est stockée, qui pèse ${size}. La supprimer aussi\u202f? Gardée, elle resservira si les copies sont réactivées\u202f; d’ici là, les pages reçoivent les images elles-mêmes.`, + other: `${count} copies WebP sont stockées, qui pèsent ${size}. Les supprimer aussi\u202f? Gardées, elles resserviront si les copies sont réactivées\u202f; d’ici là, les pages reçoivent les images elles-mêmes.`, + }), + avif: (n: number, count: string, size: string) => + plural(n, { + one: `Une copie AVIF est stockée, qui pèse ${size}. La supprimer aussi\u202f? Gardée, elle est toujours donnée aux pages, et aucune autre n’est faite.`, + other: `${count} copies AVIF sont stockées, qui pèsent ${size}. Les supprimer aussi\u202f? Gardées, elles sont toujours données aux pages, et aucune autre n’est faite.`, + }), + keep: (n: number) => plural(n, { one: 'Désactiver et la garder', other: 'Désactiver et les garder' }), + delete: (n: number) => plural(n, { one: 'Désactiver et la supprimer', other: 'Désactiver et les supprimer' }), + kept: (why: string) => `Désactivé, et les copies restent\u202f: ${why}`, + deleting: (n: number, count: string, format: string, size: string) => + plural(n, { one: `Suppression de la copie ${format} (${size})…`, other: `Suppression de ${count} copies ${format} (${size})…` }), + }, copies: 'Copies pour les pages', copiesLine: (webp: string, avif: string, squares: string) => `WebP : ${webp} · AVIF : ${avif} · carrés pour Plex : ${squares}`, copiesOf: (n: string, size: string) => `${n} (${size})`, @@ -1706,9 +1735,15 @@ export const fr: Dict = { one: 'Une image stockée n’a encore ni esquisse ni première copie. Préparez-la maintenant, ou une page fait faire ses copies à mesure qu’elle les demande.', other: `${n} images stockées n’ont encore ni esquisse ni première copie. Préparez-les maintenant, ou une page fait faire ses copies à mesure qu’elle les demande.`, }), + unpreparedWhole: (n: number) => + plural(n, { + one: 'Une image stockée n’a pas encore d’esquisse. Préparez-la maintenant, ou une page la dessine sans esquisse jusqu’à son arrivée.', + other: `${n} images stockées n’ont pas encore d’esquisse. Préparez-les maintenant, ou une page les dessine sans esquisse jusqu’à leur arrivée.`, + }), allPrepared: 'Toutes les images stockées sont préparées.', prepareNow: 'Préparer les images', preparing: 'Préparation des images en cours', + purging: 'Suppression des copies en cours', serve: 'Comment servir les médias d’un bucket', serveProxy: 'Par ce serveur', serveRedirect: 'Renvoyé vers l’adresse du bucket', @@ -3400,8 +3435,8 @@ export const fr: Dict = { 'export.nfo': 'Un document .nfo pour chaque œuvre, pour une bibliothèque que lisent Plex, Jellyfin ou Kodi.', 'media.store': 'Chaque image et chaque musique que le catalogue référence et qui n’est pas encore stockée, récupérée maintenant, quelques-unes à la fois. Peut être arrêtée ; le reste se récupère au fil de l’eau.', 'media.sweep': 'Les médias que plus rien ne référence — une fiche supprimée, une image abandonnée par sa source, un envoi retiré — oubliés et effacés.', - 'media.variants': 'Chaque image stockée avant cette version — ou avant qu’il y ait des copies — préparée comme l’est une image stockée aujourd’hui : la couleur et l’esquisse qu’une page affiche pendant le chargement calculées, et la copie que son type demande en premier faite, 480 pixels de large pour une affiche et 1280 pour un fond. Quelques-unes à la fois ; on peut l’arrêter, et le passage suivant reprend là où elle s’est arrêtée. Jamais lancée d’elle-même : d’ici là, une page reçoit les copies qu’elle demande, faites à la demande.', - 'media.avif': 'Une copie AVIF de chaque affiche, en 480 et 960 pixels de large, et de chaque fond, en 1280 — un tiers plus légère que le WebP que l’interface reçoit sinon, et une seconde ou deux de processeur à faire chacune. La nuit, une fois « Copies AVIF la nuit » activé sur la page Médias. Peut être arrêtée.', + 'media.variants': 'Chaque image stockée avant cette version — ou avant qu’il y ait des copies — préparée comme l’est une image stockée aujourd’hui\u202f: la couleur et l’esquisse qu’une page affiche pendant le chargement calculées, et, tant que les copies WebP sont activées, la copie que son type demande en premier faite, 480 pixels de large pour une affiche et 1280 pour un fond. Quelques-unes à la fois\u202f; on peut l’arrêter, et le passage suivant reprend là où elle s’est arrêtée. Jamais lancée d’elle-même\u202f: d’ici là, une page dessine ces images sans esquisse.', + 'media.avif': 'Une copie AVIF de chaque affiche, en 480 et 960 pixels de large, et de chaque fond, en 1280 et 1920 — un tiers plus légère qu’une copie WebP de même largeur, et une seconde ou deux de processeur à faire chacune. La nuit, une fois «\u202fCopies AVIF la nuit\u202f» activé sur la page Médias. Peut être arrêtée.', 'tls.renew': 'Le certificat que présente la porte des clients, examiné chaque jour et réémis par l’autorité de ce serveur quand il lui reste moins de trente jours — chargé sans redémarrage.', 'update.check': 'La dernière version de ce projet demandée à GitHub, une minute après le démarrage puis chaque jour — une requête anonyme, et seulement si la vérification est activée dans les réglages. Une version plus récente est signalée sur le tableau de bord, et annoncée une fois au webhook.', 'arr.seed': 'Chaque série de votre Sonarr et chaque film de votre Radarr, importés ici s’ils n’y sont pas encore, l’un après l’autre — avant de les tourner vers ce serveur, ou plus tard pour ce qui est arrivé depuis. Peut être arrêté.', diff --git a/frontend/src/lib/media.test.ts b/frontend/src/lib/media.test.ts index 4744769..f8573b7 100644 --- a/frontend/src/lib/media.test.ts +++ b/frontend/src/lib/media.test.ts @@ -11,12 +11,15 @@ import { latestEpisode, lookOf, nextEpisode, + offerWebp, pick, poster, ratingsOf, seasonName, seasonNumbers, sized, + type ImageRole, + type Picture, } from './media' describe('an image in a role', () => { @@ -44,35 +47,77 @@ describe('an image in a role', () => { const kept = `${stem}.png` const at = (widths: number[], format = 'webp') => widths.map((w) => `${stem}-${w}.${format} ${w}w`).join(', ') - it('is offered at its role’s widths, as WebP, the original kept to fall back on', () => { - const sourced = sized(kept, 'poster') - expect(sourced.src).toBe(`${stem}-480.webp`) - expect(sourced.srcSet).toBe(at([240, 480, 960])) - expect(sourced.avif).toBeUndefined() - expect(sourced.original).toBe(kept) - expect(sized(kept, 'backdrop').srcSet).toBe(at([640, 1280, 1920])) - expect(sized(kept, 'still').srcSet).toBe(at([320, 640])) - expect(sized(kept, 'headshot').srcSet).toBe(at([120, 240])) - }) + describe('with the WebP copies on', () => { + const webp = (role: ImageRole, look?: Picture) => sized(kept, role, look, true) - it('is never offered wider than 1920, the lightbox’s included', () => { - expect(sized(kept, 'full').srcSet).toBe(at([640, 1280, 1920])) - }) + it('is offered at its role’s widths, as WebP, the original kept to fall back on', () => { + const sourced = webp('poster') + expect(sourced.src).toBe(`${stem}-480.webp`) + expect(sourced.srcSet).toBe(at([240, 480, 960])) + expect(sourced.avif).toBeUndefined() + expect(sourced.original).toBe(kept) + expect(webp('backdrop').srcSet).toBe(at([640, 1280, 1920])) + expect(webp('still').srcSet).toBe(at([320, 640])) + expect(webp('headshot').srcSet).toBe(at([120, 240])) + }) + + it('is never offered wider than 1920, the lightbox’s included', () => { + expect(webp('full').srcSet).toBe(at([640, 1280, 1920])) + }) + + it('is never offered wider than it is, and offered at its own width', () => { + const sourced = webp('card', { width: 680, height: 1000 }) + expect(sourced.srcSet).toBe(at([240, 480, 680])) + expect([sourced.width, sourced.height]).toEqual([480, 706]) + expect(webp('card', { width: 200, height: 300 }).srcSet).toBe(at([200])) + }) - it('is never offered wider than it is, and offered at its own width', () => { - const sourced = sized(kept, 'card', { width: 680, height: 1000 }) - expect(sourced.srcSet).toBe(at([240, 480, 680])) - expect([sourced.width, sourced.height]).toEqual([480, 706]) - expect(sized(kept, 'card', { width: 200, height: 300 }).srcSet).toBe(at([200])) + it('offers its AVIF copies where they reach as wide as the WebP', () => { + const poster = { width: 2000, height: 3000, avif: [480, 960] } + expect(webp('card', poster).avif).toBe(at([480, 960], 'avif')) + expect(webp('thumb', poster).avif).toBe(at([480], 'avif')) + // A background's 1280 would draw a dense screen's 1920 soft. + expect(webp('backdrop', { width: 3840, height: 2160, avif: [1280] }).avif).toBeUndefined() + expect(webp('backdrop', { width: 1280, height: 720, avif: [1280] }).avif).toBe(at([1280], 'avif')) + }) }) - it('offers its AVIF copies where they reach as wide as the WebP', () => { - const poster = { width: 2000, height: 3000, avif: [480, 960] } - expect(sized(kept, 'card', poster).avif).toBe(at([480, 960], 'avif')) - expect(sized(kept, 'thumb', poster).avif).toBe(at([480], 'avif')) - // A background's 1280 would draw a dense screen's 1920 soft. - expect(sized(kept, 'backdrop', { width: 3840, height: 2160, avif: [1280] }).avif).toBeUndefined() - expect(sized(kept, 'backdrop', { width: 1280, height: 720, avif: [1280] }).avif).toBe(at([1280], 'avif')) + describe('with the WebP copies off', () => { + const whole = (role: ImageRole, look?: Picture) => sized(kept, role, look, false) + + it('is the picture itself, its own size giving the box its proportions', () => { + expect(whole('poster', { width: 2000, height: 3000 })).toEqual({ src: kept, width: 2000, height: 3000, original: kept }) + expect(whole('backdrop', { width: 3840, height: 2160 })).toEqual({ src: kept, width: 3840, height: 2160, original: kept }) + expect(whole('card')).toEqual({ src: kept, original: kept }) + }) + + it('is offered as AVIF over the picture where those copies reach as wide as its role is drawn', () => { + const poster = { width: 2000, height: 3000, avif: [480, 960] } + expect(whole('card', poster)).toEqual({ + src: kept, + sizes: sized(kept, 'card', poster, true).sizes, + avif: at([480, 960], 'avif'), + width: 2000, + height: 3000, + original: kept, + }) + expect(whole('thumb', poster).avif).toBe(at([480], 'avif')) + expect(whole('backdrop', { width: 1280, height: 720, avif: [1280] }).avif).toBe(at([1280], 'avif')) + // Not as wide: the picture alone, with no width to choose among. + const backdrop = whole('backdrop', { width: 3840, height: 2160, avif: [1280] }) + expect(backdrop).toEqual({ src: kept, width: 3840, height: 2160, original: kept }) + }) + + it('is what the server said last, where the caller does not say', () => { + expect(sized(kept, 'card').src).toBe(kept) + offerWebp(true) + try { + expect(sized(kept, 'card').srcSet).toBe(at([240, 480, 960])) + } finally { + offerWebp(false) + } + expect(sized(kept, 'card').srcSet).toBeUndefined() + }) }) it('is found by its address once a work has said what it looks like', () => { @@ -80,8 +125,9 @@ describe('an image in a role', () => { const item = work({ images: [image('p', { url: `${stem}.jpg` })], pictures: { [`${stem}.jpg`]: look } }) expect(poster(item)).toBe(`${stem}.jpg`) expect(lookOf(`${stem}.jpg`)).toEqual(look) - expect(sized(`${stem}.jpg`, 'card').srcSet).toBe(at([240, 480, 960])) - expect(sized(`${stem}.jpg`, 'poster').width).toBe(480) + expect(sized(`${stem}.jpg`, 'card', undefined, true).srcSet).toBe(at([240, 480, 960])) + expect(sized(`${stem}.jpg`, 'poster', undefined, true).width).toBe(480) + expect(sized(`${stem}.jpg`, 'poster', undefined, false).width).toBe(1000) }) }) diff --git a/frontend/src/lib/media.ts b/frontend/src/lib/media.ts index e0962ac..b6dd1d9 100644 --- a/frontend/src/lib/media.ts +++ b/frontend/src/lib/media.ts @@ -7,6 +7,8 @@ * three different ways in three components. */ +import { useSyncExternalStore } from 'react' + import * as fmt from './format' import type { Credit, Episode, Image, MediaItem, Rating } from './types' @@ -172,36 +174,86 @@ export function lookOf(url: string): Picture | undefined { return LOOKS.get(url) } +/** + * Whether the server gives the pages WebP copies of the pictures it keeps + * (`media.webp`), as it said last: its options say so, and are told here + * before they are answered (see `useAuthOptions`). Off until it says + * otherwise, as the setting is, and the pages are then given the pictures + * themselves. + */ +let webpOffered = false +const webpWatchers = new Set<() => void>() + +function watchWebp(watcher: () => void) { + webpWatchers.add(watcher) + return () => { + webpWatchers.delete(watcher) + } +} + +/** What the server says of its WebP copies, told to every picture drawn. */ +export function offerWebp(on: boolean) { + if (webpOffered === on) return + webpOffered = on + for (const watcher of webpWatchers) watcher() +} + +/** + * Whether the pictures kept are offered as WebP copies, for what draws one: + * drawn again when the server turns them on or off. + */ +export function useWebpOffered(): boolean { + return useSyncExternalStore(watchWebp, () => webpOffered, () => false) +} + /** * The right file for an image in a role. * - * A copy this server keeps is offered at the widths of its role's ladder, - * as WebP — and as AVIF where the night's copies reach as wide. TMDB gets - * its ladder. TheTVDB publishes a thumbnail beside each artwork — the same - * path with `_t` before the extension — which is used for the small roles, - * with the original kept as a fallback because an old upload is not - * guaranteed to have one. Anything else, including a URL somebody typed by - * hand, is used as it is. + * A picture this server keeps is offered at the widths of its role's + * ladder, as WebP while the server makes those copies (`webp`: as it said + * last, unless the caller says) — and otherwise as itself, whole, in its + * own proportions; either way as AVIF too where the night's copies reach + * as wide. TMDB gets its ladder. TheTVDB publishes a thumbnail beside each + * artwork — the same path with `_t` before the extension — which is used + * for the small roles, with the original kept as a fallback because an old + * upload is not guaranteed to have one. Anything else, including a URL + * somebody typed by hand, is used as it is. */ -export function sized(url: string, role: ImageRole, look: Picture | undefined = lookOf(url)): Sourced { +export function sized( + url: string, + role: ImageRole, + look: Picture | undefined = lookOf(url), + webp: boolean = webpOffered, +): Sourced { const kept = KEPT.exec(url) if (kept) { const stem = `${kept[1]}${kept[2]}` const widths = keptWidths(role, look?.width) - const srcSet = widths.map((width) => `${stem}-${width}.webp ${width}w`).join(', ') + // AVIF only where its copies reach as wide as the role is offered: a + // source the browser picks is the only one it looks at, and a narrower + // one would draw a dense screen's picture soft. + const largest = widths[widths.length - 1]! + const covered = (look?.avif ?? []).filter((width) => width <= largest) + const avif = covered.includes(largest) ? covered.map((width) => `${stem}-${width}.avif ${width}w`).join(', ') : undefined + + if (!webp) { + // The picture itself, its own size giving the box its proportions, + // under the AVIF copies where they reach as wide. + return { + src: url, + ...(avif ? { sizes: SIZES[role], avif } : {}), + ...(look?.width && look.height ? { width: look.width, height: look.height } : {}), + original: url, + } + } + // The middle of the ladder for a browser that reads no `srcset`. - const src = `${stem}-${widths[Math.min(1, widths.length - 1)]}.webp` const drawn = widths[Math.min(1, widths.length - 1)]! - // AVIF only where its copies reach as wide as the WebP ones: a source - // the browser picks is the only one it looks at, and a narrower one - // would draw a dense screen's picture soft. - const largest = widths[widths.length - 1]! - const avif = (look?.avif ?? []).filter((width) => width <= largest) return { - src, - srcSet, + src: `${stem}-${drawn}.webp`, + srcSet: widths.map((width) => `${stem}-${width}.webp ${width}w`).join(', '), sizes: SIZES[role], - avif: avif.includes(largest) ? avif.map((width) => `${stem}-${width}.avif ${width}w`).join(', ') : undefined, + avif, ...(look?.width && look.height ? { width: drawn, height: Math.max(1, Math.round((look.height * drawn) / look.width)) } : {}), diff --git a/frontend/src/lib/preferences.ts b/frontend/src/lib/preferences.ts index ab212ec..02f6e75 100644 --- a/frontend/src/lib/preferences.ts +++ b/frontend/src/lib/preferences.ts @@ -1,11 +1,12 @@ /** - * What a person keeps of their own way of working in the administration — + * What a person keeps of their own way of working — in the administration, * the views of the catalogue they saved, which columns each table shows them - * and how tightly it is set, whether the editor shows its preview — kept - * with their account, so it follows them from one browser to the next - * (`/account/preferences`). + * and how tightly it is set, whether the editor shows its preview; in the + * catalogue, whether the posters of works for adults are shown (see + * `adultPosters.ts`) — kept with their account, so it follows them from one + * browser to the next (`/account/preferences`). * - * The whole set is read once and held for the administration: each + * The whole set is read once and held for every page: each * preference is read from that one answer, its last value kept in this * browser as well, so a page is drawn as it was left before the answer is * in, and written back a moment after it changes — the last of several diff --git a/frontend/src/lib/prefs.test.ts b/frontend/src/lib/prefs.test.ts index 7f1229a..0611de7 100644 --- a/frontend/src/lib/prefs.test.ts +++ b/frontend/src/lib/prefs.test.ts @@ -112,4 +112,39 @@ describe('the posters of works for adults', () => { const prefs = await fresh() expect(prefs.adultPostersShown()).toBe(true) }) + + it('are as the account signed in chose, over this browser, until nobody is', async () => { + local.kept.set('ams.adultPosters', 'shown') + const prefs = await fresh() + + prefs.adoptAdultPosters(false) + expect(prefs.adultPostersShown()).toBe(false) + + prefs.adoptAdultPosters(undefined) + expect(prefs.adultPostersShown()).toBe(true) + }) + + it('are shown for an account that asked, in a browser that never did, and the browser keeps nothing of it', async () => { + const prefs = await fresh() + + prefs.adoptAdultPosters(true) + expect(prefs.adultPostersShown()).toBe(true) + expect(local.kept.has('ams.adultPosters')).toBe(false) + + // Signed out: whoever uses the browser next sees them veiled. + prefs.adoptAdultPosters(undefined) + expect(prefs.adultPostersShown()).toBe(false) + }) + + it('keep the browser’s own choice, made while an account’s counted, for when it is gone', async () => { + const prefs = await fresh() + + prefs.adoptAdultPosters(false) + prefs.showAdultPosters(true) + expect(prefs.adultPostersShown()).toBe(false) + expect(local.kept.get('ams.adultPosters')).toBe('shown') + + prefs.adoptAdultPosters(undefined) + expect(prefs.adultPostersShown()).toBe(true) + }) }) diff --git a/frontend/src/lib/prefs.ts b/frontend/src/lib/prefs.ts index 1df8395..3a1fad2 100644 --- a/frontend/src/lib/prefs.ts +++ b/frontend/src/lib/prefs.ts @@ -1,11 +1,14 @@ /** * The reader's own settings, kept in this browser as the interface's * language and the lamp are: the language the catalogue's texts are read in, - * and whether the posters of works for adults are shown as they are. + * and whether the posters of works for adults are shown as they are — which + * an account signed in may have chosen for itself, and then its choice is + * the one that counts (see `adultPosters.ts`). * * Neither leaves the browser but as what it changes: the language rides on * every read of the catalogue as its `?language=` (see `api.ts`), and the - * posters are the page's business alone. + * posters are the page's business alone — the account keeps its own choice, + * never this browser's. */ import { useSyncExternalStore } from 'react' @@ -123,24 +126,50 @@ export function useTextLanguage(): string | undefined { /* ── Posters for adults ───────────────────────────────────────────────────── */ +/** This browser's choice, as read from the storage once: `null` until it has been. */ let adultPosters: boolean | null = null +/** + * The choice of the account signed in, where it made one: nothing for a + * visitor, nor for an account that never chose. Held for the visit alone — + * the account keeps it, and puts it here again on the next. + */ +let accountPosters: boolean | undefined + /** * Whether the reader asked to see the posters of works for adults as they - * are. Not until they have: a catalogue shown on a living-room screen, or - * over a shoulder, was showing one to whoever looked. + * are: as the account signed in chose, else as this browser did. Not until + * one of them has: a catalogue shown on a living-room screen, or over a + * shoulder, was showing one to whoever looked. */ export function adultPostersShown(): boolean { + if (accountPosters !== undefined) return accountPosters if (adultPosters === null) adultPosters = read(ADULT_POSTERS_KEY) === 'shown' return adultPosters } +/** + * This browser's choice: a visitor's, and that of anybody signed in whose + * account has not made one. Kept in the storage, for the next visit. + */ export function showAdultPosters(shown: boolean) { adultPosters = shown write(ADULT_POSTERS_KEY, shown ? 'shown' : undefined) told() } +/** + * The choice of the account signed in, which counts over this browser's + * while it is there; `undefined` when nobody is signed in, or the account + * never chose, and this browser's counts again. Never written to the + * storage: whoever uses the browser next is not the account. + */ +export function adoptAdultPosters(shown: boolean | undefined) { + if (accountPosters === shown) return + accountPosters = shown + told() +} + export function useAdultPostersShown(): boolean { return useSyncExternalStore(subscribe, adultPostersShown, () => false) } diff --git a/frontend/src/lib/session.test.ts b/frontend/src/lib/session.test.ts new file mode 100644 index 0000000..b56470a --- /dev/null +++ b/frontend/src/lib/session.test.ts @@ -0,0 +1,308 @@ +import { MutationObserver, QueryClient, QueryObserver } from '@tanstack/react-query' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +import { ApiError } from './api' +import { endSession, sessionCaches } from './session' + +/** Who is looking, as much of it as these need: whether they may write, and their account. */ +interface Me { + canWrite: boolean + user?: { id: string } +} + +/** A browser's storage, kept in a map. */ +function storage(initial: Record = {}) { + const kept = new Map(Object.entries(initial)) + return { + kept, + getItem: (key: string) => kept.get(key) ?? null, + setItem: (key: string, value: string) => void kept.set(key, value), + removeItem: vi.fn((key: string) => void kept.delete(key)), + } +} + +/** A cache as the interface's, without its retries: a refusal is the answer here. */ +function cache() { + return new QueryClient({ defaultOptions: { queries: { retry: false } } }) +} + +describe('the end of a session', () => { + let local: ReturnType + + beforeEach(() => { + local = storage({ 'ams.recent': JSON.stringify([{ id: 'a', title: 'Heat', kind: 'movie', at: 1 }]) }) + vi.stubGlobal('localStorage', local) + }) + + afterEach(() => vi.unstubAllGlobals()) + + it('tells whoever reads who is looking, at once, and asks again', async () => { + const queryClient = cache() + let signedIn = true + const me = new QueryObserver(queryClient, { + queryKey: ['me'], + queryFn: () => Promise.resolve({ canWrite: signedIn }), + }) + const stop = me.subscribe(() => undefined) + await vi.waitFor(() => expect(me.getCurrentResult().data).toEqual({ canWrite: true })) + + signedIn = false + endSession(queryClient) + // Emptying the cache left this reader the answer it had: the bar kept + // "Admin" until the page was loaded again. + expect(me.getCurrentResult().data).toBeUndefined() + await vi.waitFor(() => expect(me.getCurrentResult().data).toEqual({ canWrite: false })) + stop() + }) + + it('keeps nothing the account was answered or sent', async () => { + const queryClient = cache() + let signedIn = true + // An answer nobody reads any more, and one a page still does. + queryClient.setQueryData(['account', 'keys'], { keys: ['a key'] }) + const sessions = new QueryObserver(queryClient, { + queryKey: ['account', 'sessions'], + queryFn: () => (signedIn ? Promise.resolve(['this one']) : Promise.reject(new Error('Unauthorized'))), + }) + const stop = sessions.subscribe(() => undefined) + await vi.waitFor(() => expect(sessions.getCurrentResult().data).toEqual(['this one'])) + // What was sent, and what came back. + await new MutationObserver(queryClient, { + mutationFn: (name: string) => Promise.resolve({ name, secret: 'shown once' }), + }).mutate('a key') + expect(queryClient.getMutationCache().getAll()).toHaveLength(1) + + signedIn = false + endSession(queryClient) + expect(queryClient.getQueryCache().find({ queryKey: ['account', 'keys'] })).toBeUndefined() + expect(queryClient.getQueryData(['account', 'sessions'])).toBeUndefined() + expect(queryClient.getMutationCache().getAll()).toEqual([]) + // Asked again, as whoever is here now, and refused: still nothing of theirs. + await vi.waitFor(() => expect(sessions.getCurrentResult().status).toBe('error')) + expect(sessions.getCurrentResult().data).toBeUndefined() + stop() + }) + + it('drops an answer that was still on its way for the account that left', async () => { + const queryClient = cache() + let answer: (me: Me) => void = () => undefined + const late = new Promise((resolve) => { + answer = resolve + }) + let asked = 0 + const me = new QueryObserver(queryClient, { + queryKey: ['me'], + queryFn: () => { + asked += 1 + return asked === 1 ? late : Promise.resolve({ canWrite: false }) + }, + }) + const stop = me.subscribe(() => undefined) + expect(asked).toBe(1) + + endSession(queryClient) + answer({ canWrite: true }) + await vi.waitFor(() => expect(me.getCurrentResult().data).toEqual({ canWrite: false })) + await new Promise((settled) => setTimeout(settled, 20)) + expect(me.getCurrentResult().data).toEqual({ canWrite: false }) + stop() + }) + + it('forgets the shelf of the works they opened', () => { + endSession(cache()) + expect(local.kept.has('ams.recent')).toBe(false) + }) +}) + +describe('a session ended somewhere else', () => { + let local: ReturnType + + beforeEach(() => { + local = storage({ 'ams.recent': JSON.stringify([{ id: 'a', title: 'Heat', kind: 'movie', at: 1 }]) }) + vi.stubGlobal('localStorage', local) + }) + + afterEach(() => vi.unstubAllGlobals()) + + /** A cache as the interface's: told of every refusal, and without its retries. */ + function listening() { + const queryClient: QueryClient = new QueryClient({ + ...sessionCaches(() => queryClient), + defaultOptions: { queries: { retry: false } }, + }) + return queryClient + } + + /** The server's refusal for want of a credential. */ + const refusal = () => new ApiError(401, 'unauthorized', 'Unauthorized') + + const somebody: Me = { canWrite: true, user: { id: 'u1' } } + const nobody: Me = { canWrite: false } + + /** Who is looking, read as the bar reads it, and how many times it was asked. */ + async function looking(queryClient: QueryClient, answer: () => Promise) { + let asked = 0 + const me = new QueryObserver(queryClient, { + queryKey: ['me'], + queryFn: () => { + asked += 1 + return answer() + }, + staleTime: 5 * 60_000, + }) + const stop = me.subscribe(() => undefined) + await vi.waitFor(() => expect(me.getCurrentResult().data).toEqual(somebody)) + return { me, stop, asked: () => asked } + } + + /** A page that reads something the server now refuses. */ + function refusedRead(queryClient: QueryClient, name: string) { + return new QueryObserver(queryClient, { + queryKey: ['users', name], + queryFn: () => Promise.reject(refusal()), + }).subscribe(() => undefined) + } + + /** Long enough for anything still to come to have come. */ + const settled = () => new Promise((done) => setTimeout(done, 20)) + + it('asks again who is looking when a read is refused, and ends the session when nobody is', async () => { + const queryClient = listening() + let session = true + const { me, stop, asked } = await looking(queryClient, () => Promise.resolve(session ? somebody : nobody)) + queryClient.setQueryData(['account', 'keys'], { keys: ['a key'] }) + + // Closed from another browser: nothing here hears of it but what is refused. + session = false + const page = refusedRead(queryClient, 'list') + await vi.waitFor(() => expect(me.getCurrentResult().data).toEqual(nobody)) + expect(local.kept.has('ams.recent')).toBe(false) + expect(queryClient.getQueryCache().find({ queryKey: ['account', 'keys'] })).toBeUndefined() + // Asked again, then once more by whoever is here now, and no further. + await settled() + expect(asked()).toBe(3) + page() + stop() + }) + + it('takes a refusal to say who is looking, on a private site, as the answer, and asks nothing after it', async () => { + const queryClient = listening() + let session = true + const { me, stop, asked } = await looking(queryClient, () => + session ? Promise.resolve(somebody) : Promise.reject(refusal()), + ) + + // A write, this time: refused like a read. + session = false + await expect( + new MutationObserver(queryClient, { mutationFn: () => Promise.reject(refusal()) }).mutate(), + ).rejects.toThrow('Unauthorized') + await vi.waitFor(() => expect(local.kept.has('ams.recent')).toBe(false)) + // The account's name went with the session: a refused answer used to + // leave the last one in place, and the bar with it. + await vi.waitFor(() => expect(me.getCurrentResult().status).toBe('error')) + expect(me.getCurrentResult().data).toBeUndefined() + await settled() + expect(asked()).toBe(3) + expect(local.removeItem).toHaveBeenCalledTimes(1) + stop() + }) + + it('ends nothing for a password typed wrong: the session goes on', async () => { + const queryClient = listening() + const { me, stop, asked } = await looking(queryClient, () => Promise.resolve(somebody)) + queryClient.setQueryData(['account', 'keys'], { keys: ['a key'] }) + + // Changing it, with the wrong current one: refused the same way. + await expect( + new MutationObserver(queryClient, { mutationFn: () => Promise.reject(refusal()) }).mutate(), + ).rejects.toThrow('Unauthorized') + await vi.waitFor(() => expect(asked()).toBe(2)) + await settled() + expect(me.getCurrentResult().data).toEqual(somebody) + expect(queryClient.getQueryData(['account', 'keys'])).toEqual({ keys: ['a key'] }) + expect(local.kept.has('ams.recent')).toBe(true) + stop() + }) + + it('asks once for refusals that come together, and ends the session once', async () => { + const queryClient = listening() + let session = true + const { stop, asked } = await looking(queryClient, () => Promise.resolve(session ? somebody : nobody)) + + // A page of the administration opening on a session gone: every read refused at once. + session = false + const pages = ['list', 'counts', 'invitations'].map((name) => refusedRead(queryClient, name)) + await vi.waitFor(() => expect(local.kept.has('ams.recent')).toBe(false)) + await settled() + expect(asked()).toBe(3) + expect(local.removeItem).toHaveBeenCalledTimes(1) + for (const page of pages) page() + stop() + }) + + it('asks nothing for a visitor, whose refusals are only the site’s', async () => { + const queryClient = listening() + let asked = 0 + const me = new QueryObserver(queryClient, { + queryKey: ['me'], + queryFn: () => { + asked += 1 + return Promise.resolve(nobody) + }, + }) + const stop = me.subscribe(() => undefined) + await vi.waitFor(() => expect(me.getCurrentResult().data).toEqual(nobody)) + + const page = refusedRead(queryClient, 'list') + await settled() + expect(asked).toBe(1) + expect(local.kept.has('ams.recent')).toBe(true) + page() + stop() + }) + + it('ends nothing when the server cannot say', async () => { + const queryClient = listening() + let reachable = true + const { me, stop, asked } = await looking(queryClient, () => + reachable ? Promise.resolve(somebody) : Promise.reject(new TypeError('Failed to fetch')), + ) + + reachable = false + const page = refusedRead(queryClient, 'list') + await vi.waitFor(() => expect(asked()).toBe(2)) + await settled() + expect(me.getCurrentResult().data).toEqual(somebody) + expect(local.kept.has('ams.recent')).toBe(true) + page() + stop() + }) + + it('ends it too when the answer, gone old, is asked again after the session ran out', async () => { + const queryClient = listening() + let session = true + const { me, stop } = await looking(queryClient, () => Promise.resolve(session ? somebody : nobody)) + + // A page drawn later asks again, as `useMe` does once the answer is old. + session = false + await me.refetch() + await vi.waitFor(() => expect(local.kept.has('ams.recent')).toBe(false)) + expect(me.getCurrentResult().data).toEqual(nobody) + stop() + }) + + it('is ended once by the way out: the answer that follows it asks nothing more', async () => { + const queryClient = listening() + let session = true + const { me, stop, asked } = await looking(queryClient, () => Promise.resolve(session ? somebody : nobody)) + + session = false + endSession(queryClient) + await vi.waitFor(() => expect(me.getCurrentResult().data).toEqual(nobody)) + await settled() + expect(asked()).toBe(2) + expect(local.removeItem).toHaveBeenCalledTimes(1) + stop() + }) +}) diff --git a/frontend/src/lib/session.ts b/frontend/src/lib/session.ts new file mode 100644 index 0000000..eb1ebab --- /dev/null +++ b/frontend/src/lib/session.ts @@ -0,0 +1,100 @@ +/** + * A session over, in this browser: signed out from either side of the site + * (`SignOut`); ended with every other one by a new password + * (`ChangePassword`), or by an administrator closing their own sessions or + * untying their own account from the identity provider (`UserDetail`); or + * ended somewhere else, and heard of here (`sessionCaches`). + * + * Everything the cache holds was answered for somebody who is now gone. It + * used to be emptied, and nobody was told: whoever was reading an answer went + * on reading the one it had. Who is looking is read once, above every page, + * by a part that is never drawn again by itself — so the bar went on offering + * "Admin" and the way out to a visitor until the page was loaded again, and + * the sign-in that followed, which asks again only what the cache still + * holds, never reached it either. + */ + +import { MutationCache, QueryCache, type QueryClient, type QueryKey } from '@tanstack/react-query' + +import { ApiError } from './api' +import { forget } from './recent' +import type { Me } from './types' + +/** + * The account each cache was last told is signed in, by its id: what an + * answer saying nobody is set against. Kept apart from the answer itself, + * which the cache already holds by the time it is heard. + */ +const signedIn = new WeakMap() + +/** + * Forget the account that left: the shelf of the works it opened, what it + * sent, and every answer it was given. An answer somebody is reading goes + * back to none, they are told, and it is asked again — by whoever is here + * now; one nobody is reading leaves the cache altogether. An answer still on + * its way is dropped with the rest rather than kept when it lands. + */ +export function endSession(queryClient: QueryClient) { + signedIn.delete(queryClient) + forget() + queryClient.getMutationCache().clear() + void queryClient.resetQueries() + queryClient.removeQueries({ predicate: (query) => query.getObserversCount() === 0 }) +} + +/** + * The caches of the interface's client, which hear of a session that ended + * somewhere else: it ran out, or it was closed from another browser, by + * another administrator, or by a new password set elsewhere. Nothing tells + * this page so. The server only refuses what it is asked from then on, and + * the bar went on offering "Admin" and the way out until a reload. + * + * A refusal for want of a credential, while somebody is signed in here, is a + * reason to ask again who is, not the answer: a password typed wrong is + * refused the same way, signing in or changing it, and the session goes on. + * Who is looking is asked once for any number of refusals at a time, and an + * answer that says nobody, where somebody was, ends the session here — once: + * after it, nobody is signed in, and the refusals that follow ask nothing. So + * do a visitor's, which on a private site are every answer. + * + * The same answer ends it when it comes some other way: asked again because + * it was old by the time a page wanted it, after the session had run out. + * + * Made before the client they belong to, which they reach when they hear + * something. + */ +export function sessionCaches(client: () => QueryClient) { + return { + queryCache: new QueryCache({ + onSuccess: (data, query) => { + if (isMe(query.queryKey)) heard(client(), (data as Me | undefined)?.user?.id) + }, + onError: (error, query) => { + if (!isMe(query.queryKey)) refused(client(), error) + else if (error instanceof ApiError && error.isUnauthorized) heard(client(), undefined) + }, + }), + mutationCache: new MutationCache({ onError: (error) => refused(client(), error) }), + } +} + +/** The question every page shares: who is looking (`useMe`). */ +function isMe(key: QueryKey) { + return key.length === 1 && key[0] === 'me' +} + +/** Who is looking, as the server just answered: an account, or nobody. */ +function heard(queryClient: QueryClient, account: string | undefined) { + if (account) signedIn.set(queryClient, account) + else if (signedIn.has(queryClient)) endSession(queryClient) +} + +/** + * A read or a write refused: if it was for want of a credential, and + * somebody is signed in, who is looking is asked again — joining the + * question already on its way rather than asking it once per refusal. + */ +function refused(queryClient: QueryClient, error: unknown) { + if (!(error instanceof ApiError && error.isUnauthorized) || !signedIn.has(queryClient)) return + void queryClient.invalidateQueries({ queryKey: ['me'], exact: true }, { cancelRefetch: false }) +} diff --git a/frontend/src/lib/types.ts b/frontend/src/lib/types.ts index 161be72..e4d5cce 100644 --- a/frontend/src/lib/types.ts +++ b/frontend/src/lib/types.ts @@ -317,6 +317,12 @@ export type SourceRule = Schema<'Laid'> /** How the media store stands. */ export type MediaStatus = Schema<'MediaStatus'> +/** A format whose copies can be deleted once they are no longer made: `webp` or `avif`. */ +export type MediaVariantFormat = Schema<'MediaVariantFormat'> + +/** What there was of a format's copies as their deletion began. */ +export type VariantsDeleted = Schema<'VariantsDeleted'> + /** One medium kept, or in line to be. */ export type MediaAsset = Schema<'Asset'> diff --git a/frontend/src/routes/Account.tsx b/frontend/src/routes/Account.tsx index 77f6ce3..38e950e 100644 --- a/frontend/src/routes/Account.tsx +++ b/frontend/src/routes/Account.tsx @@ -1,6 +1,7 @@ /** - * A person's own account: what they are called, how they sign in, where - * they are signed in, and the keys that act in their name. + * A person's own account: what they are called, how the catalogue is shown + * to them, how they sign in, where they are signed in, and the keys that act + * in their name. * * The same page in both shells: a member reaches it from the catalogue, an * editor or an administrator from the administration's sidebar. What it @@ -34,7 +35,9 @@ import { Segmented, Skeleton, Spinner, + Toggle, } from '../components/ui' +import { useAdultPosters } from '../lib/adultPosters' import { describeAgent } from '../lib/agent' import { ApiError, api } from '../lib/api' import { dateTime } from '../lib/format' @@ -83,6 +86,7 @@ export function Account() {
+ {me.hasPassword ? ( @@ -233,6 +237,34 @@ function Profile({ me }: { me: User }) { ) } +/* ── Reading ──────────────────────────────────────────────────────────────── */ + +/** + * How the catalogue is shown to them: the posters of works for adults, as + * they are or veiled. Kept with the account, as the bar's reading menu keeps + * it, and taken at once — not by the profile's Save, which waits for what is + * typed above it. + */ +function Reading() { + const { t } = useI18n() + const [adult, showAdult] = useAdultPosters() + + return ( + + +
+
+ +

{t.account.adultPostersHint}

+
+ +
+ + ) +} + /* ── Sessions ─────────────────────────────────────────────────────────────── */ function Sessions() { diff --git a/frontend/src/routes/Home.tsx b/frontend/src/routes/Home.tsx index 6c13127..f8a5693 100644 --- a/frontend/src/routes/Home.tsx +++ b/frontend/src/routes/Home.tsx @@ -91,8 +91,10 @@ export function Home() {
{featured ? : null} {/* Why it is the work of the day, where somebody said so: under the - plate, which has no room to spare on a phone. */} - {featured ? : null} + plate, which has no room to spare on a phone. Nothing where nobody + did, for whoever may write one too: the administration's page of + notes is where one is begun, not the way in. */} + {featured ? : null}
diff --git a/frontend/src/routes/admin/Media.tsx b/frontend/src/routes/admin/Media.tsx index 162670b..0d5ad4f 100644 --- a/frontend/src/routes/admin/Media.tsx +++ b/frontend/src/routes/admin/Media.tsx @@ -40,9 +40,22 @@ import { type Column, } from '../../components/ui' import { ApiError, api, query } from '../../lib/api' +import { cn } from '../../lib/cn' import * as fmt from '../../lib/format' import { useI18n } from '../../lib/i18n' -import type { MediaAsset, MediaByWork, MediaStatus, TaskId } from '../../lib/types' +import type { MediaAsset, MediaByWork, MediaStatus, MediaVariantFormat, TaskId, VariantsDeleted } from '../../lib/types' + +/** The switches that have a format's copies made, which can be deleted once they are off. */ +const FORMAT_OF: Partial> = { 'media.webp': 'webp', 'media.avif': 'avif' } +const SETTING_OF: Record = { webp: 'media.webp', avif: 'media.avif' } +const FORMAT_NAME: Record = { webp: 'WebP', avif: 'AVIF' } + +/** How many copies of a format are kept, and what they weigh. */ +function copiesOf(variants: MediaStatus['variants'], format: MediaVariantFormat): { copies: number; bytes: number } { + return format === 'webp' + ? { copies: variants.webp, bytes: variants.webpBytes } + : { copies: variants.avif, bytes: variants.avifBytes } +} /** An address as a row shows it: its host, and the end of its path. */ function shortAddress(origin: string): string { @@ -63,12 +76,19 @@ export function Media() { // One line under the buttons says what the last action did. const [notice, setNotice] = useState(null) const [asking, setAsking] = useState(false) + // The format being turned off while copies of it are kept, as the page + // asks what becomes of them; and the deletion the last answer started. + const [turningOff, setTurningOff] = useState(null) + const [purged, setPurged] = useState(null) const status = useQuery({ queryKey: ['media', 'status'], queryFn: () => api.get('/media/status'), // Live while something runs: the counts are the progress. - refetchInterval: (q) => (q.state.data?.fetching || q.state.data?.sweeping || q.state.data?.preparing ? 3_000 : 30_000), + refetchInterval: (q) => { + const now = q.state.data + return now?.fetching || now?.sweeping || now?.preparing || now?.purging ? 3_000 : 30_000 + }, }) const on = status.data?.backend !== undefined && status.data.backend !== 'off' @@ -83,13 +103,77 @@ export function Media() { void queryClient.invalidateQueries({ queryKey: ['media'] }) void queryClient.invalidateQueries({ queryKey: ['tasks'] }) } + // What a switch changes: the figures here, and — the WebP copies' — what + // this browser's pages are given, which the server's options say. + const changed = (key: string) => { + invalidate() + if (key === 'media.webp') void queryClient.invalidateQueries({ queryKey: ['auth', 'options'] }) + } const set = useMutation({ mutationFn: ({ key, value }: { key: string; value: string }) => api.put('/settings/server/-', { key, value }), - onSuccess: invalidate, + onSuccess: (_, { key }) => changed(key), + }) + + // Every copy of a format deleted, in the background on the server, which + // answers at once with what there was. Asked once more a second later + // when it is refused for now (409): of several instances, the one asked + // may not have heard yet that the format was turned off. + const purge = useMutation({ + mutationFn: async (format: MediaVariantFormat) => { + const path = `/media/variants/${format}` + try { + return await api.delete(path) + } catch (error) { + if (!(error instanceof ApiError) || error.status !== 409) throw error + await new Promise((resolve) => window.setTimeout(resolve, 1_000)) + return await api.delete(path) + } + }, + onSuccess: (deleted) => { + setTurningOff(null) + setPurged(deleted) + invalidate() + }, }) + // A format turned off from the dialog — the server deletes none of its + // copies while it is on — and then, where that was the answer, its + // copies deleted. + const turnOff = useMutation({ + mutationFn: ({ format }: { format: MediaVariantFormat; purge: boolean }) => + api.put('/settings/server/-', { key: SETTING_OF[format], value: 'false' }), + onSuccess: (_, { format, purge: deleting }) => { + changed(SETTING_OF[format]) + if (deleting) purge.mutate(format) + else setTurningOff(null) + }, + }) + + // A switch pressed. Turning a format's copies off while some are kept + // asks first whether to delete them — counted again before it asks, as + // the pages have copies made between two looks at the store; anything + // else is done at once. + const flip = (key: string, next: boolean) => { + const format = FORMAT_OF[key] + if (!format || next) { + set.mutate({ key, value: String(next) }) + return + } + void status.refetch().then(({ data }) => { + if (data && copiesOf(data.variants, format).copies > 0) { + turnOff.reset() + purge.reset() + setTurningOff(format) + } else { + set.mutate({ key, value: 'false' }) + } + }) + } + const answering = turnOff.isPending || purge.isPending + const asked = turningOff && status.data ? { format: turningOff, ...copiesOf(status.data.variants, turningOff) } : null + const run = useMutation({ mutationFn: (id: TaskId) => api.post<{ jobId?: string }>(`/tasks/${id}/run`), onMutate: () => setNotice(null), @@ -191,11 +275,12 @@ export function Media() { )}

- {(status.data.fetching || status.data.sweeping || status.data.preparing || !status.data.publicUrl) ? ( + {(status.data.fetching || status.data.sweeping || status.data.preparing || status.data.purging || !status.data.publicUrl) ? (
{status.data.fetching ? {p.fetching} : null} {status.data.sweeping ? {p.sweeping} : null} {status.data.preparing ? {p.preparing} : null} + {status.data.purging ? {p.purging} : null} {!status.data.publicUrl ? (

@@ -229,6 +314,7 @@ export function Media() { ['media.store', p.storeOn, p.storeOnHint, status.data.storing], ['media.people', p.people, p.peopleHint, status.data.people], ['media.audio', p.audio, p.audioHint, status.data.audio], + ['media.webp', p.webp, p.webpHint, status.data.webp], ['media.avif', p.avif, p.avifHint, status.data.avif], ] as const ).map(([key, label, hint, checked]) => ( @@ -243,8 +329,8 @@ export function Media() { id={`media-${key}`} checked={checked} label={label} - disabled={set.isPending} - onChange={(next) => set.mutate({ key, value: String(next) })} + disabled={set.isPending || answering} + onChange={(next) => flip(key, next)} /> ))} @@ -267,6 +353,18 @@ export function Media() { ) : null} + {/* What the last deletion started on: the lamp above says + while it runs, and the copies' line counts it down. */} +

+ {purged + ? p.turnOff.deleting( + purged.copies, + fmt.count(purged.copies, locale), + FORMAT_NAME[purged.format], + fmt.bytes(purged.bytes, locale), + ) + : ''} +

{set.isError ? (

{set.error instanceof ApiError ? set.error.message : t.common.actionFailed} @@ -309,7 +407,11 @@ export function Media() {

{status.data.unprepared > 0 ? ( <> -

{p.unprepared(status.data.unprepared)}

+ {/* Without the WebP copies, preparing a picture is + working out its placeholder alone. */} +

+ {status.data.webp ? p.unprepared(status.data.unprepared) : p.unpreparedWhole(status.data.unprepared)} +

+ + + + ) : null + } + > + {asked + ? p.turnOff[asked.format](asked.copies, fmt.count(asked.copies, locale), fmt.bytes(asked.bytes, locale)) + : null} + {turnOff.isError ? ( +

+ {turnOff.error instanceof ApiError ? turnOff.error.message : t.common.actionFailed} +

+ ) : purge.isError ? ( +

+ {p.turnOff.kept(purge.error instanceof ApiError ? purge.error.message : t.common.actionFailed)} +

+ ) : null} + + { + refresh() + void queryClient.invalidateQueries({ queryKey: ['me'] }) + } + : refresh return (
@@ -154,13 +168,13 @@ export function UserDetail() {
- - - {user.oidcLinked ? : null} + + {mine ? : } + {user.oidcLinked ? : null}
- + {mine ? null : ( @@ -296,16 +310,53 @@ function PasswordReset({ id }: { id: string }) { ) } -/** The tie to the identity provider, and the way to undo it. */ -function ProviderLink({ user, onChanged }: { user: User; onChanged: () => void }) { +/** + * Your own password, which is not given from here. The server refuses it — a + * session left open must not be enough to lock its owner out — and the page + * offered it all the same, to answer with that refusal. It is changed from + * your account page, which asks for the current one first and signs you out + * everywhere once it is done. + */ +function OwnPassword() { const { t } = useI18n() + + return ( + + +

+ {t.admin.people.resetOwn} + + {t.admin.account} + + +

+ + ) +} + +/** + * The tie to the identity provider, and the way to undo it — which closes the + * account's sessions, and on your own account this one with them. + */ +function ProviderLink({ user, mine, onChanged }: { user: User; mine: boolean; onChanged: () => void }) { + const { t } = useI18n() + const navigate = useNavigate() + const queryClient = useQueryClient() const [asking, setAsking] = useState(false) const unlink = useMutation({ mutationFn: () => api.delete(`/users/${encodeURIComponent(user.id)}/oidc`), onSuccess: () => { setAsking(false) - onChanged() + if (mine) { + endSession(queryClient) + navigate('/login', { replace: true }) + } else { + onChanged() + } }, }) @@ -335,24 +386,47 @@ function ProviderLink({ user, onChanged }: { user: User; onChanged: () => void } } >

{t.admin.people.unlinkBody}

+ {mine ?

{t.admin.people.unlinkOwn}

: null} + {unlink.isError ? ( +

+ {unlink.error.message} +

+ ) : null} ) } +/** + * Where the account is signed in, and the way to close it all. On your own + * account that is this session too, so it is asked first and then signs you + * out here, as the way out does. + */ function SessionsPanel({ id, sessions, + mine, onChanged, }: { id: string sessions: Detail['sessions'] + mine: boolean onChanged: () => void }) { const { t, locale } = useI18n() + const navigate = useNavigate() + const queryClient = useQueryClient() + const [asking, setAsking] = useState(false) const closeAll = useMutation({ mutationFn: () => api.delete(`/users/${segment(id)}/sessions`), - onSuccess: onChanged, + onSuccess: () => { + if (mine) { + endSession(queryClient) + navigate('/login', { replace: true }) + } else { + onChanged() + } + }, }) return ( @@ -361,7 +435,13 @@ function SessionsPanel({ title={t.account.sessions} action={ sessions.length ? ( - ) : null @@ -383,6 +463,28 @@ function SessionsPanel({ ) : ( )} + + setAsking(false)} + footer={ + <> + + + + } + > + {t.admin.people.closeOwnBody} + {closeAll.isError ? ( +

+ {closeAll.error.message} +

+ ) : null} + ) } diff --git a/scripts/e2e-stack.py b/scripts/e2e-stack.py index 63274cf..d9bfebe 100755 --- a/scripts/e2e-stack.py +++ b/scripts/e2e-stack.py @@ -14,6 +14,12 @@ number of failed checks, capped at 100; skipped checks (with the reason) are not failures. +The series and the film entered by hand (HAND_SERIES, HAND_FILM) are made +through the native API by the first phase that needs one, and found by the +others in the catalogue: no phase counts on another having run. A phase that +locks a title lifts the lock again — the push phase leaves its own, as the +real works' are. + Python 3.8 or later, the standard library and a `curl` on the path (the clients' door is reached by the names the clients call it by, which curl resolves with --resolve and Python's ssl would need a hosts file for). @@ -58,6 +64,31 @@ FRENCH = {"id": 2, "name": "French"} # Sonarr's Language.French ENGLISH = {"id": 1, "name": "English"} +# Works entered by hand that nothing elsewhere names — a series with no +# TheTVDB, TMDB or Fankai id, a film with no TMDB id — are handed a number of +# the server's own, from 1 000 000 000 to 2 147 483 647, made of the work's id +# in the catalogue. Sonarr keeps it as the series' TheTVDB id, Radarr as the +# film's TMDB id. The titles are ones no real work bears, so that a search by +# text finds these and not what a provider lists under a name like them; +# scripts/e2e-stack.sh names the videos Plex scans after the same ones. +HAND_SERIES = "Quenzabar Tides" +HAND_SERIES_YEAR = 2024 +HAND_EPISODES = ((1, "The Tide Goes Out", "2024-01-10"), (2, "The Tide Comes Back", "2024-01-17")) +# A spelling a release group might use. Sonarr's scene mapping takes it from +# the catalogue; the title a series is served under is one Sonarr knows +# already, so a series with no other title is mapped under none. +HAND_ALTERNATIVE = "Tides of Quenzabar" +HAND_FILM = "Brindlemoss Without Source" +HAND_FILM_YEAR = 2019 +HAND_FILM_STUDIO = "Hand Films" +# What the catalogue files them under: a lock on a title leaves the slug as it +# was made, where the title itself shows the lock. +HAND_SLUGS = {"series": "quenzabar-tides", "movie": "brindlemoss-without-source"} +OWN_FIRST, OWN_LAST = 1_000_000_000, 2_147_483_647 +# What the seed says of a work its library holds under such a number once the +# catalogue is new: nothing here is made of it, and no source could be asked. +SEED_NO_ID = "held there under no id to look it up by" + PLEX_IDENTIFIER = "tv.plex.agents.custom.arrmetadataserver" SONARR_ROOT = "/config/tv" @@ -261,6 +292,15 @@ def item(self, kind, key, value): return item return None + def by_hand(self, kind): + """The series or the film entered by hand for these checks, if the + catalogue holds it: found by its slug, which a lock on its title leaves + as it was made.""" + for item in self.items(kind): + if item.get("isManual") and item.get("slug", "").startswith(HAND_SLUGS[kind]): + return item + return None + def lock_title(self, item_id, title): self.call("PUT", f"/items/{item_id}/overrides", {"field": "title", "value": title}) @@ -373,6 +413,58 @@ def settle(native, kind, key, wanted, label): raise Failure(f"{label}: the catalogue is not whole after six refreshes: {last}") +# ── Works entered by hand ────────────────────────────────────────────────── + + +def own_number(catalogue_id): + """The number the server hands a work entered by hand: its catalogue id's + FNV-1a hash on 64 bits, taken into the range (`own` in src/service/ids.rs). + It is worked out here as it is there because a client keeps the number for + good: were it made another way one day, every such work a Sonarr or a + Radarr holds would be under a number nobody answers for.""" + hashed = 0xCBF29CE484222325 + for byte in catalogue_id.encode(): + hashed = ((hashed ^ byte) * 0x100000001B3) & 0xFFFFFFFFFFFFFFFF + return OWN_FIRST + hashed % (OWN_LAST - OWN_FIRST + 1) + + +def entered_by_hand(native, kind): + """The series or the film entered by hand, made as the administration's own + editor makes it — *New entry*, and for a series an alternative title, a + season and its episodes — unless the catalogue holds it already: a phase + before, or a run before, made it, and each phase must find the one work + rather than make another of the same title, which would be handed another + number.""" + held = native.by_hand(kind) + if held is not None: + return held + if kind == "movie": + return native.call("POST", "/items", { + "kind": "movie", "title": HAND_FILM, "year": HAND_FILM_YEAR, "status": "released", + "overview": "A film entered by hand, with no source behind it.", "runtime": 93, + "inCinemas": f"{HAND_FILM_YEAR}-05-01", "studio": HAND_FILM_STUDIO, "genres": ["Drama"], + }) + made = native.call("POST", "/items", { + "kind": "series", "title": HAND_SERIES, "year": HAND_SERIES_YEAR, "status": "continuing", + "overview": "A series entered by hand, with no source behind it.", "network": "Hand TV", + "firstAired": HAND_EPISODES[0][2], "genres": ["Animation"], + }) + native.call("POST", f"/items/{made['id']}/alternative-titles", {"title": HAND_ALTERNATIVE, "titleType": "working", "language": "en"}) + native.call("POST", f"/items/{made['id']}/seasons", {"seasonNumber": 1}) + for number, title, aired in HAND_EPISODES: + native.call("POST", f"/items/{made['id']}/episodes", { + "seasonNumber": 1, "episodeNumber": number, "title": title, "airDate": aired, + "overview": f"{title}, typed in.", "runtime": 24, + }) + return made + + +def lift_title(native, item_id): + """The lock on a work's title taken off again: the work is left as it was + typed, for the next phase and the next run to find.""" + native.call("DELETE", f"/items/{item_id}/overrides/item/title") + + # ── Sonarr 4 and Sonarr 5 ────────────────────────────────────────────────── @@ -504,9 +596,10 @@ def _kind_fits(value, kind): return True -def contract_problems(show, five): +def contract_problems(show, five, optional=()): """What of a served series Sonarr cannot read: a missing field it needs, a - field of another kind, in the series, its episodes and what they hold.""" + field of another kind, in the series, its episodes and what they hold. + `optional` names the fields this series may lack.""" problems = [] def check(node, fields, where): @@ -525,7 +618,7 @@ def check(node, fields, where): problems.append(f"{where}.{name}[{i}] is not an object") for name in _SHOW_NEEDED + (_SHOW_NEEDED_5 if five else ()): - if show.get(name) in (None, ""): + if name not in optional and show.get(name) in (None, ""): problems.append(f"the series has no {name}") check(show, _SHOW, "show") for i, episode in enumerate(show.get("episodes") or []): @@ -706,6 +799,9 @@ def phase_sonarr(R, stack, arr): held = arr.get(f"{arr.api}/series/{ids[LOCKED]}") expect(held["title"] == text, f"Sonarr's title is {held['title']!r}, not {text!r}") + # ── A series entered by hand: no id anywhere, a number of the server's own ─ + phase_sonarr_by_hand(R, stack, arr) + # ── Scene mappings: Sonarr's own list, this catalogue's titles added ─── phase_scene(R, stack, arr, ids) @@ -863,6 +959,122 @@ def phase_frozen(R, stack, arr, ids): ) +def phase_sonarr_by_hand(R, stack, arr): + """A series entered by hand, which nothing elsewhere names: Sonarr finds it + by its title and by `tvdb:` and the number it was handed, adds it with the + season and the episodes it was given, and takes a title locked here at its + next refresh — under the same number throughout. A check whose work is + missing is skipped: the first one says why.""" + label, v5, native = arr.label, arr.version.startswith("5."), stack.native + item = number = sonarr_id = None + + with R.check(f"{label}: a series entered by hand is in the catalogue, with its season and episodes, and no id elsewhere"): + item = entered_by_hand(native, "series") + number = own_number(item["id"]) + named = {k: v for k, v in (item.get("externalIds") or {}).items() if v} + expect(not named, f"it is named by {named}: not a series nothing elsewhere names") + R.note(f"{label}: catalogue id {item['id']}, which makes the number {number}") + with R.check(f"{label}: a search by its title lists the series entered by hand, under a number of the server's own"): + if item is None: + raise Skipped("no series entered by hand") + found = arr.get(f"{arr.api}/series/lookup?term={urllib.parse.quote(HAND_SERIES.lower())}") + listed = [s for s in found if s.get("title") == HAND_SERIES] + expect(listed, f"not among {[(s['title'], s['tvdbId']) for s in found[:6]]}") + expect(len(listed) == 1, f"listed {len(listed)} times, under {[s['tvdbId'] for s in listed]}") + expect(OWN_FIRST <= listed[0]["tvdbId"] <= OWN_LAST, f"listed under {listed[0]['tvdbId']}, which is not in the server's own range") + expect(listed[0]["tvdbId"] == number, f"listed under {listed[0]['tvdbId']}; its catalogue id makes {number}") + with R.check(f"{label}: a search by tvdb: and that number finds the series entered by hand, and it alone"): + if number is None: + raise Skipped("no series entered by hand") + found = arr.get(f"{arr.api}/series/lookup?term=tvdb:{number}") + expect( + len(found) == 1 and found[0]["tvdbId"] == number and found[0]["title"] == HAND_SERIES, + f"answered {[(s['title'], s['tvdbId']) for s in found[:4]]}", + ) + # A number of the range no work was handed names nothing, and the + # server says so at once: no source has it to be asked. + unhanded = OWN_FIRST if number != OWN_FIRST else OWN_FIRST + 1 + began = time.time() + found = arr.get(f"{arr.api}/series/lookup?term=tvdb:{unhanded}") + expect(not found, f"tvdb:{unhanded}, which no work was handed, found {[(s['title'], s['tvdbId']) for s in found[:4]]}") + expect(time.time() - began < 15, f"tvdb:{unhanded} took {time.time() - began:.0f}s to find nothing: was a source asked for it?") + with R.check(f"{label}: the server serves the series entered by hand under that number on both editions, with its episodes"): + if number is None: + raise Skipped("no series entered by hand") + for edition, path in (("v1", f"/v1/tvdb/shows/en/{number}"), ("v2", f"/v2/tvdb/shows/eng/{number}/official")): + shown = stack.door.get("skyhook.sonarr.tv", path) + expect(shown["tvdbId"] == number and shown["title"] == HAND_SERIES, f"{edition}: {shown['title']!r} as {shown['tvdbId']}") + expect(len(shown["episodes"]) == len(HAND_EPISODES), f"{edition}: {len(shown['episodes'])} episodes") + expect(all(e.get("tvdbShowId", number) == number for e in shown["episodes"]), f"{edition}: an episode is of another series") + with R.check(f"{label}: the series entered by hand is added"): + if number is None: + raise Skipped("no series entered by hand") + # A catalogue the seed phase replaced handed the series another number, + # and Sonarr, never told, still holds it under the old one, at the very + # path this add needs: that one goes, as a run that began afresh has none. + for stale in arr.get(f"{arr.api}/series"): + if OWN_FIRST <= stale["tvdbId"] <= OWN_LAST and stale["tvdbId"] != number: + arr.call("DELETE", f"{arr.api}/series/{stale['id']}?deleteFiles=false") + sonarr_id = add_series(stack, arr, number) + settle_commands(arr, SONARR_REFRESHES) + with R.check(f"{label}: the series entered by hand has the season and the episodes it was given, as the server serves them"): + if sonarr_id is None: + raise Skipped("the series entered by hand is not added") + held = arr.get(f"{arr.api}/series/{sonarr_id}") + episodes = arr.get(f"{arr.api}/episode?seriesId={sonarr_id}") + served = served_show(stack, arr, number) + expect(held["tvdbId"] == number, f"Sonarr holds it under {held['tvdbId']}, not {number}") + expect(held["title"] == served["title"] == HAND_SERIES, f"title {held['title']!r}, served {served['title']!r}") + seasons_here = sorted(s["seasonNumber"] for s in held["seasons"]) + seasons_served = sorted(s["seasonNumber"] for s in served["seasons"]) + expect(seasons_here == seasons_served == [1], f"seasons: Sonarr {seasons_here}, served {seasons_served}") + typed = {(1, n): (title, aired) for n, title, aired in HAND_EPISODES} + here = {key_of(e): (e["title"], e.get("airDate")) for e in episodes} + there = {key_of(e): (e["title"], e.get("airDate")) for e in served["episodes"]} + expect(here == typed, f"Sonarr holds {sorted(here.items())}; it was given {sorted(typed.items())}") + expect(here == there, f"Sonarr holds {sorted(here.items())}; served {sorted(there.items())}") + R.note(f"{label}: {held['title']!r} {held.get('year')}, {len(episodes)} episodes, status {held.get('status')}, path {held.get('path')}") + with R.check(f"{label}: the series entered by hand is served with every field Sonarr {arr.version[:1]} reads, as it reads it"): + if number is None: + raise Skipped("no series entered by hand") + # No source says which language a series entered by hand was made in, + # and nobody typed one: Sonarr 5 takes its own default for it. + problems = contract_problems(served_show(stack, arr, number), v5, optional=("originalLanguage",)) + expect(not problems, "; ".join(problems)) + if v5: + with R.check(f"{label}: Sonarr 5 is sent metadataSource local for the series entered by hand, in a search and when it asks for it"): + if number is None: + raise Skipped("no series entered by hand") + searched = stack.door.get("skyhook.sonarr.tv", f"/v2/tvdb/search?term={urllib.parse.quote(HAND_SERIES.lower())}&language=eng") + hit = [s for s in searched if s.get("tvdbId") == number] + expect(hit and hit[0].get("metadataSource") == "local", f"the search says {[(s.get('tvdbId'), s.get('metadataSource')) for s in searched[:4]]}") + source = served_show(stack, arr, number).get("metadataSource") + expect(source == "local", f"the series is served with metadataSource {source!r}") + # Sonarr may not show it back: what it holds is then the server's answer above. + held = arr.get(f"{arr.api}/series/{sonarr_id}") if sonarr_id else {} + if "metadataSource" in held: + expect(held["metadataSource"] == "local", f"Sonarr holds metadataSource {held['metadataSource']!r}") + else: + R.note(f"{label}: Sonarr's series resource shows no metadataSource; the server's own answer says local") + with R.check(f"{label}: a title locked on the series entered by hand appears in Sonarr after its refresh, and goes with the lock"): + if sonarr_id is None: + raise Skipped("the series entered by hand is not added") + text = f"Locked by the e2e stack ({label}, by hand)" + native.lock_title(item["id"], text) + try: + refresh_series(arr, sonarr_id) + held = arr.get(f"{arr.api}/series/{sonarr_id}") + expect(held["title"] == text, f"Sonarr's title is {held['title']!r}, not {text!r}") + expect(held["tvdbId"] == number, f"Sonarr holds it under {held['tvdbId']} after the refresh, not {number}") + finally: + lift_title(native, item["id"]) + refresh_series(arr, sonarr_id) + held = arr.get(f"{arr.api}/series/{sonarr_id}") + expect(held["title"] == HAND_SERIES, f"Sonarr's title is {held['title']!r} after the lock was lifted") + episodes = arr.get(f"{arr.api}/episode?seriesId={sonarr_id}") + expect(len(episodes) == len(HAND_EPISODES), f"{len(episodes)} episodes after the refreshes") + + def phase_scene(R, stack, arr, ids): """Sonarr takes a release for a series by the titles of its scene-mapping list; this server adds the catalogue's. A Sonarr 5 skips a mapping that @@ -885,6 +1097,23 @@ def phase_scene(R, stack, arr, ids): misses.append((mapping["title"], mapping["tvdbId"], found)) expect(not misses, f"{len(misses)} of {min(len(added), 6)} releases are not taken for their series: {misses[:3]}") R.note(f"{label}: {len(added)} mappings added by this catalogue, tried {min(len(added), 6)}") + with R.check(f"{label}: the scene mapping names the series entered by hand by its own number, and a release named so finds it"): + item = stack.native.by_hand("series") + if item is None: + raise Skipped("the catalogue holds no series entered by hand") + if not hasattr(stack, "scene_began"): + raise Skipped("Update Scene Mapping did not complete above") + number = own_number(item["id"]) + added = scene_additions(stack, {number}) + expect(added, f"this catalogue adds no mapping under {number}") + if series_by_tvdb(arr, number) is None: + raise Skipped("Sonarr does not hold the series entered by hand: its own phase adds it") + for mapping in added: + release = re.sub(r"[\W_]+", ".", mapping["title"]).strip(".") + ".S01E01.1080p.WEB.x264-GRP" + parsed = arr.get(f"/api/v3/parse?title={urllib.parse.quote(release)}") + found = (parsed.get("series") or {}).get("tvdbId") + expect(found == number, f"{release!r} is taken for tvdb:{found}, not {number}") + R.note(f"{label}: mapped as {[m['title'] for m in added]}") if arr.version.startswith("5."): with R.check(f"{label}: no scene mapping is skipped for want of a mappingId"): # Only what Sonarr logged since the command: the log of a stack kept @@ -988,12 +1217,114 @@ def phase_radarr(R, stack, arr): expect(state.get("status") == "completed", f"RefreshMovie ended {state.get('status')}") movie = arr.get(f"/api/v3/movie/{held[603]}") expect(movie["title"] == text, f"Radarr's title is {movie['title']!r}, not {text!r}") + + # ── A film entered by hand: no TMDB id, a number of the server's own ─── + phase_radarr_by_hand(R, stack, arr) + + # Last: it reads the whole of Radarr's log. with R.check(f"{label}: Radarr refused no certificate"): log = subprocess.run(["docker", "logs", "ams-e2e-radarr"], capture_output=True, text=True) lines = [l for l in (log.stdout + log.stderr).splitlines() if "RemoteCertificate" in l] expect(not lines, f"{len(lines)} refused certificates in Radarr's log: {lines[0][:160] if lines else ''}") +def phase_radarr_by_hand(R, stack, arr): + """A film entered by hand, which has no TMDB id: Radarr finds it by its + title and by `tmdb:` and the number it was handed, adds it, holds it as the + server serves it, and takes a title locked here at its next refresh — under + the same number throughout. A check whose work is missing is skipped: the + first one says why.""" + label, native = arr.label, stack.native + item = number = movie_id = None + + with R.check(f"{label}: a film entered by hand is in the catalogue, with no id elsewhere"): + item = entered_by_hand(native, "movie") + number = own_number(item["id"]) + named = {k: v for k, v in (item.get("externalIds") or {}).items() if v} + expect(not named, f"it is named by {named}: not a film nothing elsewhere names") + R.note(f"{label}: catalogue id {item['id']}, which makes the number {number}") + with R.check(f"{label}: a search by its title lists the film entered by hand, under a number of the server's own"): + if item is None: + raise Skipped("no film entered by hand") + found = arr.get(f"/api/v3/movie/lookup?term={urllib.parse.quote(HAND_FILM.lower())}") + listed = [m for m in found if m.get("title") == HAND_FILM] + expect(listed, f"not among {[(m['title'], m['tmdbId']) for m in found[:6]]}") + expect(len(listed) == 1, f"listed {len(listed)} times, under {[m['tmdbId'] for m in listed]}") + expect(OWN_FIRST <= listed[0]["tmdbId"] <= OWN_LAST, f"listed under {listed[0]['tmdbId']}, which is not in the server's own range") + expect(listed[0]["tmdbId"] == number, f"listed under {listed[0]['tmdbId']}; its catalogue id makes {number}") + with R.check(f"{label}: a search by tmdb: and that number finds the film entered by hand, and it alone"): + if number is None: + raise Skipped("no film entered by hand") + found = arr.get(f"/api/v3/movie/lookup?term=tmdb:{number}") + expect( + len(found) == 1 and found[0]["tmdbId"] == number and found[0]["title"] == HAND_FILM, + f"answered {[(m['title'], m['tmdbId']) for m in found[:4]]}", + ) + # A number of the range no work was handed names nothing, and the + # server says so at once: no source has it to be asked. + unhanded = OWN_FIRST if number != OWN_FIRST else OWN_FIRST + 1 + began = time.time() + found = arr.get(f"/api/v3/movie/lookup?term=tmdb:{unhanded}") + expect(not found, f"tmdb:{unhanded}, which no work was handed, found {[(m['title'], m['tmdbId']) for m in found[:4]]}") + expect(time.time() - began < 15, f"tmdb:{unhanded} took {time.time() - began:.0f}s to find nothing: was a source asked for it?") + with R.check(f"{label}: the film entered by hand is added"): + if number is None: + raise Skipped("no film entered by hand") + # One a catalogue the seed phase replaced handed another number is still + # held under the old one, at the very path this add needs: it goes. + for stale in arr.get("/api/v3/movie"): + if OWN_FIRST <= stale["tmdbId"] <= OWN_LAST and stale["tmdbId"] != number: + arr.call("DELETE", f"/api/v3/movie/{stale['id']}?deleteFiles=false") + existing = [m for m in arr.get("/api/v3/movie") if m.get("tmdbId") == number] + if existing: + movie_id = existing[0]["id"] + else: + body = [m for m in arr.get(f"/api/v3/movie/lookup?term=tmdb:{number}") if m.get("tmdbId") == number][0] + body.update( + qualityProfileId=1, rootFolderPath=RADARR_ROOT, monitored=False, minimumAvailability="released", + addOptions={"monitor": "none", "searchForMovie": False}, + ) + try: + movie_id = arr.call("POST", "/api/v3/movie", body)["id"] + except HttpError as e: + raise Failure(f"Radarr refused the add of tmdb:{number}: {e}") from None + settle_commands(arr, RADARR_REFRESHES) + with R.check(f"{label}: the film entered by hand is refreshed, and is what the server serves"): + if movie_id is None: + raise Skipped("the film entered by hand is not added") + state = arr.command("RefreshMovie", movieIds=[movie_id]) + expect(state.get("status") == "completed", f"RefreshMovie ended {state.get('status')}: {state.get('message')}") + movie = arr.get(f"/api/v3/movie/{movie_id}") + served = stack.door.get("api.radarr.video", f"/v1/movie/{number}") + expect(movie["tmdbId"] == number == served["TmdbId"], f"Radarr holds it under {movie['tmdbId']}, served {served['TmdbId']}, made {number}") + expect(movie["title"] == served["Title"] == HAND_FILM, f"title {movie['title']!r}, served {served['Title']!r}") + expect(movie["year"] == served["Year"] == HAND_FILM_YEAR, f"year {movie['year']}, served {served['Year']}") + expect(movie["runtime"] == served["Runtime"] == 93, f"runtime {movie['runtime']}, served {served['Runtime']}") + expect((movie.get("studio") or "") == (served.get("Studio") or "") == HAND_FILM_STUDIO, f"studio {movie.get('studio')!r}, served {served.get('Studio')!r}") + expect(movie.get("overview") == served.get("Overview"), "the overview is not the served one") + with R.check(f"{label}: a title locked on the film entered by hand appears in Radarr after its refresh, and goes with the lock"): + if movie_id is None: + raise Skipped("the film entered by hand is not added") + text = f"Locked by the e2e stack ({label}, by hand)" + # An instant, UTC, a moment before the edit: what was changed since. + since = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(time.time() - 5)) + native.lock_title(item["id"], text) + try: + changed = stack.door.get("api.radarr.video", f"/v1/movie/changed?since={since}") + expect(number in changed, f"the films changed since {since} are {len(changed)}, and not {number}") + state = arr.command("RefreshMovie", movieIds=[movie_id]) + expect(state.get("status") == "completed", f"RefreshMovie ended {state.get('status')}: {state.get('message')}") + movie = arr.get(f"/api/v3/movie/{movie_id}") + expect(movie["title"] == text, f"Radarr's title is {movie['title']!r}, not {text!r}") + expect(movie["tmdbId"] == number, f"Radarr holds it under {movie['tmdbId']} after the refresh, not {number}") + finally: + lift_title(native, item["id"]) + state = arr.command("RefreshMovie", movieIds=[movie_id]) + expect(state.get("status") == "completed", f"RefreshMovie ended {state.get('status')}") + movie = arr.get(f"/api/v3/movie/{movie_id}") + expect(movie["title"] == HAND_FILM, f"Radarr's title is {movie['title']!r} after the lock was lifted") + + # ── The other way: the server asks Sonarr and Radarr ─────────────────────── @@ -1013,6 +1344,68 @@ def newest_command(arr, name, since_ids): return None +def held_as(arr, kind, number): + """The series or the film a client holds under that id: its TheTVDB id for + a series, its TMDB id for a film.""" + if kind == "series": + return series_by_tvdb(arr, number) + return next((m for m in arr.get("/api/v3/movie") if m.get("tmdbId") == number), None) + + +def push_after_lock(R, native, arr, name, ids_field, kind, work): + """A lock put on a work sends the client its refresh — RefreshSeries or + RefreshMovie, for that work — and the server records the push. `work` is + the id the client holds the work under, TheTVDB's or TMDB's; of the series + or the film entered by hand, which nothing elsewhere names, it is None: the + client holds it under the number the server made of its catalogue id. The + lock is left on, as is the real work's: lifting it would send a second + command a moment later, to whichever client is connected by then.""" + label, is_sonarr, by_hand = arr.label, kind == "series", work is None + held_by = f"the {kind} entered by hand" if by_hand else f"{kind} {work}" + skipped = False + with R.check(f"push: a lock put on the {kind} entered by hand sends {name} to {label}" if by_hand else f"push: a lock put in the catalogue sends {name} to {label}"): + if by_hand: + item = native.by_hand(kind) + work = own_number(item["id"]) if item else None + else: + item = native.item(kind, "tvdb" if is_sonarr else "tmdb", work) + find = held_as(arr, kind, work) if work else None + if find is None: + skipped = True + raise Skipped(f"{label} does not hold {held_by}: run its own phase first") + expect(item is not None, f"{held_by} is not in the catalogue") + before = {c["id"] for c in arr.get("/api/v3/command")} + text = f"Pushed to {label} at {int(time.time())}" + native.lock_title(item["id"], text) + found = None + # Ten seconds after the last change, or a minute after the first. + for _ in range(100): + command = newest_command(arr, name, before) + if command and find["id"] in (command.get("body") or {}).get(ids_field, []): + found = command + break + time.sleep(2) + expect(found, f"{label} was sent no {name} for its {kind} {find['id']} in 200 s") + for _ in range(120): + state = arr.get(f"/api/v3/command/{found['id']}") + if state.get("status") in ("completed", "failed", "aborted"): + break + time.sleep(1) + expect(state.get("status") == "completed", f"the command ended {state.get('status')}") + path = f"{arr.api}/series/{find['id']}" if is_sonarr else f"/api/v3/movie/{find['id']}" + held = arr.get(path) + expect(held["title"] == text, f"{label} holds the title {held['title']!r}, not the lock {text!r}") + expect((held["tvdbId"] if is_sonarr else held["tmdbId"]) == work, f"{label} holds it under another id than {work}") + with R.check(f"push: the server's own record of it, for {label}, the {kind} entered by hand" if by_hand else f"push: the server's own record of it, for {label}"): + if skipped: + raise Skipped(f"no lock was put on {held_by}") + app = "sonarr" if is_sonarr else "radarr" + sent = [p for p in native.call("GET", "/admin/arr").get("pushes", []) if p.get("app") == app and p.get("outcome") == "sent"] + expect(sent, f"the page lists no {app} work sent") + expect("lock" in sent[0].get("reasons", []), f"the latest send was for {sent[0].get('reasons')}, not for a lock") + R.note(f"push: {label}: {sent[0].get('title')!r} sent as {sent[0].get('detail')!r}") + + def phase_push(R, stack): native = stack.native pairs = [] @@ -1030,39 +1423,9 @@ def phase_push(R, stack): connect(native, sonarr, radarr, push=True) tested = native.call("POST", "/admin/arr/test", {"app": "sonarr" if is_sonarr else "radarr"}) expect(tested.get("ok"), f"the test answered {tested}") - with R.check(f"push: a lock put in the catalogue sends {name} to {label}"): - find = series_by_tvdb(arr, work) if is_sonarr else next((m for m in arr.get("/api/v3/movie") if m.get("tmdbId") == work), None) - if find is None: - raise Skipped(f"{label} does not hold {kind} {work}: run its own phase first") - item = native.item("series" if is_sonarr else "movie", "tvdb" if is_sonarr else "tmdb", work) - expect(item is not None, f"{kind} {work} is not in the catalogue") - before = {c["id"] for c in arr.get("/api/v3/command")} - text = f"Pushed to {label} at {int(time.time())}" - native.lock_title(item["id"], text) - found = None - # Ten seconds after the last change, or a minute after the first. - for _ in range(100): - command = newest_command(arr, name, before) - if command and find["id"] in (command.get("body") or {}).get(ids_field, []): - found = command - break - time.sleep(2) - expect(found, f"{label} was sent no {name} for its {kind} {find['id']} in 200 s") - for _ in range(120): - state = arr.get(f"/api/v3/command/{found['id']}") - if state.get("status") in ("completed", "failed", "aborted"): - break - time.sleep(1) - expect(state.get("status") == "completed", f"the command ended {state.get('status')}") - path = f"{arr.api}/series/{find['id']}" if is_sonarr else f"/api/v3/movie/{find['id']}" - title = arr.get(path)["title"] - expect(title == text, f"{label} holds the title {title!r}, not the lock {text!r}") - with R.check(f"push: the server's own record of it, for {label}"): - app = "sonarr" if is_sonarr else "radarr" - sent = [p for p in native.call("GET", "/admin/arr").get("pushes", []) if p.get("app") == app and p.get("outcome") == "sent"] - expect(sent, f"the page lists no {app} work sent") - expect("lock" in sent[0].get("reasons", []), f"the latest send was for {sent[0].get('reasons')}, not for a lock") - R.note(f"push: {label}: {sent[0].get('title')!r} sent as {sent[0].get('detail')!r}") + # A work of the catalogue, then the one entered by hand. + push_after_lock(R, native, arr, name, ids_field, kind, work) + push_after_lock(R, native, arr, name, ids_field, kind, None) def phase_seed(R, stack): @@ -1072,12 +1435,25 @@ def phase_seed(R, stack): with R.check("seed: the catalogue begins empty of what the libraries hold"): held = [i for i in native.items("series") if i["externalIds"].get("tvdb") in stack.series] held += [i for i in native.items("movie") if i["externalIds"].get("tmdb") in stack.films] + # The works entered by hand in the catalogue this one replaced are gone with it. + held += [i for i in (native.by_hand("series"), native.by_hand("movie")) if i is not None] expect(not held, f"the catalogue already holds {[i['title'] for i in held]}: the server was not restarted on a fresh database") sources = [(stack.sonarr5 or stack.sonarr4), stack.radarr] with R.check("seed: Sonarr and Radarr are connected"): connect(native, sources[0], stack.radarr, push=False) expected_series = [t for t in stack.series if series_by_tvdb(sources[0], t)] if sources[0] else [] expected_films = [m for m in stack.films if stack.radarr and any(x.get("tmdbId") == m for x in stack.radarr.get("/api/v3/movie"))] + # What the libraries hold under a number of the server's own: the works + # entered by hand in the catalogue the restart replaced. Nothing here can + # answer for those numbers any more, and no source ever could. + library = [("series", s) for s in (sources[0].get(f"{sources[0].api}/series") if sources[0] else [])] + library += [("movie", m) for m in (stack.radarr.get("/api/v3/movie") if stack.radarr else [])] + handed = [ + f"{w['title']} ({w['year']})" if w.get("year") else w["title"] + for kind, w in library + if OWN_FIRST <= (w.get("tvdbId" if kind == "series" else "tmdbId") or 0) <= OWN_LAST + ] + job = run = None with R.check(f"seed: Seed from Sonarr and Radarr brings {len(expected_series)} series and {len(expected_films)} films"): expect(expected_series or expected_films, "the libraries hold none of the stack's works: run the sonarr and radarr phases first") started = native.call("POST", "/tasks/arr.seed/run") @@ -1094,6 +1470,20 @@ def phase_seed(R, stack): missing = [t for t in expected_series if native.item("series", "tvdb", t) is None] missing += [m for m in expected_films if native.item("movie", "tmdb", m) is None] expect(not missing, f"not in the catalogue after the run: {missing}") + with R.check(f"seed: the {len(handed)} works entered by hand the libraries hold are not imported: reported held under no id to look them up by"): + if not handed: + raise Skipped("the libraries hold no work entered by hand: run the sonarr and radarr phases first") + if run is None: + raise Skipped("the run above did not end") + listed = native.call("GET", f"/jobs/{job}") + noted = [e for e in listed["entries"] if e.get("note") == SEED_NO_ID] + expect(sorted(e.get("title") for e in noted) == sorted(handed), f"reported held under no id: {sorted(e.get('title') for e in noted)}; held by hand: {sorted(handed)}") + expect(not any(e.get("outcome") == "imported" or e.get("mediaId") for e in noted), f"one was made here: {noted}") + # Nothing came of them but a note: the others are the run's own count, and the libraries' whole. + expect( + re.search(rf"\b{len(handed)} not found, 0 failed, of {len(library)}\b", run.get("detail") or ""), + f"the run says {run.get('detail')!r}; {len(handed)} works by hand, {len(library)} in the libraries", + ) with R.check("seed: a second run finds everything held, and fetches nothing"): run = None job = native.call("POST", "/tasks/arr.seed/run")["jobId"] @@ -1105,6 +1495,9 @@ def phase_seed(R, stack): expect(run["status"] == "succeeded", f"the run {run['status']}: {run.get('error')}") detail = run.get("detail") or "" expect(re.search(r"\b0 imported\b", detail), f"the second run says: {detail}") + # What cannot be held here is reported as often as it is asked. + if handed: + expect(re.search(rf"\b{len(handed)} not found, 0 failed\b", detail), f"the second run says: {detail}") # ── Plex ─────────────────────────────────────────────────────────────────── @@ -1357,6 +1750,87 @@ def picture(path): ) expect(shows, "the French library holds nothing") expect(shows[0].get("summary") == french["summary"], "the French library's summary is not the French one the catalogue serves") + # ── A series and a film entered by hand: matched under the catalogue's own id ── + # Libraries of their own, on the same agent — this provider alone, with no + # Plex provider beside it to guess a match by title — and folders of their + # own: the checks above count what theirs hold. The works are made in the + # catalogue before the libraries are, so that the first scan finds them. + hand = {} + with R.check("plex: a library of series and one of films entered by hand are made on the agent that is this provider alone"): + series, film = entered_by_hand(native, "series"), entered_by_hand(native, "movie") + sections = {s.get("title"): s for s in plex_get(base, "/library/sections").findall("Directory")} + wanted = 0 + for title, kind, scanner, location in ( + ("Series by hand AMS", "2", "Plex TV Series", "/media/By hand/TV Shows"), + ("Films by hand AMS", "1", "Plex Movie", "/media/By hand/Movies"), + ): + if title not in sections: + wanted += 1 + plex_post(base, "/library/sections", { + "name": title, "type": kind, "scanner": scanner, "agent": PLEX_IDENTIFIER, + "metadataAgentProviderGroupId": group_id, "language": "en-US", "location": location, + }) + sections = {s.get("title"): s for s in plex_wait_scan(base, len(sections) + wanted)} + for title in ("Series by hand AMS", "Films by hand AMS"): + expect(title in sections, f"no library {title}") + expect(sections[title].get("agent") == PLEX_IDENTIFIER, f"{title} uses the agent {sections[title].get('agent')!r}") + expect(sections[title].get("metadataAgentProviderGroupId") == group_id, f"{title} is not on the group") + hand = {"series": series, "movie": film, "series_key": sections["Series by hand AMS"].get("key"), "film_key": sections["Films by hand AMS"].get("key")} + with R.check("plex: the series entered by hand is matched as show-id-…, with the title, summary, season and episodes the catalogue holds"): + if not hand: + raise Skipped("no library for the works entered by hand") + series = hand["series"] + shows = eventually( + lambda: plex_get(base, f"/library/sections/{hand['series_key']}/all").findall("Directory"), + lambda found: len(found) == 1 and found[0].get("guid", "").startswith(PLEX_IDENTIFIER) and found[0].get("summary"), + ) + expect(len(shows) == 1, f"the library holds {len(shows)} series: {[(s.get('title'), s.get('guid')) for s in shows]}") + rating = f"show-id-{series['id']}" + ours = provider_json(f"/library/metadata/{rating}") + show = refreshed(shows[0].get("ratingKey"), "Directory", "title", ours["title"]) + expect(show.get("guid") == f"{PLEX_IDENTIFIER}://show/{rating}", f"guid {show.get('guid')!r}") + expect(show.get("title") == ours["title"] == HAND_SERIES, f"title {show.get('title')!r}, served {ours['title']!r}") + expect(show.get("year") == str(ours["year"]) == str(HAND_SERIES_YEAR), f"year {show.get('year')!r}") + expect(show.get("summary") == ours["summary"] and ours["summary"], "the summary is not the catalogue's") + seasons = plex_get(base, f"/library/metadata/{show.get('ratingKey')}/children").findall("Directory") + expect(len(seasons) == 1 and seasons[0].get("index") == "1", f"Plex lists {len(seasons)} seasons: {[s.get('title') for s in seasons]}") + season_ours = provider_json(f"/library/metadata/season-id-{series['id']}-s1") + expect(seasons[0].get("title") == season_ours["title"], f"season title {seasons[0].get('title')!r}, served {season_ours['title']!r}") + episodes = eventually( + lambda: plex_get(base, f"/library/metadata/{seasons[0].get('ratingKey')}/children").findall("Video"), + lambda found: len(found) == len(HAND_EPISODES) and all(e.get("summary") for e in found), + ) + expect(len(episodes) == len(HAND_EPISODES), f"Plex lists {len(episodes)} episodes") + typed = {str(n): title for n, title, _ in HAND_EPISODES} + for episode in episodes: + number = episode.get("index") + theirs = provider_json(f"/library/metadata/episode-id-{series['id']}-s1e{number}") + expect(episode.get("title") == theirs["title"] == typed.get(number), f"episode {number}: title {episode.get('title')!r}, served {theirs['title']!r}, typed {typed.get(number)!r}") + expect(episode.get("summary") == theirs.get("summary") and episode.get("summary"), f"episode {number}: the summary is not the catalogue's") + # Nothing elsewhere names it, so no id is listed for Plex to map it by: one made + # up for it would be taken for another work's. + named = [g.get("id") for g in show.findall("Guid")] + expect(not named, f"Plex lists the ids {named} for a series nothing elsewhere names") + R.note(f"plex: {show.get('title')}: {[e.get('title') for e in episodes]}") + with R.check("plex: the film entered by hand is matched as movie-id-…, with the title, year and summary the catalogue holds"): + if not hand: + raise Skipped("no library for the works entered by hand") + film = hand["movie"] + videos = eventually( + lambda: plex_get(base, f"/library/sections/{hand['film_key']}/all").findall("Video"), + lambda found: len(found) == 1 and found[0].get("guid", "").startswith(PLEX_IDENTIFIER) and found[0].get("summary"), + ) + expect(len(videos) == 1, f"the library holds {len(videos)} films: {[(v.get('title'), v.get('guid')) for v in videos]}") + rating = f"movie-id-{film['id']}" + ours = provider_json(f"/library/metadata/{rating}") + node = refreshed(videos[0].get("ratingKey"), "Video", "title", ours["title"]) + expect(node.get("guid") == f"{PLEX_IDENTIFIER}://movie/{rating}", f"guid {node.get('guid')!r}") + expect(node.get("title") == ours["title"] == HAND_FILM, f"title {node.get('title')!r}, served {ours['title']!r}") + expect(node.get("year") == str(ours["year"]) == str(HAND_FILM_YEAR), f"year {node.get('year')!r}") + expect(node.get("summary") == ours["summary"] and ours["summary"], "the summary is not the catalogue's") + named = [g.get("id") for g in node.findall("Guid")] + expect(not named, f"Plex lists the ids {named} for a film nothing elsewhere names") + R.note(f"plex: {node.get('title')} ({node.get('year')})") with R.check("plex: this server counted the calls Plex made, and refused none"): apis = {a["api"]: a for a in native.call("GET", "/admin/access")["apis"]} served = apis["plex"]["served"] - served_before diff --git a/scripts/e2e-stack.sh b/scripts/e2e-stack.sh index 1aac6d1..3d25150 100755 --- a/scripts/e2e-stack.sh +++ b/scripts/e2e-stack.sh @@ -26,22 +26,43 @@ # chosen at the add, the seasons' titles; a title locked # through the native API appears after a refresh; the # catalogue's scene-mapping titles take a release to its -# series +# series; and a series entered by hand, one nothing +# elsewhere names — made through the native API, with an +# alternative title and a season of two episodes: found by +# its title and by tvdb: under a number of the server's +# own (from 1 000 000 000, made of its catalogue id), a +# number no work was handed finding nothing; added, its +# season and episodes as served; a locked title after a +# refresh; its alternative title in the scene mapping, a +# release named by it taken for it; on Sonarr 5 its +# metadataSource, local # radarr a search by text and by tmdb:; two films added (one in a # collection); title, year, runtime, studio and # classification as served; the collection; the -# recommendations; a locked title after a refresh +# recommendations; a locked title after a refresh; and a +# film entered by hand, one with no TMDB id: found by its +# title and by tmdb: under such a number; added, as +# served; named among the films changed once edited; a +# locked title after a refresh # push the server connected to each of them with its API key # (PUT /api/v1/admin/arr), a lock made here sends the # client a RefreshSeries / RefreshMovie, found in its own -# history of commands +# history of commands — for the series and the film +# entered by hand too, which the client holds under their +# numbers # seed the server restarted on a new, empty catalogue, then # Seed from Sonarr and Radarr: what their libraries hold -# arrives, and a second run fetches nothing +# arrives, and a second run fetches nothing; the works +# entered by hand that they hold under their numbers are +# reported held under no id to look them up by, none made # plex the provider added to Plex over its own API, an agent # made of it, a library of films, one of series, one of # series in French; the scan; the film, the series, its -# season and its episodes as the catalogue has them +# season and its episodes as the catalogue has them; and a +# series and a film entered by hand, in libraries of their +# own on the same agent, matched as show-id-… and +# movie-id-…, with the catalogue's title, summary, season +# and episodes, and no id elsewhere # plexkey the server restarted with AMS_PLEX_AUTH=apikey: /plex # refused without a key; Plex given /plex/ instead, # a lock reaching it there, a key never issued answered @@ -278,6 +299,16 @@ if has_phase plex; then make_video "$WORK/media/Movies/The Matrix (1999)/The Matrix (1999).mp4" make_video "$WORK/media/TV Shows/Breaking Bad (2008)/Season 01/Breaking Bad (2008) - S01E01 - Pilot.mp4" make_video "$WORK/media/TV Shows/Breaking Bad (2008)/Season 01/Breaking Bad (2008) - S01E02 - Cat's in the Bag....mp4" + # And a series of two episodes and a film entered by hand, under the titles + # scripts/e2e-stack.py makes them with (HAND_SERIES, HAND_FILM, HAND_EPISODES). + # Their own folders, for libraries of their own: the checks of the real works + # count what theirs hold. + hand_series="$WORK/media/By hand/TV Shows/Quenzabar Tides (2024)/Season 01" + hand_film="$WORK/media/By hand/Movies/Brindlemoss Without Source (2019)" + mkdir -p "$hand_series" "$hand_film" + make_video "$hand_series/Quenzabar Tides (2024) - S01E01 - The Tide Goes Out.mp4" + make_video "$hand_series/Quenzabar Tides (2024) - S01E02 - The Tide Comes Back.mp4" + make_video "$hand_film/Brindlemoss Without Source (2019).mp4" fi step "the containers start (${SERVICES# })" diff --git a/src/api/by_hand_tests.rs b/src/api/by_hand_tests.rs new file mode 100644 index 0000000..f0c33e6 --- /dev/null +++ b/src/api/by_hand_tests.rs @@ -0,0 +1,393 @@ +//! A work entered by hand, as each client finds it: made through the +//! administration's own routes, then searched for and asked for by Sonarr — +//! both editions — Radarr and Plex, through the routes and guards they call. +//! The server's whole surface, on a database of its own, every source +//! switched off: a work entered by hand is the catalogue's alone. + +use std::net::SocketAddr; + +use axum::{ + Router, + body::Body, + extract::ConnectInfo, + http::{Request, StatusCode}, +}; +use serde_json::{Value, json}; +use tower::ServiceExt as _; + +use crate::{ + config, + service::ids::{self, OWN_OFFSET}, + state::AppState, +}; + +/// The whole server with authentication switched off — the administration +/// open to whoever calls, as `AMS_AUTH_DISABLED` leaves it — and no source +/// to ask. +async fn server() -> AppState { + let mut config = config::Config::from_env().expect("a configuration"); + config.mode = config::Mode::Single; + config.database = crate::db::testing::database().await; + config.security.bootstrap_admin = None; + config.security.auth_disabled = true; + config.cache.redis_url = None; + config.media.storage = config::MediaStorage::Off; + config.clients = None; + config.tmdb.api_key = None; + config.tmdb.language = "en-US".into(); + config.tvdb.api_key = None; + config.tvdb.enabled = false; + config.skyhook.enrich = false; + config.skyhook.fallback = false; + config.radarr_metadata.enrich = false; + config.radarr_metadata.fallback = false; + config.tvmaze.enabled = false; + config.anilist.enabled = false; + config.mal.enabled = false; + config.server.public_url = Some("http://ams.lan:8479".into()); + + AppState::bootstrap(config).await.expect("a server") +} + +/// Every surface, under its own guard, as the server mounts them. +fn surfaces(state: &AppState) -> Router { + let (router, _) = super::build(state.clone()); + router.with_state(state.clone()) +} + +/// One request from this very machine, with a JSON body when there is one; +/// the answer's status and its body, read as JSON. +async fn call( + router: &Router, + method: &str, + uri: &str, + body: Option, +) -> (StatusCode, Value) { + let mut request = Request::builder().method(method).uri(uri); + if body.is_some() { + request = request.header("content-type", "application/json"); + } + let mut request = request + .body(body.map_or_else(Body::empty, |body| Body::from(body.to_string()))) + .expect("a request"); + request + .extensions_mut() + .insert(ConnectInfo(SocketAddr::from(([127, 0, 0, 1], 40_000)))); + let response = router.clone().oneshot(request).await.expect("an answer"); + let status = response.status(); + let bytes = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .expect("a body"); + ( + status, + serde_json::from_slice(&bytes).unwrap_or(Value::Null), + ) +} + +async fn get(router: &Router, uri: &str) -> (StatusCode, Value) { + call(router, "GET", uri, None).await +} + +/// Made as the editor's *New entry* makes it, with a season of two episodes +/// for a series: its catalogue id. +async fn entered_by_hand(router: &Router, work: Value) -> String { + let series = work["kind"] == "series"; + let (status, made) = call(router, "POST", "/api/v1/items", Some(work)).await; + assert_eq!(status, StatusCode::CREATED, "{made}"); + assert_eq!(made["isManual"], true); + let id = made["id"].as_str().expect("an id").to_string(); + if series { + let (status, _) = call( + router, + "POST", + &format!("/api/v1/items/{id}/seasons"), + Some(json!({ "seasonNumber": 1 })), + ) + .await; + assert_eq!(status, StatusCode::CREATED); + for number in [1, 2] { + let (status, _) = call( + router, + "POST", + &format!("/api/v1/items/{id}/episodes"), + Some(json!({ + "seasonNumber": 1, + "episodeNumber": number, + "title": format!("Typed in, {number}"), + "airDate": format!("2024-0{number}-10"), + })), + ) + .await; + assert_eq!(status, StatusCode::CREATED); + } + } + id +} + +fn vicineko() -> Value { + json!({ + "kind": "series", + "title": "Vicineko", + "year": 2024, + "overview": "A series no source lists, entered by hand.", + "status": "continuing", + "firstAired": "2024-01-10", + "network": "Hand TV", + "genres": ["Animation"], + }) +} + +/// An id of the range that no work entered by hand is handed: the range's +/// first, unless the work at hand happens to be handed it. +fn named_by_none(handed: i64) -> i64 { + if handed == OWN_OFFSET { + OWN_OFFSET + 1 + } else { + OWN_OFFSET + } +} + +/// The user's case: a series entered by hand — no TheTVDB, TMDB or Fankai id +/// — was left out of Sonarr's search, since Sonarr keys a series by its +/// TheTVDB id and it had none. It is listed now under a number made of its +/// catalogue id, the same on both editions and at every answer, which Sonarr +/// adds it by and refreshes it by. +#[tokio::test] +async fn sonarr_finds_adds_and_refreshes_a_series_entered_by_hand() { + let state = server().await; + let router = surfaces(&state); + let id = entered_by_hand(&router, vicineko()).await; + let handed = ids::own(&id); + assert!(ids::is_own(handed), "{handed}"); + + // Sonarr v4's search, as it sends it: lower-cased. + let (status, found) = get(&router, "/v1/tvdb/search/en?term=vicineko").await; + assert_eq!(status, StatusCode::OK, "{found}"); + let found = found.as_array().expect("a list"); + assert_eq!(found.len(), 1, "{found:?}"); + assert_eq!(found[0]["tvdbId"], handed); + assert_eq!(found[0]["title"], "Vicineko"); + + // Its add, and every refresh after: the series by that id. + let (status, show) = get(&router, &format!("/v1/tvdb/shows/en/{handed}")).await; + assert_eq!(status, StatusCode::OK, "{show}"); + assert_eq!(show["tvdbId"], handed); + assert_eq!(show["title"], "Vicineko"); + assert_eq!(show["firstAired"], "2024-01-10"); + let episodes = show["episodes"].as_array().expect("episodes"); + assert_eq!(episodes.len(), 2); + assert!(episodes.iter().all(|e| e["tvdbShowId"] == handed)); + assert_eq!(episodes[0]["title"], "Typed in, 1"); + + // Sonarr v5's, the same series under the same id. + let (status, found) = get(&router, "/v2/tvdb/search?term=vicineko&language=eng").await; + assert_eq!(status, StatusCode::OK, "{found}"); + assert_eq!(found[0]["tvdbId"], handed); + assert_eq!(found[0]["metadataSource"], "local"); + let (status, show) = get(&router, &format!("/v2/tvdb/shows/eng/{handed}/official")).await; + assert_eq!(status, StatusCode::OK, "{show}"); + assert_eq!(show["tvdbId"], handed); + assert_eq!(show["episodes"].as_array().map(Vec::len), Some(2)); + + // `tvdb:` and the id, as Sonarr resolves such a term itself. + let (status, found) = get(&router, &format!("/v1/tvdb/search/en?term=tvdb:{handed}")).await; + assert_eq!(status, StatusCode::OK); + assert_eq!(found[0]["tvdbId"], handed); + + // An edit reaches the next refresh, under the same id. + let (status, _) = call( + &router, + "PUT", + &format!("/api/v1/items/{id}/overrides"), + Some(json!({ "field": "title", "value": "Vicineko, au propre" })), + ) + .await; + assert!(status.is_success(), "{status}"); + let (_, show) = get(&router, &format!("/v1/tvdb/shows/en/{handed}")).await; + assert_eq!(show["title"], "Vicineko, au propre"); + assert_eq!(show["tvdbId"], handed); + + // Switched off, it is still Sonarr's: a 404 would have Sonarr take it + // for deleted. + let (status, _) = call( + &router, + "PATCH", + &format!("/api/v1/items/{id}"), + Some(json!({ "isEnabled": false })), + ) + .await; + assert!(status.is_success(), "{status}"); + let (status, _) = get(&router, &format!("/v1/tvdb/shows/en/{handed}")).await; + assert_eq!(status, StatusCode::OK); + + // A number of the range no work is handed names nothing — at once, with + // no source to wait for — and neither does the work's once it is gone. + let (status, _) = get( + &router, + &format!("/v1/tvdb/shows/en/{}", named_by_none(handed)), + ) + .await; + assert_eq!(status, StatusCode::NOT_FOUND); + let (status, _) = call(&router, "DELETE", &format!("/api/v1/items/{id}"), None).await; + assert_eq!(status, StatusCode::NO_CONTENT); + let (status, _) = get(&router, &format!("/v1/tvdb/shows/en/{handed}")).await; + assert_eq!(status, StatusCode::NOT_FOUND); + let (status, _) = get(&router, &format!("/v2/tvdb/shows/eng/{handed}/official")).await; + assert_eq!(status, StatusCode::NOT_FOUND); +} + +/// A film entered by hand reached Radarr's search with a TMDB id of nought, +/// which Radarr refuses to add, and every such film under the same one. It +/// is listed under a number of its own, which Radarr adds, refreshes and is +/// told of a change by. +#[tokio::test] +async fn radarr_finds_adds_and_refreshes_a_film_entered_by_hand() { + let state = server().await; + let router = surfaces(&state); + let id = entered_by_hand( + &router, + json!({ + "kind": "movie", + "title": "Sans Source", + "year": 2031, + "overview": "A film no source lists, entered by hand.", + "status": "released", + "inCinemas": "2031-05-01", + "studio": "Hand Films", + }), + ) + .await; + let handed = ids::own(&id); + + let (status, found) = get(&router, "/v1/search?q=sans%20source&year=").await; + assert_eq!(status, StatusCode::OK, "{found}"); + let found = found.as_array().expect("a list"); + assert_eq!(found.len(), 1, "{found:?}"); + assert_eq!(found[0]["TmdbId"], handed); + assert_eq!(found[0]["Title"], "Sans Source"); + + let (status, film) = get(&router, &format!("/v1/movie/{handed}")).await; + assert_eq!(status, StatusCode::OK, "{film}"); + assert_eq!(film["TmdbId"], handed); + assert_eq!(film["Title"], "Sans Source"); + assert_eq!(film["Studio"], "Hand Films"); + + let (status, films) = call(&router, "POST", "/v1/movie/bulk", Some(json!([handed]))).await; + assert_eq!(status, StatusCode::OK, "{films}"); + assert_eq!(films[0]["TmdbId"], handed); + + let (status, found) = get(&router, &format!("/v1/search?q=tmdb:{handed}&year=")).await; + assert_eq!(status, StatusCode::OK); + assert_eq!(found[0]["TmdbId"], handed); + + // Named among the films changed, so that Radarr's scheduled refresh + // reads it again. + let (status, changed) = get(&router, "/v1/movie/changed?since=2020-01-01").await; + assert_eq!(status, StatusCode::OK, "{changed}"); + assert!( + changed + .as_array() + .expect("ids") + .iter() + .any(|changed| *changed == handed), + "{changed}" + ); + + let (status, _) = get(&router, &format!("/v1/movie/{}", named_by_none(handed))).await; + assert_eq!(status, StatusCode::NOT_FOUND); +} + +/// Plex is given a work entered by hand under the catalogue's own id — +/// `show-id-…`, `movie-id-…` — which a real Plex Media Server takes: a match +/// by its title, then the record, its seasons and episodes, and its +/// pictures. No `Guid` names it elsewhere, since nothing does. +#[tokio::test] +async fn plex_matches_a_work_entered_by_hand_and_is_given_its_record_and_pictures() { + let state = server().await; + let router = surfaces(&state); + let id = entered_by_hand(&router, vicineko()).await; + let (status, _) = call( + &router, + "POST", + &format!("/api/v1/items/{id}/images"), + Some(json!({ + "coverType": "poster", + "url": "https://images.example/vicineko.jpg", + "sortOrder": 0, + })), + ) + .await; + assert_eq!(status, StatusCode::CREATED); + let key = format!("show-id-{id}"); + + for manual in [false, true] { + let (status, matched) = call( + &router, + "POST", + "/plex/library/metadata/matches", + Some(json!({ "type": 2, "title": "Vicineko", "year": 2024, "manual": manual })), + ) + .await; + assert_eq!(status, StatusCode::OK, "{matched}"); + let first = &matched["MediaContainer"]["Metadata"][0]; + assert_eq!(first["ratingKey"], key.as_str(), "{matched}"); + assert_eq!( + first["guid"], + format!("tv.plex.agents.custom.arrmetadataserver://show/{key}") + ); + assert!(first.get("Guid").is_none_or(|g| g == &json!([])), "{first}"); + } + + let (status, show) = get(&router, &format!("/plex/library/metadata/{key}")).await; + assert_eq!(status, StatusCode::OK, "{show}"); + let show = &show["MediaContainer"]["Metadata"][0]; + assert_eq!(show["title"], "Vicineko"); + assert_eq!( + show["summary"], + "A series no source lists, entered by hand." + ); + assert_eq!(show["thumb"], "https://images.example/vicineko.jpg"); + assert_eq!(show["SeasonType"][0]["source"], "local"); + + let (status, images) = get(&router, &format!("/plex/library/metadata/{key}/images")).await; + assert_eq!(status, StatusCode::OK); + assert_eq!(images["MediaContainer"]["Image"][0]["type"], "coverPoster"); + + let (status, seasons) = get(&router, &format!("/plex/library/metadata/{key}/children")).await; + assert_eq!(status, StatusCode::OK); + let season = &seasons["MediaContainer"]["Metadata"][0]; + assert_eq!(season["ratingKey"], format!("season-id-{id}-s1")); + let (status, episodes) = get( + &router, + &format!("/plex/library/metadata/season-id-{id}-s1/children"), + ) + .await; + assert_eq!(status, StatusCode::OK); + let episodes = episodes["MediaContainer"]["Metadata"] + .as_array() + .expect("episodes"); + assert_eq!(episodes.len(), 2); + assert_eq!(episodes[1]["ratingKey"], format!("episode-id-{id}-s1e2")); + assert_eq!(episodes[1]["title"], "Typed in, 2"); + + // A film, the same way. + let film = entered_by_hand( + &router, + json!({ "kind": "movie", "title": "Sans Source", "year": 2031 }), + ) + .await; + let (status, matched) = call( + &router, + "POST", + "/plex/library/metadata/matches", + Some(json!({ "type": 1, "title": "Sans Source", "year": 2031 })), + ) + .await; + assert_eq!(status, StatusCode::OK, "{matched}"); + assert_eq!( + matched["MediaContainer"]["Metadata"][0]["ratingKey"], + format!("movie-id-{film}") + ); + let (status, _) = get(&router, &format!("/plex/library/metadata/movie-id-{film}")).await; + assert_eq!(status, StatusCode::OK); +} diff --git a/src/api/mod.rs b/src/api/mod.rs index 09e1826..463a4d2 100644 --- a/src/api/mod.rs +++ b/src/api/mod.rs @@ -8,6 +8,8 @@ pub mod anilist; pub mod audit; +#[cfg(test)] +mod by_hand_tests; pub mod cursor; pub mod extract; pub mod merged; diff --git a/src/api/native/items.rs b/src/api/native/items.rs index a911c01..475758c 100644 --- a/src/api/native/items.rs +++ b/src/api/native/items.rs @@ -771,6 +771,7 @@ async fn create( item.external_ids = request.external_ids; item.is_manual = true; item.slug = unique_slug(&state, kind, &title, request.year).await?; + item.id = id_of_its_own(&state, kind).await?; // An identifier another work goes by is refused where it is written, in // the work's own transaction: checked beforehand, two entries made at once @@ -1649,6 +1650,25 @@ fn checked_by_hand(mut request: CreateRequest, kind: MediaKind) -> AppResult AppResult { + let taken = service::own_holders(state, kind).await?; + for _ in 0..16 { + let id = crate::db::new_id(); + if !taken.contains_key(&service::ids::own(&id)) { + return Ok(id); + } + } + Err(AppError::Internal(anyhow::anyhow!( + "no id of a number of its own was drawn for a work entered by hand" + ))) +} + /// The statuses a work may be given by hand, as its clients spell them: /// Sonarr's for a series, Radarr's for a film. fn statuses(kind: MediaKind) -> &'static [&'static str] { diff --git a/src/api/native/media.rs b/src/api/native/media.rs index 6407090..f43843a 100644 --- a/src/api/native/media.rs +++ b/src/api/native/media.rs @@ -48,6 +48,7 @@ pub fn router() -> OpenApiRouter { .routes(routes!(retry)) .routes(routes!(retry_all)) .routes(routes!(reset)) + .routes(routes!(delete_variants)) .routes(routes!(work_media)) .routes(routes!(by_work)) .routes(routes!(forget)) @@ -102,6 +103,9 @@ pub struct MediaStatus { /// The pictures kept that this version has not prepared — no /// placeholder, no first copy — which the task `media.variants` does. pub unprepared: i64, + /// Whether WebP copies are made and given to the pages (`media.webp`); + /// off, the pages are given the pictures themselves. + pub webp: bool, /// Whether the night's task makes AVIF copies (`media.avif`). pub avif: bool, /// Whether a fetch or a sweep is under way. @@ -109,6 +113,8 @@ pub struct MediaStatus { pub sweeping: bool, /// Whether the pictures are being prepared, or their AVIF copies made. pub preparing: bool, + /// Whether every copy of a format is being deleted. + pub purging: bool, } /// How the media store stands. @@ -136,10 +142,12 @@ async fn status( variants: repo::asset::variant_counts(&state.db).await?, unprepared: repo::asset::unprepared_count(&state.db, crate::media::variants::PREPARED) .await?, + webp: state.flag("media.webp", false), avif: state.flag("media.avif", false), fetching: crate::media::worker::is_storing(), sweeping: crate::media::worker::is_sweeping(), preparing: crate::jobs::media::is_busy(), + purging: crate::jobs::media::is_purging(), })) } @@ -267,6 +275,110 @@ async fn reset( Ok(Json(Forgotten { forgotten })) } +/// A format the copies of the pictures are written as, whose copies can be +/// deleted once they are no longer made. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, ToSchema)] +#[serde(rename_all = "lowercase")] +// Named explicitly: utoipa keys schemas on the leaf name alone. +#[schema(as = MediaVariantFormat)] +pub enum VariantFormat { + /// The copies the pages are given while `media.webp` is on. + Webp, + /// The night's copies, made while `media.avif` is on. + Avif, +} + +impl VariantFormat { + fn made_as(self) -> crate::media::variants::Format { + match self { + Self::Webp => crate::media::variants::Format::Webp, + Self::Avif => crate::media::variants::Format::Avif, + } + } + + fn name(self) -> &'static str { + match self { + Self::Webp => "WebP", + Self::Avif => "AVIF", + } + } +} + +/// What there was of a format's copies as their deletion began. +#[derive(Serialize, ToSchema)] +#[serde(rename_all = "camelCase")] +pub struct VariantsDeleted { + pub format: VariantFormat, + /// How many copies there were. + pub copies: i64, + /// What they weighed, in bytes. + pub bytes: i64, +} + +/// Delete every copy of a format: the WebP copies once `media.webp` is off, +/// the AVIF copies once `media.avif` is — the space they took, back. +/// +/// Answered at once, with what there was, and done in the background: each +/// copy's file taken from the store, then its row; the status says +/// `purging` until it is over. The pictures themselves are not touched, +/// nor the squares Plex is given. Turned on again meanwhile, the setting +/// stops it where it is. +#[utoipa::path( + delete, path = "/media/variants/{format}", tag = TAG, + operation_id = "media_delete_variants", + params(("format" = VariantFormat, Path, description = "The format whose copies are deleted")), + responses( + (status = 202, body = VariantsDeleted), + (status = 400, description = "Not a format whose copies are deleted: `webp` or `avif`"), + (status = 403, description = "The caller is not an administrator"), + (status = 409, description = "The format's copies are still made — its setting is on — or the AVIF copies are being made, or copies are being deleted already"), + (status = 503, description = "No media store is configured"), + ), +)] +async fn delete_variants( + State(state): State, + Extension(identity): Extension, + ip: ClientIp, + Path(format): Path, +) -> AppResult<(StatusCode, Json)> { + identity.require_admin()?; + // Read here rather than by the extractor, so that another name is + // refused in this server's words. + let format = match format.as_str() { + "webp" => VariantFormat::Webp, + "avif" => VariantFormat::Avif, + other => { + return Err(AppError::BadRequest(format!( + "{other} is no format whose copies are deleted: webp or avif" + ))); + } + }; + let there_was = crate::jobs::media::purge_now(&state, format.made_as()).await?; + audit::record( + &state, + Event { + identity: Some(&identity), + ip: &ip, + action: Action::MediaRemoved, + target: None, + detail: Some(&format!( + "every {} copy deleted ({} files)", + format.name(), + there_was.copies + )), + }, + ) + .await; + Ok(( + StatusCode::ACCEPTED, + Json(VariantsDeleted { + format, + copies: there_was.copies, + bytes: there_was.bytes, + }), + )) +} + /// A work's media, and whether anything is kept at all. #[derive(Serialize, ToSchema)] #[serde(rename_all = "camelCase")] @@ -710,9 +822,17 @@ async fn upload( Place::Picture { cover_type, .. } => Some(*cover_type), Place::Still { .. } | Place::Theme => None, }; - let kept = crate::media::worker::keep(store, key, sha256, bytes, &inspected, cover) - .await - .map_err(|e| AppError::Internal(e.context("could not keep the upload")))?; + let kept = crate::media::worker::keep( + store, + key, + sha256, + bytes, + &inspected, + cover, + state.flag("media.webp", false), + ) + .await + .map_err(|e| AppError::Internal(e.context("could not keep the upload")))?; let by = identity.label(); repo::asset::insert_upload( &state.db, @@ -1142,6 +1262,160 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + + async fn set(state: &AppState, key: &str, value: &str) { + state + .settings + .set(crate::settings::Scope::Server, "", key, value, None) + .await + .expect("set"); + } + + /// Once the copies being deleted in the background are. + async fn deleted_at_last() { + for _ in 0..3000 { + if !crate::jobs::media::is_purging() { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + panic!("the copies are still being deleted"); + } + + /// Every copy of a format is deleted once its setting is off — each + /// file, then the rows — in the background, and nothing else: not the + /// other format's, not Plex's square, not the picture. Refused for a + /// format that is none, to anybody but an administrator, and while the + /// format's copies are still made. + #[tokio::test] + async fn every_copy_of_a_format_is_deleted_once_it_is_off() { + use crate::db::repo::asset::{Thumb, Variant}; + + let dir = crate::testing::scratch_path("media-purge"); + let state = server(&dir).await; + let store = state.media.store().unwrap().clone(); + + // A picture kept, with copies of every kind beside it — one WebP + // noted whose file is gone already. + let sha = format!("{}c0ff", "c0ffee".repeat(10)); + let key = format!("{sha}.png"); + store + .put(&key, Bytes::from_static(b"picture"), "image/png") + .await + .unwrap(); + state.media.remember( + "https://images.example/kept.png", + &key, + Thumb::None, + "image/png", + ); + for (made, format, on_disk) in [ + ("240.webp", "webp", true), + ("480.webp", "webp", true), + ("960.webp", "webp", false), + ("480.avif", "avif", true), + ("square.jpg", "jpeg", true), + ] { + let copy = format!("{sha}-{made}"); + if on_disk { + store + .put(©, Bytes::from_static(b"copy"), "image/webp") + .await + .unwrap(); + } + repo::asset::add_variant( + &state.db, + &Variant { + key: copy, + stem: sha.clone(), + width: 480, + height: 720, + format: format.into(), + bytes: 4, + }, + ) + .await + .unwrap(); + } + state.media.add_avif(&sha, 480); + let on_disk = |made: &str| { + let store = store.clone(); + let copy = format!("{sha}-{made}"); + async move { store.exists(©).await.unwrap() } + }; + let delete = |format: &str, identity: Identity| { + delete_variants( + State(state.clone()), + Extension(identity), + ClientIp::NONE, + Path(format.to_string()), + ) + }; + + assert!(matches!( + delete("jpeg", Identity::Anonymous).await, + Err(AppError::BadRequest(_)) + )); + assert!(matches!( + delete("png", Identity::Anonymous).await, + Err(AppError::BadRequest(_)) + )); + assert!(matches!( + delete("webp", Identity::Visitor).await, + Err(AppError::Forbidden) + )); + + // Still made and given to the pages: refused, and nothing deleted. + set(&state, "media.webp", "true").await; + assert!(matches!( + delete("webp", Identity::Anonymous).await, + Err(AppError::Conflict(_)) + )); + assert_eq!( + repo::asset::variant_counts(&state.db).await.unwrap().webp, + 3 + ); + assert!(on_disk("240.webp").await); + + set(&state, "media.webp", "false").await; + let (answered, Json(deleted)) = + delete("webp", Identity::Anonymous).await.expect("accepted"); + assert_eq!(answered, StatusCode::ACCEPTED); + assert_eq!( + (deleted.format, deleted.copies, deleted.bytes), + (VariantFormat::Webp, 3, 12), + "what there was" + ); + deleted_at_last().await; + let counts = repo::asset::variant_counts(&state.db).await.unwrap(); + assert_eq!((counts.webp, counts.avif, counts.jpeg), (0, 1, 1)); + assert!(!on_disk("240.webp").await && !on_disk("480.webp").await); + assert!(on_disk("480.avif").await && on_disk("square.jpg").await); + assert!(store.exists(&key).await.unwrap(), "the picture is kept"); + assert_eq!(state.media.keyed(&key).unwrap().look.avif, [480]); + + // The AVIF copies too, off as they are until somebody sets them: + // gone, and no longer offered to a page. + let (_, Json(deleted)) = delete("avif", Identity::Anonymous).await.expect("accepted"); + assert_eq!( + (deleted.format, deleted.copies, deleted.bytes), + (VariantFormat::Avif, 1, 4) + ); + deleted_at_last().await; + let counts = repo::asset::variant_counts(&state.db).await.unwrap(); + assert_eq!((counts.webp, counts.avif, counts.jpeg), (0, 0, 1)); + assert!(!on_disk("480.avif").await); + assert!(on_disk("square.jpg").await, "Plex's square is kept"); + assert!(state.media.keyed(&key).unwrap().look.avif.is_empty()); + + let Json(now) = status(State(state.clone()), Extension(Identity::Anonymous)) + .await + .expect("the status"); + assert!(!now.webp && !now.avif && !now.purging); + assert_eq!((now.variants.webp, now.variants.avif), (0, 0)); + + let _ = std::fs::remove_dir_all(&dir); + } } #[cfg(test)] diff --git a/src/api/native/signup.rs b/src/api/native/signup.rs index 733b7be..74c0e92 100644 --- a/src/api/native/signup.rs +++ b/src/api/native/signup.rs @@ -257,6 +257,10 @@ pub struct AuthOptions { /// The identity provider's button, when signing in through one is on. #[serde(skip_serializing_if = "Option::is_none")] pub oidc: Option, + /// Whether the pages are given WebP copies of the pictures this server + /// keeps (`media.webp`), rather than the pictures themselves: said here, + /// where every page reads it before it draws one. + pub webp: bool, } #[derive(Serialize, ToSchema)] @@ -288,6 +292,7 @@ async fn options(State(state): State) -> Json { oidc: state.oidc_provider().map(|_| OidcButton { label: state.oidc_button(), }), + webp: state.flag("media.webp", false), }) } @@ -641,6 +646,29 @@ mod tests { assert!(take_place_in(counts, None, now, 5, 3).is_some()); } + /// The pages are told, before they draw a picture, whether to ask for + /// its WebP copies: not until an administrator turns them on. + #[tokio::test] + async fn the_pages_are_told_whether_they_are_given_webp_copies() { + let state = crate::api::native::testing::server().await; + let Json(before) = options(State(state.clone())).await; + assert!(!before.webp); + + state + .settings + .set( + crate::settings::Scope::Server, + "", + "media.webp", + "true", + None, + ) + .await + .expect("the WebP copies on"); + let Json(after) = options(State(state)).await; + assert!(after.webp); + } + #[test] fn an_ipv6_network_counts_as_one_address() { let a: IpAddr = "2001:db8:1:2::1".parse().unwrap(); diff --git a/src/api/radarr.rs b/src/api/radarr.rs index 5d865fc..5d93d86 100644 --- a/src/api/radarr.rs +++ b/src/api/radarr.rs @@ -97,18 +97,32 @@ fn served_with( items .iter() .enumerate() - .map(|(at, item)| match beside.get(at) { - Some(beside) => from_item_beside(item, beside), - None => from_item(item), + .map(|(at, item)| { + let mut resource = match beside.get(at) { + Some(beside) => from_item_beside(item, beside), + None => from_item(item), + }; + // A film entered by hand that TMDB does not name, under the id + // Radarr is handed for it in TMDB's stead. + if resource.tmdb_id == 0 + && let Some(id) = movie::client_id(item) + { + resource.tmdb_id = id; + } + resource }) .collect() } /// A movie by TMDB id, in Radarr's format. +/// +/// The id may also be the one a film entered by hand that TMDB does not name +/// is handed in TMDB's stead, from 1 000 000 000: such a film is answered +/// from the catalogue alone. #[utoipa::path( get, path = "/v1/movie/{tmdb_id}", tag = TAG, operation_id = "radarr_by_tmdb", - params(("tmdb_id" = i64, Path, description = "TMDB movie id")), + params(("tmdb_id" = i64, Path, description = "TMDB movie id, or the one a film entered by hand is handed")), responses( (status = 200, body = MovieResource), (status = 403, description = "The caller's address is not in the allowlist"), @@ -184,6 +198,16 @@ async fn search( let term = query.q.unwrap_or_default(); let items = movie::search(&state, &term, query.year).await?; + // A result Radarr cannot add is worse than none: it would show in the + // list and fail on add, Radarr refusing a film of TMDB id nought. A film + // entered by hand is listed under the number it is handed instead, when + // that number leads back to it (see `service::addressed`). + let ids = movie::client_ids(&state, &items).await?; + let items = items + .into_iter() + .zip(ids) + .filter_map(|(item, id)| id.map(|_| item)) + .collect(); Ok(Json(served(&state, &identity, items))) } diff --git a/src/api/sonarr.rs b/src/api/sonarr.rs index 1f7fc1e..65e2710 100644 --- a/src/api/sonarr.rs +++ b/src/api/sonarr.rs @@ -70,8 +70,9 @@ struct SearchQueryV2 { /// A series by TVDB id, in Skyhook's format. /// /// The id may also be one this server synthesised for a work TMDB indexes and -/// TVDB does not; the response echoes back whichever id was asked for, because -/// Sonarr has already stored it. +/// TVDB does not, or for a series entered by hand that nothing elsewhere +/// names; the response echoes back whichever id was asked for, because Sonarr +/// has already stored it. /// /// A series the catalogue holds is answered from the catalogue, as it was, /// when the sources do not answer. An id nobody holds is a 404 only when every @@ -161,13 +162,14 @@ async fn search( let term = query.term.unwrap_or_default(); let mut items = series::search(&state, &term).await?; + let ids = series::client_ids(&state, &items).await?; // Only the work's own title here: fetching a season of episode text for // each of ten search results would turn one search into dozens of calls. - for item in &mut items { + for (item, id) in items.iter_mut().zip(&ids) { language::apply_shallow(&state, item, &language); - if let Some(id) = series::client_id(item) { - name_for_sonarr(&state, item, id).await; + if let Some(id) = id { + name_for_sonarr(&state, item, *id).await; } state.media.for_clients(item); } @@ -176,7 +178,8 @@ async fn search( // the list and then fail on add. let shows = items .iter() - .filter_map(|item| series::client_id(item).map(|id| from_item(item, id, &language))) + .zip(ids) + .filter_map(|(item, id)| id.map(|id| from_item(item, id, &language))) .collect(); Ok(Json(shows)) @@ -289,12 +292,13 @@ async fn search_v2( let callers = state.language(identity.client_id(), identity.peer_id()); let mut items = series::search(&state, &term).await?; + let ids = series::client_ids(&state, &items).await?; let mut shows = Vec::with_capacity(items.len()); - for item in &mut items { + for (item, id) in items.iter_mut().zip(ids) { // A result Sonarr cannot address is worse than no result: it would // show in the list and then fail on add. - let Some(id) = series::client_id(item) else { + let Some(id) = id else { continue; }; name_for_sonarr(&state, item, id).await; diff --git a/src/arr/push.rs b/src/arr/push.rs index 75b0f62..f8582b3 100644 --- a/src/arr/push.rs +++ b/src/arr/push.rs @@ -5,8 +5,10 @@ //! to reach them — or for somebody to press their refresh button. With //! `arr.pushRefresh` on, the change is told instead: the work is found there //! by the id it is served under (TheTVDB's, or the synthetic one a TMDB-only -//! series or a Fan-Kai is handed, for Sonarr; TMDB's for Radarr) and asked to -//! be read again, `RefreshSeries` or `RefreshMovie`. +//! series or a Fan-Kai is handed, for Sonarr; TMDB's for Radarr; for a work +//! entered by hand that nothing elsewhere names, the number made of its +//! catalogue id, for either) and asked to be read again, `RefreshSeries` or +//! `RefreshMovie`. //! //! What changes a work for its clients: a lock put on it, on one of its //! seasons or its episodes, changed or taken off; an edit of its seasons or @@ -365,7 +367,7 @@ pub async fn send(state: &AppState, due: Vec<(String, Vec)>) { }; let client_id = match item.kind { MediaKind::Series => service::series::client_id(&item), - MediaKind::Movie => item.external_ids.tmdb, + MediaKind::Movie => service::movie::client_id(&item), }; by_app.entry(App::of(item.kind)).or_default().push(Target { id, @@ -788,6 +790,43 @@ mod tests { assert_eq!(seen.lock().len(), before); } + /// A film entered by hand that TMDB does not name is looked for in + /// Radarr under the number it is served under there, rather than said to + /// have no id Radarr could know it by. + #[tokio::test] + async fn a_film_entered_by_hand_is_looked_for_under_the_number_it_is_handed() { + let _turn = TURN.lock().await; + let (state, seen) = setting_up().await; + let mut film = MediaItem::empty(MediaKind::Movie); + film.title = "Sans Source".into(); + film.slug = "sans-source".into(); + film.is_manual = true; + crate::db::repo::item::upsert( + &state.db, + crate::db::repo::item::ItemWrite { + item: &film, + replace_children: true, + }, + ) + .await + .unwrap(); + let handed = crate::service::ids::own(&film.id); + + send(&state, vec![(film.id.clone(), vec![Reason::Edit])]).await; + + let looked_for = format!("/api/v3/movie?tmdbId={handed}"); + assert!( + seen.lock() + .iter() + .any(|s| s.method == "GET" && s.target == looked_for), + "Radarr was not asked for {looked_for}" + ); + let pushes = history::latest(&state.db, 10).await.unwrap(); + let radarr = pushes.iter().find(|p| p.app == App::Radarr).unwrap(); + assert_eq!(radarr.client_id, Some(handed)); + assert_eq!(radarr.outcome, Outcome::Absent); + } + const SECOND: &str = "0192aa52-8f62-7cc3-a1f1-6a1b2c3d4e61"; const THIRD: &str = "0192aa52-8f62-7cc3-a1f1-6a1b2c3d4e62"; diff --git a/src/arr/seed.rs b/src/arr/seed.rs index 1e78e9e..9291dd1 100644 --- a/src/arr/seed.rs +++ b/src/arr/seed.rs @@ -218,6 +218,24 @@ pub enum Seeded { /// Import one work of the library unless it is held: what came of it, and /// whether a provider was asked. pub async fn seed_one(state: &AppState, app: App, held: &Held) -> (Seeded, bool) { + // A work entered by hand, under the number made of its id here: held + // here, or nowhere a source could be asked for it. + let client_id = match app { + App::Sonarr => held.tvdb_id, + App::Radarr => held.tmdb_id, + }; + if let Some(client_id) = client_id.filter(|id| ids::is_own(*id)) { + let kind = match app { + App::Sonarr => MediaKind::Series, + App::Radarr => MediaKind::Movie, + }; + return match service::by_own_id(state, kind, client_id).await { + Ok(Some(item)) => (Seeded::Here(item.id), false), + Ok(None) => (Seeded::NoId, false), + Err(e) => (Seeded::Failed(format!("{e:#}")), false), + }; + } + let ids = lookup(app, held); let Some((source, value)) = ids.first() else { return (Seeded::NoId, false); @@ -381,6 +399,57 @@ mod tests { assert!(lookup(App::Radarr, &held(5, None, Some(0), None)).is_empty()); } + /// A work entered by hand is held by Sonarr and Radarr under the number + /// made of its id here: found here by it, and no source asked — none + /// could have it, not even by the IMDb id it was typed in with. + #[tokio::test] + async fn a_work_entered_by_hand_is_found_here_by_the_number_it_was_handed() { + let state = crate::api::native::testing::server().await; + let mut made = Vec::new(); + for kind in [MediaKind::Series, MediaKind::Movie] { + let mut work = MediaItem::empty(kind); + work.title = "Vicineko".into(); + work.slug = format!("vicineko-{kind}"); + work.is_manual = true; + repo::item::upsert( + &state.db, + repo::item::ItemWrite { + item: &work, + replace_children: true, + }, + ) + .await + .unwrap(); + made.push(work.id); + } + let series = ids::own(&made[0]); + let film = ids::own(&made[1]); + + let (seeded, asked) = + seed_one(&state, App::Sonarr, &held(1, Some(series), None, None)).await; + assert!( + matches!(&seeded, Seeded::Here(id) if *id == made[0]), + "{seeded:?}" + ); + assert!(!asked); + let (seeded, asked) = seed_one(&state, App::Radarr, &held(2, None, Some(film), None)).await; + assert!( + matches!(&seeded, Seeded::Here(id) if *id == made[1]), + "{seeded:?}" + ); + assert!(!asked); + + // A film's number is no series': Sonarr holding it names nothing here. + let (seeded, asked) = seed_one( + &state, + App::Sonarr, + &held(3, Some(film), None, Some("tt0903747")), + ) + .await; + assert!(matches!(seeded, Seeded::NoId), "{seeded:?}"); + assert!(!asked); + } + #[tokio::test] async fn what_is_held_is_passed_over_and_nothing_is_fetched_twice() { let state = crate::api::native::testing::server().await; diff --git a/src/db/repo/asset.rs b/src/db/repo/asset.rs index 57b495c..6432922 100644 --- a/src/db/repo/asset.rs +++ b/src/db/repo/asset.rs @@ -1163,6 +1163,45 @@ pub async fn delete_variant(db: &Db, key: &str) -> Result<()> { Ok(()) } +/// The copies written as `format` — `webp`, `avif`, `jpeg` — after `after` +/// in the order of their keys, `limit` of them at the most: what deleting +/// every copy of a format takes, a few hundred at a time, walking them once. +pub async fn variants_by_format( + db: &Db, + format: &str, + after: &str, + limit: i64, +) -> Result> { + let rows = sqlx::query(db.sql( + "SELECT key, stem, width, height, format, bytes FROM media_variant + WHERE format = ? AND key > ? + ORDER BY key + LIMIT ?", + )) + .bind(format) + .bind(after) + .bind(limit.clamp(1, 500)) + .fetch_all(db.pool()) + .await?; + rows.iter().map(map_variant).collect() +} + +/// Forget these copies at once: their files are gone. How many were noted. +pub async fn delete_variants(db: &Db, keys: &[String]) -> Result { + let mut deleted = 0; + // Asked in slices a bind list takes, as `by_origins` asks. + for slice in keys.chunks(400) { + let marks = vec!["?"; slice.len()].join(", "); + let mut query = + sqlx::query(db.sql(&format!("DELETE FROM media_variant WHERE key IN ({marks})"))); + for key in slice { + query = query.bind(key); + } + deleted += query.execute(db.pool()).await?.rows_affected(); + } + Ok(deleted) +} + /// Every AVIF copy made, by its picture's hash and its width: what the /// index in memory tells the pages, which offer one only where it exists. pub async fn avif_index(db: &Db) -> Result> { @@ -1764,6 +1803,67 @@ mod tests { assert!(avif_index(&db).await.unwrap().is_empty()); } + /// A format's copies are walked once, a few at a time in the order of + /// their keys, and forgotten together; another format's are left as + /// they are. + #[tokio::test] + async fn a_formats_copies_are_walked_once_and_forgotten_together() { + let db = db().await; + for (made, format) in [ + ("240.webp", "webp"), + ("480.webp", "webp"), + ("960.webp", "webp"), + ("480.avif", "avif"), + ("square.jpg", "jpeg"), + ] { + add_variant( + &db, + &Variant { + key: format!("{SHA}-{made}"), + stem: SHA.into(), + width: 480, + height: 720, + format: format.into(), + bytes: 10, + }, + ) + .await + .unwrap(); + } + let keys = |copies: &[Variant]| -> Vec { + copies.iter().map(|copy| copy.key.clone()).collect() + }; + + let first = variants_by_format(&db, "webp", "", 2).await.unwrap(); + assert_eq!( + keys(&first), + [format!("{SHA}-240.webp"), format!("{SHA}-480.webp")] + ); + let rest = variants_by_format(&db, "webp", &first[1].key, 2) + .await + .unwrap(); + assert_eq!(keys(&rest), [format!("{SHA}-960.webp")]); + assert!( + variants_by_format(&db, "webp", &rest[0].key, 2) + .await + .unwrap() + .is_empty(), + "walked once" + ); + + let webp = [keys(&first), keys(&rest)].concat(); + assert_eq!(delete_variants(&db, &webp).await.unwrap(), 3); + assert_eq!( + delete_variants(&db, &webp).await.unwrap(), + 0, + "forgotten already" + ); + assert_eq!(delete_variants(&db, &[]).await.unwrap(), 0); + let counts = variant_counts(&db).await.unwrap(); + assert_eq!((counts.webp, counts.avif, counts.jpeg), (0, 1, 1)); + assert_eq!(variants_of(&db, SHA).await.unwrap().len(), 2); + } + /// What a work has a picture as decides the copies made of it; the /// pictures this version has not prepared are listed until each is, so /// a run stopped partway is taken up where it stopped. diff --git a/src/db/repo/item/read.rs b/src/db/repo/item/read.rs index 88c4dcd..d06d346 100644 --- a/src/db/repo/item/read.rs +++ b/src/db/repo/item/read.rs @@ -166,6 +166,45 @@ pub async fn find_id_by_external( .map_err(Into::into) } +/// The works of `kind` entered by hand — switched off, merged into another or +/// not — the first entered first: what an id a client was handed for one, +/// made of its catalogue id (`service::ids::own`), is looked for among. +pub async fn made_by_hand(db: &Db, kind: MediaKind) -> Result> { + let rows = + sqlx::query(db.sql( + "SELECT id FROM media_item WHERE kind = ? AND is_manual = 1 ORDER BY created_at, id", + )) + .bind(kind.as_str()) + .fetch_all(db.pool()) + .await?; + + rows.iter() + .map(|row| row.text("id").map_err(Into::into)) + .collect() +} + +/// The films entered by hand that changed at `since` or after, as +/// [`movies_changed_since`] tells a change: by their catalogue ids, which +/// the ids Radarr was handed for them are made of. +pub async fn films_by_hand_changed_since(db: &Db, since: &str) -> Result> { + let rows = sqlx::query(db.sql( + "SELECT m.id FROM media_item m + WHERE m.kind = 'movie' AND m.is_manual = 1 + AND (m.updated_at >= ? OR m.listed_at >= ? + OR EXISTS (SELECT 1 FROM media_override o + WHERE o.media_id = m.id AND o.updated_at >= ?))", + )) + .bind(since) + .bind(since) + .bind(since) + .fetch_all(db.pool()) + .await?; + + rows.iter() + .map(|row| row.text("id").map_err(Into::into)) + .collect() +} + /// The TMDB ids of the films held that changed at `since` or after — an /// instant written as [`now`](crate::db::now) writes them — switched off or not: written /// again, listed again (which a lock put on, changed or lifted asks for, and diff --git a/src/db/repo/scene.rs b/src/db/repo/scene.rs index 76fcfbf..640f4c4 100644 --- a/src/db/repo/scene.rs +++ b/src/db/repo/scene.rs @@ -17,6 +17,11 @@ use crate::{ /// One series, as the scene-mapping list needs it. #[derive(Clone, Debug, Default)] pub struct SeriesTitles { + /// The catalogue's id for it: what a series entered by hand that nothing + /// elsewhere names is handed its client id from. + pub id: String, + /// Whether it was entered by hand. + pub is_manual: bool, /// Its title, a locked one included. pub title: String, /// Whether that title is locked: served as it is, in every language. @@ -50,7 +55,8 @@ pub async fn series_titles(db: &Db) -> Result> { // would make the survivor its own homonym, and Sonarr would be served // the survivor's name with a year it never had. let rows = sqlx::query(db.sql( - "SELECT id, title, original_title, original_language, is_enabled, title_qualifier, year \ + "SELECT id, title, original_title, original_language, is_enabled, title_qualifier, year, \ + is_manual \ FROM media_item \ WHERE kind = 'series' AND merged_into IS NULL ORDER BY id", )) @@ -62,6 +68,8 @@ pub async fn series_titles(db: &Db) -> Result> { by_id.insert( id.clone(), SeriesTitles { + id, + is_manual: row.flag("is_manual")?, title: row.text("title")?, original_title: row.opt_text("original_title")?, original_language: row.opt_text("original_language")?, diff --git a/src/jobs/media.rs b/src/jobs/media.rs index c803d37..66f1a44 100644 --- a/src/jobs/media.rs +++ b/src/jobs/media.rs @@ -1,19 +1,24 @@ -//! The two tasks that make copies of the pictures kept. +//! The two tasks that make copies of the pictures kept, and what deletes +//! every copy of a format once it is off. //! //! - `media.variants`, by hand only: every picture stored before this //! version — or before there were copies at all — prepared as one stored -//! now is: its placeholder worked out, and the copy its kind is first -//! asked for made, a poster's 480 and a background's 1280. A few at a -//! time, each picture noted on its row once done, so that a run stopped -//! — by hand, or by the server going down — is taken up by the next where -//! it stopped. Never at start: a catalogue's worth of pictures is hours -//! of decoding on a small machine, and when to spend them is the -//! administrator's choice. Until then a page is given what it asks for, -//! made as it asks. +//! now is: its placeholder worked out, and — only while the WebP copies +//! are on, `media.webp` — the copy its kind is first asked for made, a +//! poster's 480 and a background's 1280. A few at a time, each picture +//! noted on its row once done, so that a run stopped — by hand, or by the +//! server going down — is taken up by the next where it stopped. Never at +//! start: a catalogue's worth of pictures is hours of decoding on a small +//! machine, and when to spend them is the administrator's choice. Until +//! then a page is given what it asks for, made as it asks. //! - `media.avif`, at night once `media.avif` is on, and by hand: an AVIF //! copy of every poster at 480 and 960, and of every background at 1280 //! — a third lighter than the WebP, and seconds to make, one after //! another. A page offers one only where it exists. +//! +//! Once `media.webp` or `media.avif` is off, an administrator may delete +//! every copy of that format, to have the space back: no run of its own in +//! the history, but the media's status says so while it is under way. use std::{ sync::{Arc, LazyLock}, @@ -169,10 +174,11 @@ async fn prepare_all(state: &AppState, record: &str, flag: &cancel::Registered) ))) } -/// Prepare one picture: its placeholder and its first copy, noted on its -/// row. A picture the store does not have, or that cannot be read, is noted -/// all the same — another run would find it no better — and one the store -/// could not be asked for is left for the next run. +/// Prepare one picture: its placeholder and — while the WebP copies are +/// on, as `media.webp` says when the picture is taken — its first copy, +/// noted on its row. A picture the store does not have, or that cannot be +/// read, is noted all the same — another run would find it no better — and +/// one the store could not be asked for is left for the next run. async fn prepare_one(state: &AppState, store: &crate::media::store::Store, asset: &Asset) -> Done { let failed = |note: String| Done { outcome: Outcome::Failed, @@ -198,7 +204,8 @@ async fn prepare_one(state: &AppState, store: &crate::media::store::Store, asset let cover = repo::asset::cover_of(&state.db, &asset.origin) .await .unwrap_or(None); - let prepared = variants::off_thread(move || variants::prepare(&bytes, cover)).await; + let webp = state.flag("media.webp", false); + let prepared = variants::off_thread(move || variants::prepare(&bytes, cover, webp)).await; let prepared = match prepared { Ok(prepared) => prepared, Err(e) => { @@ -332,7 +339,8 @@ fn short(origin: &str) -> String { /// Once a run is over, done or stopped: the works and the pages of cards /// kept as they were read are read again, so that the placeholders and the -/// AVIF copies made are told to the pages — here and on every instance. +/// AVIF copies made — or deleted — are told to the pages, here and on every +/// instance. async fn drawn_again(state: &AppState) { for space in ["items", "searches"] { state.caches.flush(space).await; @@ -579,6 +587,175 @@ pub async fn run_nights(state: AppState) { } } +// ─── deleting every copy of a format ───────────────────────────────────────── + +/// What a deletion of copies is held under among several instances. +pub const PURGE: &str = "media.purge"; + +/// How many copies are deleted at a time. +const PURGE_BATCH: i64 = 500; + +static PURGING: LazyLock>> = + LazyLock::new(|| Arc::new(tokio::sync::Mutex::new(()))); + +/// Whether every copy of a format is being deleted on this instance. +pub fn is_purging() -> bool { + PURGING.try_lock().is_err() +} + +/// The setting that has a format's copies made: none for the square Plex +/// is given, which is made whatever either says, and never deleted here. +pub fn setting_of(format: Format) -> Option<&'static str> { + match format { + Format::Webp => Some("media.webp"), + Format::Avif => Some("media.avif"), + Format::Jpeg => None, + } +} + +/// How many copies of a format there were as their deletion began, and +/// what they weighed. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Tally { + pub copies: i64, + pub bytes: i64, +} + +/// Delete every copy written as `format` — WebP or AVIF — in the +/// background, and say what there was. Refused while the setting that +/// makes them is on, as the pages are given them; for the AVIF copies, +/// while the night's are being made, here or on another instance — they +/// would come back behind the deletion — and none is made until it is +/// over; and while another deletion runs. +pub async fn purge_now(state: &AppState, format: Format) -> AppResult { + let Some(setting) = setting_of(format) else { + return Err(AppError::BadRequest( + "the squares Plex is given are not deleted".into(), + )); + }; + if !state.media.is_on() { + return Err(media_off()); + } + let name = if format == Format::Avif { + "AVIF" + } else { + "WebP" + }; + if state.flag(setting, false) { + return Err(AppError::Conflict(format!( + "the {name} copies are still made and given to the pages: turn {setting} off first" + ))); + } + let held = PURGING + .clone() + .try_lock_owned() + .map_err(|_| AppError::Conflict("copies are being deleted already".into()))?; + let making = if format == Format::Avif { + if is_making_avif() { + return Err(AppError::Conflict( + "the AVIF copies are being made; wait for the run to end, or stop it".into(), + )); + } + Some( + state + .coord + .hold_for(AVIF, "the AVIF copies are being made") + .await?, + ) + } else { + None + }; + let shared = state + .coord + .hold_for(PURGE, "copies are being deleted") + .await?; + let counts = repo::asset::variant_counts(&state.db).await?; + let tally = match format { + Format::Webp => Tally { + copies: counts.webp, + bytes: counts.webp_bytes, + }, + Format::Avif => Tally { + copies: counts.avif, + bytes: counts.avif_bytes, + }, + Format::Jpeg => Tally { + copies: counts.jpeg, + bytes: counts.jpeg_bytes, + }, + }; + + let state = state.clone(); + tokio::spawn(async move { + let _held = held; + let _making = making; + let _shared = shared; + match purge_all(&state, format, setting).await { + Ok(summary) => tracing::info!(format = format.as_str(), summary, "copies deleted"), + Err(e) => tracing::warn!( + format = format.as_str(), + error = format_args!("{e:#}"), + "the copies could not all be deleted" + ), + } + drawn_again(&state).await; + }); + Ok(tally) +} + +/// The deletion itself, a few hundred copies at a time in the order of +/// their keys: each file taken from the store — one gone already is gone — +/// then the rows of those that went, and an AVIF no longer offered to a +/// page. A file the store would not let go of keeps its row, for the next +/// deletion to try again. Turned on again meanwhile, the setting stops it +/// where it is: what is left is the pages' again. +async fn purge_all(state: &AppState, format: Format, setting: &str) -> Result { + let store = state.media.store().context("no media store")?; + let (mut deleted, mut troubles) = (0u64, 0u64); + let mut after = String::new(); + loop { + if state.flag(setting, false) { + return Ok(format!( + "stopped, {setting} being on again: {deleted} deleted, {troubles} could not be" + )); + } + let line = repo::asset::variants_by_format(&state.db, format.as_str(), &after, PURGE_BATCH) + .await?; + let Some(last) = line.last() else { + break; + }; + after.clone_from(&last.key); + let mut gone = Vec::with_capacity(line.len()); + for copy in &line { + match store.delete(©.key).await { + Ok(()) => gone.push(copy), + Err(e) => { + tracing::warn!( + key = %copy.key, + error = format_args!("{e:#}"), + "could not delete a copy's file" + ); + troubles += 1; + } + } + } + let keys: Vec = gone.iter().map(|copy| copy.key.clone()).collect(); + deleted += repo::asset::delete_variants(&state.db, &keys).await?; + if format == Format::Avif { + for copy in gone { + if let Ok(width) = u32::try_from(copy.width) { + state.media.drop_avif(©.stem, width); + } + } + } + } + let mut summary = format!("{deleted} deleted"); + if troubles > 0 { + summary.push_str(&format!(", {troubles} could not be (see the log)")); + } + Ok(summary) +} + #[cfg(test)] mod tests { use std::io::Cursor; @@ -695,14 +872,29 @@ mod tests { origins } + async fn webp_on(state: &AppState) { + state + .settings + .set( + crate::settings::Scope::Server, + "", + "media.webp", + "true", + None, + ) + .await + .expect("the WebP copies on"); + } + /// The task prepares what no version prepared, a few at a time: a /// picture prepared before a run was stopped is not prepared again, the - /// copy a poster is first asked for is made and noted, and a page is - /// told what the picture looks like. + /// copy a poster is first asked for is made and noted — the WebP copies + /// being on — and a page is told what the picture looks like. #[tokio::test] async fn the_pictures_are_prepared_once_and_a_stopped_run_is_taken_up() { let dir = crate::testing::scratch_path("variants"); let state = server(&dir).await; + webp_on(&state).await; let store = state.media.store().unwrap().clone(); let origins = kept_as_before(&state, &[600, 620, 640]).await; assert_eq!( @@ -768,6 +960,50 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + /// While the WebP copies are off, as they are until somebody turns them + /// on, a picture is prepared all the same — its placeholder worked out, + /// and told to the pages — and no copy is made of it. + #[tokio::test] + async fn a_picture_is_prepared_without_a_copy_while_the_webp_copies_are_off() { + let dir = crate::testing::scratch_path("variants-off"); + let state = server(&dir).await; + let store = state.media.store().unwrap().clone(); + let origins = kept_as_before(&state, &[660]).await; + let asset = repo::asset::unprepared(&state.db, variants::PREPARED, "", 1) + .await + .unwrap() + .remove(0); + assert_eq!( + prepare_one(&state, &store, &asset).await.outcome, + Outcome::Written + ); + + let asset = repo::asset::by_origin(&state.db, &origins[0]) + .await + .unwrap() + .unwrap(); + assert!(asset.color.is_some() && asset.thumbhash.is_some()); + assert_eq!( + repo::asset::unprepared_count(&state.db, variants::PREPARED) + .await + .unwrap(), + 0, + "prepared, and not to be prepared again" + ); + let key = asset.key.unwrap(); + let stem = file::stem_of(&key); + assert!( + repo::asset::variants_of(&state.db, stem) + .await + .unwrap() + .is_empty() + ); + assert!(!store.exists(&format!("{stem}-480.webp")).await.unwrap()); + assert!(state.media.keyed(&key).unwrap().look.thumbhash.is_some()); + + let _ = std::fs::remove_dir_all(&dir); + } + /// The night's AVIF copies of a poster are made once — at 480 and 960, /// and at its own width where it is narrower: a small one here, as an /// unoptimised AVIF encoder takes seconds over a 480 — and offered from diff --git a/src/media/serve.rs b/src/media/serve.rs index 7af94b7..813beef 100644 --- a/src/media/serve.rs +++ b/src/media/serve.rs @@ -10,7 +10,10 @@ //! //! A copy of a picture — `-480.webp`, `-square.jpg` — is made //! the first time it is asked for and kept: see [`super::variants`]. An -//! AVIF is the night's to make, never a request's. +//! AVIF is the night's to make, never a request's. A WebP is made only +//! while `media.webp` is on: off, one kept already is served, and a page +//! that asks for one that is not is sent to the picture itself — for this +//! time only, as they may be turned on again. use std::{ops::Range, sync::LazyLock, time::Duration}; @@ -255,7 +258,8 @@ async fn copy( return not_modified(&etag); } // Sent to the bucket itself, when that is how it is set and the copy - // is there already; one that is not is made, and answered from here. + // is there already; one that is not is made — a WebP only while they + // are on — and answered from here. if state.text("media.serve").as_deref() == Some("redirect") && store.exists(&key).await.unwrap_or(false) && let Ok(Some(url)) = store.presign(&key, method.clone(), PRESIGNED_FOR).await @@ -263,6 +267,23 @@ async fn copy( return redirect(url.as_str(), "private, max-age=3000"); } + // The WebP copies off: one kept already is served, and none is made — + // the page is sent to the picture itself, and asks again next time. + if format == Format::Webp && !state.flag("media.webp", false) { + return match store.read(&key).await { + Ok(Some(bytes)) => whole(method, format_type(format), &key, &etag, bytes, IMMUTABLE), + Ok(None) => redirect(&state.media.url_for(&original), "no-cache"), + Err(e) => { + tracing::warn!( + key, + error = format_args!("{e:#}"), + "could not read a medium" + ); + redirect(&state.media.url_for(&original), "no-cache") + } + }; + } + match kept_or_made(state, store, &original, asked.stem, made, format, &key).await { Ok(bytes) => whole(method, format_type(format), &key, &etag, bytes, IMMUTABLE), Err(e) => { @@ -279,8 +300,9 @@ async fn copy( /// An AVIF copy: the night's task makes them, and a page is offered one /// only where it exists. One whose file has gone — a store moved, a restore -/// onto a new machine — is answered with the WebP of the same width, not -/// kept for long, and forgotten, so that the next night makes it again. +/// onto a new machine — is answered with the WebP of the same width, or, +/// while the WebP copies are off, with the picture itself; not kept for +/// long, and forgotten, so that the next night makes it again. #[allow(clippy::too_many_arguments)] async fn avif( state: &AppState, @@ -316,6 +338,9 @@ async fn avif( tracing::debug!(key, error = %e, "could not forget a copy whose file is gone"); } } + if !state.flag("media.webp", false) { + return redirect(&state.media.url_for(original), "no-cache"); + } let webp = file::made_key(stem, made, Format::Webp.ext()); match kept_or_made(state, store, original, stem, made, Format::Webp, &webp).await { Ok(bytes) => whole( @@ -720,4 +745,191 @@ mod tests { assert_eq!(response.status(), StatusCode::RANGE_NOT_SATISFIABLE); assert_eq!(response.headers()[header::CONTENT_RANGE], "bytes */1000"); } + + /// The whole server, on a database of its own, keeping its media in + /// `dir`. + async fn server(dir: &std::path::Path) -> AppState { + let mut config = crate::config::Config::from_env().expect("a configuration"); + config.mode = crate::config::Mode::Single; + config.database = crate::db::testing::database().await; + config.security.bootstrap_admin = None; + config.cache.redis_url = None; + config.media.storage = crate::config::MediaStorage::Filesystem; + config.media.dir = dir.to_path_buf(); + config.clients = None; + config.tmdb.api_key = None; + config.tvdb.api_key = None; + config.tvdb.enabled = false; + AppState::bootstrap(config).await.expect("a server") + } + + /// A poster kept, as the worker keeps one — in the store and in the + /// index — and no copy of it yet: its key and its hash. `shade` makes + /// it a picture of its own, whose copies no other test makes. + async fn kept(state: &AppState, shade: u8) -> (String, String) { + let mut bytes = Vec::new(); + image::RgbImage::from_fn(600, 900, |x, y| { + image::Rgb([(x % 256) as u8, (y % 256) as u8, shade]) + }) + .write_to( + &mut std::io::Cursor::new(&mut bytes), + image::ImageFormat::Png, + ) + .unwrap(); + let (key, sha) = file::key_for(&bytes, "png"); + state + .media + .store() + .unwrap() + .put(&key, Bytes::from(bytes), "image/png") + .await + .unwrap(); + state.media.remember( + &format!("https://images.example/{sha}.png"), + &key, + repo::asset::Thumb::None, + "image/png", + ); + (key, sha) + } + + async fn asked(state: &AppState, key: &str) -> Response { + get( + State(state.clone()), + Method::GET, + Path(key.to_string()), + HeaderMap::new(), + ) + .await + } + + async fn webp_copies(state: &AppState, sha: &str) -> Vec<(String, i32)> { + repo::asset::variants_of(&state.db, sha) + .await + .unwrap() + .into_iter() + .filter(|copy| copy.format == "webp") + .map(|copy| (copy.key, copy.width)) + .collect() + } + + async fn turn(state: &AppState, key: &str, on: bool) { + state + .settings + .set( + crate::settings::Scope::Server, + "", + key, + if on { "true" } else { "false" }, + None, + ) + .await + .unwrap(); + } + + /// While the WebP copies are off — as they are until somebody turns + /// them on — a page that asks for one is sent to the picture itself, + /// and asks again next time: nothing is made, and nothing noted. On, + /// one is made, kept and noted as it is asked for; off again, the copy + /// kept is still served, and no other is made. + #[tokio::test] + async fn a_webp_copy_is_made_only_while_they_are_on() { + let dir = crate::testing::scratch_path("serve-webp"); + let state = server(&dir).await; + let store = state.media.store().unwrap().clone(); + let (key, sha) = kept(&state, 31).await; + let copy = format!("{sha}-480.webp"); + + let answer = asked(&state, ©).await; + assert_eq!(answer.status(), StatusCode::FOUND); + assert_eq!(answer.headers()[header::LOCATION], format!("/media/{key}")); + assert_eq!(answer.headers()[header::CACHE_CONTROL], "no-cache"); + assert!(!store.exists(©).await.unwrap(), "nothing made"); + assert!(webp_copies(&state, &sha).await.is_empty(), "nothing noted"); + // A width no page asks for is refused all the same. + let refused = asked(&state, &format!("{sha}-481.webp")).await; + assert_eq!(refused.status(), StatusCode::NOT_FOUND); + + turn(&state, "media.webp", true).await; + let answer = asked(&state, ©).await; + assert_eq!(answer.status(), StatusCode::OK); + assert_eq!(answer.headers()[header::CONTENT_TYPE], "image/webp"); + assert_eq!(answer.headers()[header::CACHE_CONTROL], IMMUTABLE); + assert!(store.exists(©).await.unwrap(), "made and kept"); + assert_eq!(webp_copies(&state, &sha).await, [(copy.clone(), 480)]); + + turn(&state, "media.webp", false).await; + let answer = asked(&state, ©).await; + assert_eq!(answer.status(), StatusCode::OK, "the copy kept is served"); + assert_eq!(answer.headers()[header::CONTENT_TYPE], "image/webp"); + let other = format!("{sha}-240.webp"); + let answer = asked(&state, &other).await; + assert_eq!(answer.status(), StatusCode::FOUND); + assert_eq!(answer.headers()[header::LOCATION], format!("/media/{key}")); + assert!(!store.exists(&other).await.unwrap()); + assert_eq!(webp_copies(&state, &sha).await.len(), 1); + + let _ = std::fs::remove_dir_all(&dir); + } + + /// An AVIF of 480 of the picture of `sha`, noted and offered, whose + /// file is not there: a store moved, say. Its key. + async fn avif_without_its_file(state: &AppState, sha: &str) -> String { + let avif = format!("{sha}-480.avif"); + repo::asset::add_variant( + &state.db, + &repo::asset::Variant { + key: avif.clone(), + stem: sha.to_string(), + width: 480, + height: 720, + format: "avif".into(), + bytes: 10, + }, + ) + .await + .unwrap(); + state.media.add_avif(sha, 480); + avif + } + + /// An AVIF whose file has gone is forgotten, and answered — while the + /// WebP copies are off — with the picture itself, not with a WebP made + /// for it; while they are on, with the WebP of its width, as before. + #[tokio::test] + async fn an_avif_gone_is_answered_with_the_picture_while_the_webp_copies_are_off() { + let dir = crate::testing::scratch_path("serve-avif"); + let state = server(&dir).await; + let store = state.media.store().unwrap().clone(); + let (key, sha) = kept(&state, 47).await; + + let avif = avif_without_its_file(&state, &sha).await; + assert_eq!(state.media.keyed(&key).unwrap().look.avif, [480]); + let answer = asked(&state, &avif).await; + assert_eq!(answer.status(), StatusCode::FOUND); + assert_eq!(answer.headers()[header::LOCATION], format!("/media/{key}")); + assert_eq!(answer.headers()[header::CACHE_CONTROL], "no-cache"); + assert!(state.media.keyed(&key).unwrap().look.avif.is_empty()); + assert!( + repo::asset::variants_of(&state.db, &sha) + .await + .unwrap() + .is_empty(), + "forgotten, and no WebP made for it" + ); + assert!(!store.exists(&format!("{sha}-480.webp")).await.unwrap()); + + turn(&state, "media.webp", true).await; + let avif = avif_without_its_file(&state, &sha).await; + let answer = asked(&state, &avif).await; + assert_eq!(answer.status(), StatusCode::OK); + assert_eq!(answer.headers()[header::CONTENT_TYPE], "image/webp"); + assert_eq!(answer.headers()[header::CACHE_CONTROL], "no-cache"); + assert_eq!( + webp_copies(&state, &sha).await, + [(format!("{sha}-480.webp"), 480)] + ); + + let _ = std::fs::remove_dir_all(&dir); + } } diff --git a/src/media/variants.rs b/src/media/variants.rs index 66a7bd9..5de20ca 100644 --- a/src/media/variants.rs +++ b/src/media/variants.rs @@ -9,6 +9,11 @@ //! one the interface asks for first — a poster's 480, a background's 1280 — //! as the picture is stored. A picture is never made larger than it is: a //! width past its own is its own width. +//! +//! The WebP copies are made only while `media.webp` is on: off, as it is +//! until somebody sets it, the pages are given the pictures themselves, and +//! no WebP is made — not as a picture is stored, not when one is asked for. +//! The square Plex is given is made whatever it says. use std::{ panic::AssertUnwindSafe, @@ -50,9 +55,9 @@ pub const AVIF_BACKDROP_WIDTHS: [u32; 2] = [1280, 1920]; pub const SQUARE_SIDE: u32 = 1080; /// What has been done to a picture as it was stored, by this version: its -/// placeholder worked out, and the copy its kind is first asked for made. -/// A picture prepared by an earlier version — or by none — is the task -/// `media.variants`' to prepare. +/// placeholder worked out, and — while the WebP copies are on — the copy +/// its kind is first asked for made. A picture prepared by an earlier +/// version — or by none — is the task `media.variants`' to prepare. pub const PREPARED: i64 = 1; /// The copy made of a picture as it is stored, by what the work it was @@ -353,23 +358,24 @@ pub fn make( } } -/// What preparing a picture as it is stored came to: its placeholder, the -/// copy its kind is first asked for, and its size as decoded. +/// What preparing a picture as it is stored came to: its placeholder, and +/// the copy its kind is first asked for, when one was made. pub struct Prepared { pub placeholder: Option, pub copies: Vec, } -/// Prepare a picture: its placeholder and, where its kind has one, its -/// first copy — the picture decoded once for both. `cover` is what the -/// picture is to its work, where that is known. -pub fn prepare(bytes: &[u8], cover: Option) -> Result { +/// Prepare a picture: its placeholder and, where its kind has one and +/// `webp` says the WebP copies are on (`media.webp`), its first copy — the +/// picture decoded once for both. `cover` is what the picture is to its +/// work, where that is known. +pub fn prepare(bytes: &[u8], cover: Option, webp: bool) -> Result { let decoded = Decoded::read(bytes)?; let placeholder = super::placeholder::of(&decoded) .inspect_err(|e| tracing::debug!(error = format_args!("{e:#}"), "no placeholder")) .ok(); let mut copies = Vec::new(); - if let Some(width) = eager_width(cover) { + if webp && let Some(width) = eager_width(cover) { copies.push(make(&decoded, Made::Width(width), Format::Webp, cover)?); } Ok(Prepared { @@ -582,22 +588,38 @@ mod tests { #[test] fn a_picture_is_prepared_with_the_copy_its_kind_is_first_asked_for() { let poster = png(1000, 1500); - let prepared = prepare(&poster, Some(CoverType::Poster)).unwrap(); + let prepared = prepare(&poster, Some(CoverType::Poster), true).unwrap(); assert!(prepared.placeholder.is_some()); assert_eq!(prepared.copies.len(), 1); assert_eq!( (prepared.copies[0].width, prepared.copies[0].height), (480, 720) ); + assert_eq!(prepared.copies[0].format, Format::Webp); let backdrop = png(1600, 900); - let prepared = prepare(&backdrop, Some(CoverType::Fanart)).unwrap(); + let prepared = prepare(&backdrop, Some(CoverType::Fanart), true).unwrap(); assert_eq!(prepared.copies[0].width, 1280); - let still = prepare(&png(640, 360), Some(CoverType::Screenshot)).unwrap(); + let still = prepare(&png(640, 360), Some(CoverType::Screenshot), true).unwrap(); assert!(still.copies.is_empty() && still.placeholder.is_some()); - assert!(prepare(b"not a picture", Some(CoverType::Poster)).is_err()); + assert!(prepare(b"not a picture", Some(CoverType::Poster), true).is_err()); + } + + /// With the WebP copies off, a picture is prepared all the same — its + /// placeholder worked out — and no copy is made of it. + #[test] + fn a_picture_is_prepared_without_a_copy_while_the_webp_copies_are_off() { + for (picture, cover) in [ + (png(1000, 1500), CoverType::Poster), + (png(1600, 900), CoverType::Fanart), + ] { + let prepared = prepare(&picture, Some(cover), false).unwrap(); + assert!(prepared.placeholder.is_some(), "{cover:?}"); + assert!(prepared.copies.is_empty(), "{cover:?}"); + } + assert!(prepare(b"not a picture", Some(CoverType::Poster), false).is_err()); } #[tokio::test] diff --git a/src/media/worker.rs b/src/media/worker.rs index 544421e..e5895bf 100644 --- a/src/media/worker.rs +++ b/src/media/worker.rs @@ -287,7 +287,16 @@ async fn store_one(state: &AppState, asset: &Asset) -> Result<(), Trouble> { }), Kind::Audio => None, }; - let kept = keep(store, key, sha256, fetched.bytes, &fetched.inspected, cover).await?; + let kept = keep( + store, + key, + sha256, + fetched.bytes, + &fetched.inspected, + cover, + state.flag("media.webp", false), + ) + .await?; // Forgotten meanwhile, it is not remembered either: the bytes are the // next sweep's to remove. if repo::asset::mark_stored(&state.db, &asset.id, &kept.as_row()).await? { @@ -365,11 +374,12 @@ impl Kept { } /// Put bytes in the store under their key — `file::key_for`'s — and, for a -/// picture, work out its placeholder and put beside it the copy its kind is -/// first asked for: a poster's 480, a background's 1280, by `cover`. Put -/// every time, though the same picture from two providers may be there -/// already: the same key is the same bytes, and a put is what tells a sweep -/// that looked a moment ago that they are wanted. +/// picture, work out its placeholder and, while the WebP copies are on +/// (`webp`, as `media.webp` says), put beside it the copy its kind is first +/// asked for: a poster's 480, a background's 1280, by `cover`. Put every +/// time, though the same picture from two providers may be there already: +/// the same key is the same bytes, and a put is what tells a sweep that +/// looked a moment ago that they are wanted. /// /// A picture the decoders here cannot read — or panic on, which is caught — /// is kept all the same, and served whole. @@ -380,6 +390,7 @@ pub async fn keep( bytes: Bytes, inspected: &file::Inspected, cover: Option, + webp: bool, ) -> Result { let size = i64::try_from(bytes.len()).unwrap_or(i64::MAX); store @@ -405,7 +416,7 @@ pub async fn keep( } let prepared = - super::variants::off_thread(move || super::variants::prepare(&bytes, cover)).await; + super::variants::off_thread(move || super::variants::prepare(&bytes, cover, webp)).await; // Done either way: a picture that cannot be read now never will be. kept.prepared = super::variants::PREPARED; match prepared { diff --git a/src/service/ids.rs b/src/service/ids.rs index 83a1734..ede6ab2 100644 --- a/src/service/ids.rs +++ b/src/service/ids.rs @@ -16,6 +16,19 @@ //! Fankai's productions take a second range, from 200 000 000: a Fan-Kai is a //! recut TheTVDB and TMDB list nothing of, so its client id is Fankai's own id //! plus that offset, and the two ranges are told apart by size alone. +//! +//! A work entered by hand that nothing elsewhere names — a series with no +//! TheTVDB, TMDB or Fankai id, a film with no TMDB id — takes a third range, +//! from 1 000 000 000 to the top of the `int` both clients keep an id in: a +//! number made of the catalogue's own id for the work ([`own`]), which nothing +//! ever changes, so that the work is handed the same one at every answer, after +//! a restart, on every instance. Radarr is handed it as the film's TMDB id, and +//! TMDB's film ids are nowhere near a billion either. Neither client takes an +//! id of nought or less for a work it adds, so the range is a positive one. A +//! client asking by such an id is answered from the works entered by hand +//! ([`crate::service::by_own_id`]); no source is asked, since none has it. + +use crate::domain::{ExternalIds, MediaKind}; /// Start of the synthetic id range. pub const SYNTHETIC_OFFSET: i64 = 100_000_000; @@ -26,9 +39,18 @@ pub const SYNTHETIC_OFFSET: i64 = 100_000_000; /// — never fill, so an id says which it is by its size. pub const FANKAI_OFFSET: i64 = 200_000_000; -/// Largest id that still fits Sonarr's 32-bit column. +/// Start of the range a work entered by hand is handed an id in, when nothing +/// elsewhere names it. Above Fankai's, which its productions — a few hundred — +/// never fill. +pub const OWN_OFFSET: i64 = 1_000_000_000; + +/// Largest id that still fits Sonarr's 32-bit column, and Radarr's. const MAX_SONARR_ID: i64 = i32::MAX as i64; +/// How many ids the range of works entered by hand holds: a little over a +/// billion. +const OWN_SPAN: u64 = (MAX_SONARR_ID - OWN_OFFSET + 1) as u64; + /// Wrap a TMDB id as a synthetic TVDB id. /// /// Returns `None` if the result would not fit in the client's integer column, @@ -47,14 +69,69 @@ pub fn from_synthetic(id: i64) -> Option { } /// Wrap Fankai's id for a production as a client id. +/// +/// `None` past the range's end, where a work entered by hand's begin. pub fn to_fankai(fankai_id: i64) -> Option { let encoded = FANKAI_OFFSET.checked_add(fankai_id)?; - (fankai_id > 0 && encoded <= MAX_SONARR_ID).then_some(encoded) + (fankai_id > 0 && encoded < OWN_OFFSET).then_some(encoded) } /// Recover Fankai's id from a client id made by [`to_fankai`]. pub fn from_fankai(id: i64) -> Option { - (id >= FANKAI_OFFSET).then(|| id - FANKAI_OFFSET) + (FANKAI_OFFSET..OWN_OFFSET) + .contains(&id) + .then(|| id - FANKAI_OFFSET) +} + +/// The id a client is handed for a work entered by hand that nothing +/// elsewhere names: made of the catalogue's own id for it, which no refresh, +/// restart or edit changes. +/// +/// A hash, so two catalogue ids may make one number — a chance in a little +/// over a billion for each two works: the one entered first keeps the number +/// ([`crate::service::own_holders`]), and a work entered by hand now is given +/// a catalogue id whose number no other holds. +pub fn own(catalogue_id: &str) -> i64 { + // FNV-1a, as `own_episode` below, on 64 bits: stable across builds and + // platforms, and spread over the whole range. + let mut hash: u64 = 0xcbf2_9ce4_8422_2325; + for byte in catalogue_id.bytes() { + hash ^= u64::from(byte); + hash = hash.wrapping_mul(0x0000_0100_0000_01b3); + } + // Below the span, so within an i64. + OWN_OFFSET + (hash % OWN_SPAN) as i64 +} + +/// Whether a client's id is one [`own`] makes: one only a work entered by +/// hand is handed. +pub fn is_own(id: i64) -> bool { + (OWN_OFFSET..=MAX_SONARR_ID).contains(&id) +} + +/// The id Sonarr (a series) or Radarr (a film) keeps a work under: for a +/// series TheTVDB's, else one made of TMDB's or of Fankai's; for a film +/// TMDB's; and for a work entered by hand that none of those names, one made +/// of the catalogue's own id for it ([`own`]). `None` for a work the client +/// has no way to address. +pub fn for_client( + kind: MediaKind, + catalogue_id: &str, + is_manual: bool, + ids: &ExternalIds, +) -> Option { + let named = match kind { + MediaKind::Series => series_named(ids.tvdb, ids.tmdb, ids.fankai), + MediaKind::Movie => ids.tmdb, + }; + named.or_else(|| is_manual.then(|| own(catalogue_id))) +} + +/// The id Sonarr keeps a series under from what names it elsewhere: +/// TheTVDB's, else one made of TMDB's, else one made of Fankai's. +pub fn series_named(tvdb: Option, tmdb: Option, fankai: Option) -> Option { + tvdb.or_else(|| tmdb.and_then(to_synthetic)) + .or_else(|| fankai.and_then(to_fankai)) } /// The id an episode is served under when an order carries it as a record @@ -142,6 +219,10 @@ mod tests { assert_eq!(from_fankai(100_001_396), None); assert_eq!(to_fankai(0), None); assert_eq!(to_fankai(i64::MAX), None); + // It ends where the works entered by hand begin. + assert_eq!(to_fankai(OWN_OFFSET - FANKAI_OFFSET), None); + assert_eq!(from_fankai(OWN_OFFSET), None); + assert_eq!(from_fankai(OWN_OFFSET - 1), Some(799_999_999)); } #[test] @@ -150,9 +231,60 @@ mod tests { for id in [1, 81189, 499_999, 1_000_000, 99_999_999] { assert_eq!(from_synthetic(id), None, "{id} should not look synthetic"); assert_eq!(from_fankai(id), None, "{id} should not look like a Fan-Kai"); + assert!(!is_own(id), "{id} should not look entered by hand"); } } + #[test] + fn a_work_entered_by_hand_takes_the_third_range_under_a_number_of_its_own_id() { + let id = "01a17789-b167-736d-8873-605960d7e943"; + let handed = own(id); + // The same from one answer, one restart, one build to the next. + assert_eq!(handed, own(id)); + assert_eq!(handed, 1_827_537_293); + assert!(is_own(handed)); + // In no other range, and within the clients' column. + assert_eq!(from_synthetic(handed), None); + assert_eq!(from_fankai(handed), None); + assert!(handed <= i64::from(i32::MAX)); + assert_eq!(own("01a17789-b191-73f2-9253-421af11df012"), 1_564_715_190); + // The top and the bottom of the range, and nothing past either. + assert!(is_own(OWN_OFFSET) && is_own(i64::from(i32::MAX))); + assert!(!is_own(OWN_OFFSET - 1) && !is_own(i64::from(i32::MAX) + 1)); + assert!(!is_own(-5)); + } + + #[test] + fn a_client_is_handed_what_names_a_work_elsewhere_before_its_own_number() { + let named = ExternalIds { + tvdb: Some(81189), + tmdb: Some(1396), + ..Default::default() + }; + let id = "01a17789-b167-736d-8873-605960d7e943"; + assert_eq!(for_client(MediaKind::Series, id, true, &named), Some(81189)); + assert_eq!(for_client(MediaKind::Movie, id, true, &named), Some(1396)); + + let nothing = ExternalIds::default(); + // Entered by hand: its own number, for either client. + assert_eq!( + for_client(MediaKind::Series, id, true, ¬hing), + Some(own(id)) + ); + assert_eq!( + for_client(MediaKind::Movie, id, true, ¬hing), + Some(own(id)) + ); + // A work a source gave with none of them is no client's. + assert_eq!(for_client(MediaKind::Series, id, false, ¬hing), None); + // An IMDb id alone names no series to Sonarr, nor a film to Radarr. + let imdb = ExternalIds { + imdb: Some("tt0903747".into()), + ..Default::default() + }; + assert_eq!(for_client(MediaKind::Movie, id, true, &imdb), Some(own(id))); + } + #[test] fn ids_that_would_overflow_the_client_are_refused() { // Sonarr stores this in a 32-bit column; a value past that would be diff --git a/src/service/merge.rs b/src/service/merge.rs index 5de15e8..c3d4c85 100644 --- a/src/service/merge.rs +++ b/src/service/merge.rs @@ -676,12 +676,17 @@ pub async fn merge( ( into.to_string(), before.title.clone(), - client_id(before.kind, &survivor_ids), + client_id(before.kind, into, kept_card.is_manual, &survivor_ids), ), ( merged.to_string(), gone.title.clone(), - client_id(gone_card.kind, &gone_card.external_ids), + client_id( + gone_card.kind, + merged, + gone_card.is_manual, + &gone_card.external_ids, + ), ), ], ); @@ -719,12 +724,10 @@ fn kept_values(item: &MediaItem, keep: &BTreeMap) -> BTreeMap Option { - match kind { - MediaKind::Series => crate::service::series::client_id_of(ids.tvdb, ids.tmdb, ids.fankai), - MediaKind::Movie => ids.tmdb, - } +/// for a series it does not list; TMDB's for a film; for a work entered by +/// hand that nothing elsewhere names, the number made of its catalogue id. +fn client_id(kind: MediaKind, id: &str, is_manual: bool, ids: &ExternalIds) -> Option { + crate::service::ids::for_client(kind, id, is_manual, ids) } /// The journal's line for the record merged in, or taken out again: its @@ -1036,14 +1039,14 @@ pub async fn unmerge( ( tombstone.into.clone(), before.title.clone(), - after - .as_ref() - .and_then(|after| client_id(after.kind, &after.external_ids)), + after.as_ref().and_then(|after| { + client_id(after.kind, &after.id, after.is_manual, &after.external_ids) + }), ), ( merged.to_string(), restored.title.clone(), - client_id(restored.kind, &ids), + client_id(restored.kind, merged, restored.is_manual, &ids), ), ], ); diff --git a/src/service/mod.rs b/src/service/mod.rs index 79b72e8..928e1f5 100644 --- a/src/service/mod.rs +++ b/src/service/mod.rs @@ -1240,6 +1240,73 @@ async fn or_held( Ok(Some(item)) } +// ─── works entered by hand ─────────────────────────────────────────────────── + +/// Which work entered by hand each number made of a catalogue id +/// ([`ids::own`]) leads to, for the works of `kind`: the one entered first, +/// should two catalogue ids make one number. Every one of them, switched off +/// or merged into another: Sonarr and Radarr keep asking by the number they +/// were handed. +pub async fn own_holders( + state: &AppState, + kind: MediaKind, +) -> Result> { + let mut holders = std::collections::HashMap::new(); + for id in repo::item::made_by_hand(&state.db, kind).await? { + holders.entry(ids::own(&id)).or_insert(id); + } + Ok(holders) +} + +/// The work entered by hand that Sonarr (a series) or Radarr (a film) keeps +/// under `client_id`, a number made of its catalogue id: as it is held, +/// switched off or not — a 404 would have the client take it for deleted — +/// or the work a merge folded it into. No source is asked: none has it. +pub async fn by_own_id( + state: &AppState, + kind: MediaKind, + client_id: i64, +) -> Result> { + let Some(id) = own_holders(state, kind).await?.remove(&client_id) else { + return Ok(None); + }; + Ok(load(state, &id).await?.filter(|item| item.kind == kind)) +} + +/// The id a client is handed for each of `items`, as `named` gives it, in +/// their order — a work entered by hand only under a number that leads back +/// to it. Two catalogue ids can make one number, and the work entered first +/// keeps it: one entered later under the same number is left out, as a +/// result the client could not add is, rather than offered under a number +/// that would add the other. Works entered by hand now are given a catalogue +/// id of a number of their own; one entered before may share it. +pub async fn addressed( + state: &AppState, + kind: MediaKind, + items: &[MediaItem], + named: impl Fn(&MediaItem) -> Option, +) -> Result>> { + let mut holders = None; + let mut handed = Vec::with_capacity(items.len()); + for item in items { + let id = match named(item) { + Some(id) if ids::is_own(id) => { + if holders.is_none() { + holders = Some(own_holders(state, kind).await?); + } + holders + .as_ref() + .and_then(|holders| holders.get(&id)) + .filter(|holder| **holder == item.id) + .map(|_| id) + } + other => other, + }; + handed.push(id); + } + Ok(handed) +} + /// Serve from cache, or run `fetch` and cache its result. /// The key a search is cached under. /// @@ -2131,3 +2198,160 @@ mod light_refresh_tests { assert!(due_light(&unscheduled, now)); } } + +#[cfg(test)] +mod by_hand_tests { + use super::{testing::server, *}; + + /// Two catalogue ids that make one number: found by trying, since a + /// work entered by hand now is given an id of a number of its own. The + /// one entered first sorts after the other, so that it is the order they + /// were entered in that is seen to decide. + const FIRST: &str = "01a00000-0000-7000-8000-000000052148"; + const LATER: &str = "01a00000-0000-7000-8000-000000011f4f"; + + async fn entered(state: &AppState, id: &str, kind: MediaKind, title: &str, at: &str) { + let mut work = MediaItem::empty(kind); + work.id = id.into(); + work.title = title.into(); + work.slug = crate::domain::make_slug(title, None); + work.is_manual = true; + work.created_at = at.into(); + repo::item::upsert( + &state.db, + repo::item::ItemWrite { + item: &work, + replace_children: true, + }, + ) + .await + .expect("stored"); + } + + #[tokio::test] + async fn a_number_two_works_entered_by_hand_make_leads_to_the_first_and_no_source_is_asked() { + assert_eq!(ids::own(FIRST), ids::own(LATER)); + let shared = ids::own(FIRST); + let (state, nowhere) = server().await; + // The later one stored first, and its id sorting first: what decides + // is when each was entered. + entered( + &state, + LATER, + MediaKind::Series, + "Later", + "2026-10-02T00:00:00.000Z", + ) + .await; + entered( + &state, + FIRST, + MediaKind::Series, + "First", + "2026-10-01T00:00:00.000Z", + ) + .await; + + let found = series::by_client_id(&state, shared) + .await + .expect("answered") + .expect("found"); + assert_eq!(found.id, FIRST); + assert_eq!( + nowhere.asked(), + 0, + "a source was asked for a work entered by hand" + ); + + // Searched for, the later one is left out: under that number Sonarr + // would add the first. + let both = [ + load(&state, LATER).await.unwrap().unwrap(), + load(&state, FIRST).await.unwrap().unwrap(), + ]; + assert_eq!( + series::client_ids(&state, &both).await.unwrap(), + [None, Some(shared)] + ); + // Nor is a film the series' number: each client's works apart. + assert!( + by_own_id(&state, MediaKind::Movie, shared) + .await + .unwrap() + .is_none() + ); + assert_eq!(nowhere.asked(), 0); + } + + #[tokio::test] + async fn a_work_entered_by_hand_switched_off_or_merged_away_is_still_answered_by_its_number() { + let (state, nowhere) = server().await; + + // Switched off: still Radarr's, as a film it holds. + entered( + &state, + FIRST, + MediaKind::Movie, + "Sans Source", + "2026-10-01T00:00:00.000Z", + ) + .await; + assert!( + repo::item::set_enabled(&state.db, FIRST, false) + .await + .unwrap() + ); + state.caches.touched(FIRST).await; + let found = movie::by_tmdb_id(&state, ids::own(FIRST)) + .await + .expect("answered") + .expect("found"); + assert_eq!(found.id, FIRST); + assert!(!found.is_enabled); + + // Merged into a series its sources fill: Sonarr, which holds it + // under its number, is answered with the work it went into. + entered( + &state, + LATER, + MediaKind::Series, + "Vicineko", + "2026-10-02T00:00:00.000Z", + ) + .await; + let mut kept = MediaItem::empty(MediaKind::Series); + kept.title = "Vicineko".into(); + kept.slug = "vicineko-tvdb".into(); + kept.external_ids.tvdb = Some(999_002); + repo::item::upsert( + &state.db, + repo::item::ItemWrite { + item: &kept, + replace_children: true, + }, + ) + .await + .expect("stored"); + merge::merge( + &state, + &merge::Asker { by: None, ip: None }, + LATER, + &kept.id, + Default::default(), + ) + .await + .expect("asked") + .expect("merged"); + let found = series::by_client_id(&state, ids::own(LATER)) + .await + .expect("answered") + .expect("found"); + assert_eq!(found.id, kept.id); + assert_eq!(found.external_ids.tvdb, Some(999_002)); + assert_eq!( + nowhere.asked(), + 0, + "a source was asked for a work entered by hand" + ); + } +} diff --git a/src/service/movie.rs b/src/service/movie.rs index 9da38a6..4c3b887 100644 --- a/src/service/movie.rs +++ b/src/service/movie.rs @@ -25,7 +25,14 @@ const MAX_BULK: usize = 100; /// A film by its TMDB id: as it is held, or fetched again when it is due — /// and as it is held after all when nothing answers (see /// [`crate::service::or_held`]). +/// +/// Or by the id Radarr was handed in its stead for a film entered by hand +/// that TMDB does not name — see [`ids::own`]: the catalogue's alone, so +/// TMDB is not asked for it. pub async fn by_tmdb_id(state: &AppState, tmdb_id: i64) -> Result> { + if ids::is_own(tmdb_id) { + return crate::service::by_own_id(state, MediaKind::Movie, tmdb_id).await; + } let value = tmdb_id.to_string(); if let Some(item) = local(state, ExternalSource::TmdbMovie, &value).await? { return Ok(Some(item)); @@ -602,11 +609,39 @@ pub async fn changed_since(state: &AppState, since: DateTime) -> Result Option { + ids::for_client( + MediaKind::Movie, + &item.id, + item.is_manual, + &item.external_ids, + ) +} + +/// The id Radarr is handed for each of `items`, as [`client_id`] gives it, +/// in their order — a film entered by hand only under a number that leads +/// back to it: see [`crate::service::addressed`]. +pub async fn client_ids(state: &AppState, items: &[MediaItem]) -> Result>> { + crate::service::addressed(state, MediaKind::Movie, items, client_id).await +} + /// The instant `since` names: as Radarr writes it — `2026-10-09T11:45:00`, /// in UTC without saying so — with an offset, or a bare date, from midnight /// UTC. diff --git a/src/service/scene.rs b/src/service/scene.rs index c2974e2..f21ba3a 100644 --- a/src/service/scene.rs +++ b/src/service/scene.rs @@ -49,7 +49,7 @@ use unicode_properties::general_category::{ use crate::{ db::repo, - service::{language, series}, + service::{ids, language, series}, state::AppState, }; @@ -417,13 +417,20 @@ fn plain_title(s: &repo::scene::SeriesTitles, server: &str) -> String { fn homonyms_of(catalogue: &[repo::scene::SeriesTitles], plain: &[String]) -> Homonyms { let mut by_title: HashMap> = HashMap::new(); for (s, title) in catalogue.iter().zip(plain) { - if let Some(id) = series::client_id_of(s.tvdb, s.tmdb, s.fankai) { + if let Some(id) = client_id(s) { by_title.entry(clean_title(title)).or_default().insert(id); } } by_title } +/// The id Sonarr keeps a series under, as [`series::client_id`] tells it: a +/// series entered by hand that nothing elsewhere names is listed under the +/// number made of its catalogue id, as it is served under it. +fn client_id(s: &repo::scene::SeriesTitles) -> Option { + series::client_id_of(s.tvdb, s.tmdb, s.fankai).or_else(|| s.is_manual.then(|| ids::own(&s.id))) +} + /// Whether another series than `tvdb_id` is served the plain title `plain`. fn shares(homonyms: &Homonyms, tvdb_id: i64, plain: &str) -> bool { homonyms @@ -520,7 +527,7 @@ pub fn held( .into_iter() .zip(plain) .filter_map(|(s, plain)| { - let tvdb_id = series::client_id_of(s.tvdb, s.tmdb, s.fankai)?; + let tvdb_id = client_id(&s)?; let served = served_title(&s, &plain, tvdb_id, &homonyms); let original = s.original_language.clone().unwrap_or_default(); @@ -1159,6 +1166,37 @@ mod tests { assert!(held(vec![work], "en", "en", true).is_empty()); } + /// A series entered by hand that nothing elsewhere names is mapped under + /// the number it is served under, and told apart from a homonym by its + /// year as one TheTVDB does not list is. + #[test] + fn a_series_entered_by_hand_is_mapped_under_the_number_it_is_handed() { + let mut by_hand = stored("Vicineko", false, &[]); + by_hand.id = "01a17789-b167-736d-8873-605960d7e943".into(); + by_hand.is_manual = true; + by_hand.tvdb = None; + by_hand.year = Some(2024); + let mut listed = stored("Vicineko", false, &[]); + listed.tvdb = Some(81189); + + let held = held(vec![by_hand, listed], "en-US", "en-US", true); + + let mapped: Vec<(i64, &str)> = held + .iter() + .map(|h| (h.tvdb_id, h.served.as_str())) + .collect(); + assert_eq!( + mapped, + [ + ( + ids::own("01a17789-b167-736d-8873-605960d7e943"), + "Vicineko (2024)" + ), + (81189, "Vicineko"), + ] + ); + } + #[test] fn thexems_names_are_read_as_sonarr_reads_them() { let body = br#"{"result":"success","data":{"412806":[{"Fuufu Ijou, Koibito Miman":-1},{"FuuKoi":-1}],"x":[{"y":1}]}}"#; diff --git a/src/service/series.rs b/src/service/series.rs index 41363eb..56ca96b 100644 --- a/src/service/series.rs +++ b/src/service/series.rs @@ -20,8 +20,9 @@ const RESOLVING: &[&str] = &[names::TMDB, names::TVDB, names::SKYHOOK, names::FA /// Resolve a series by the id Sonarr asked for. /// -/// The id is either a real TVDB id or one this server synthesised for a -/// TMDB-only show or a Fan-Kai production — see [`crate::service::ids`]. +/// The id is either a real TVDB id or one this server made: for a TMDB-only +/// show, a Fan-Kai production, or a series entered by hand that nothing +/// elsewhere names — see [`crate::service::ids`]. /// /// No language: a work is stored once, in the language the server fetches in, /// and [`crate::service::language`] overlays the one the caller asked for on @@ -29,8 +30,14 @@ const RESOLVING: &[&str] = &[names::TMDB, names::TVDB, names::SKYHOOK, names::FA /// /// `None` only when every source that could resolve the id answered that /// there is no such series; an error when nothing is held and a source did -/// not answer, which a client is told as a 502 (see [`resolved`]). +/// not answer, which a client is told as a 502 (see [`resolved`]). The id of +/// a series entered by hand is the catalogue's alone: `None` at once when no +/// such series is held any more, since no source could answer for it. pub async fn by_client_id(state: &AppState, requested_id: i64) -> Result> { + if ids::is_own(requested_id) { + return crate::service::by_own_id(state, MediaKind::Series, requested_id).await; + } + if let Some(fankai_id) = ids::from_fankai(requested_id) { return by_fankai_id(state, fankai_id).await; } @@ -351,8 +358,10 @@ async fn by_anime_id( pub async fn search(state: &AppState, term: &str) -> Result> { // A prefixed term is a lookup, not a search: answer with the one match. match ids::classify(term) { + // By whatever id Sonarr keeps a series under, as Sonarr itself looks + // a `tvdb:` term up: one this server made among them. ids::TermLookup::Tvdb(id) => { - return Ok(by_tvdb_id(state, id).await?.into_iter().collect()); + return Ok(by_client_id(state, id).await?.into_iter().collect()); } ids::TermLookup::Tmdb(id) => return Ok(by_tmdb_id(state, id).await?.into_iter().collect()), ids::TermLookup::Imdb(id) => { @@ -598,11 +607,23 @@ fn same_id(a: Option, b: Option) -> bool { /// The id a client should be handed for this series. /// /// A real TVDB id when there is one; otherwise a synthetic id derived from -/// TMDB's, or from Fankai's for a Fan-Kai. Returns `None` for a work the client -/// has no way to address. +/// TMDB's, or from Fankai's for a Fan-Kai; for a series entered by hand that +/// none of those names, one made of the catalogue's own id ([`ids::own`]). +/// Returns `None` for a work the client has no way to address. pub fn client_id(item: &MediaItem) -> Option { - let ids = &item.external_ids; - client_id_of(ids.tvdb, ids.tmdb, ids.fankai) + ids::for_client( + MediaKind::Series, + &item.id, + item.is_manual, + &item.external_ids, + ) +} + +/// The id Sonarr is handed for each of `items`, as [`client_id`] gives it, +/// in their order — but a series entered by hand only under a number that +/// leads back to it: see [`crate::service::addressed`]. +pub async fn client_ids(state: &AppState, items: &[MediaItem]) -> Result>> { + crate::service::addressed(state, MediaKind::Series, items, client_id).await } /// The title Sonarr is given for a series: its own — a locked one exactly as @@ -634,10 +655,10 @@ pub fn sonarr_title( } } -/// [`client_id`], from the three ids it is chosen among. +/// [`client_id`], from the three ids it is chosen among: what a series a +/// source gave is handed. pub fn client_id_of(tvdb: Option, tmdb: Option, fankai: Option) -> Option { - tvdb.or_else(|| tmdb.and_then(ids::to_synthetic)) - .or_else(|| fankai.and_then(ids::to_fankai)) + ids::series_named(tvdb, tmdb, fankai) } #[cfg(test)] @@ -728,6 +749,18 @@ mod tests { assert_eq!(client_id(&series(None, None)), None); } + #[test] + fn a_series_entered_by_hand_that_nothing_names_is_handed_a_number_of_its_own() { + let mut item = series(None, None); + item.is_manual = true; + let handed = client_id(&item).expect("an id"); + assert_eq!(handed, ids::own(&item.id)); + assert!(ids::is_own(handed)); + // Given an id elsewhere since, it is handed that one. + item.external_ids.tvdb = Some(81189); + assert_eq!(client_id(&item), Some(81189)); + } + #[test] fn merging_skips_works_already_in_the_list() { let mut results = vec![series(Some(1), Some(10))]; diff --git a/src/settings/reference.rs b/src/settings/reference.rs index d6cb4b7..192e56d 100644 --- a/src/settings/reference.rs +++ b/src/settings/reference.rs @@ -585,13 +585,22 @@ const GROUPS: &[Group] = &[ "media.audio", "The themes: a Fan-Kai's, a few megabytes each.", ), + set( + "media.webp", + "Make the WebP copies the pages ask for, at the widths they ask for them — the \ + first as a picture is stored, any other the first time a page asks — and give \ + the pages those rather than the pictures themselves. Off until it is set, and \ + the pages are given the originals, whole; the copies made while it was on are \ + kept until an administrator deletes them. See \ + [The copies the pages are given](media.md#the-copies-the-pages-are-given).", + ), set( "media.avif", "Make an AVIF copy of every poster, at 480 and 960 pixels, and of every \ - background, at 1280, each night at 03:00 UTC — the task `media.avif`, also run \ - by hand from **Tasks**. A third lighter than the WebP the interface is given \ - otherwise, and a second or two of a processor each to make: off until it is \ - set. See [AVIF](media.md#avif).", + background, at 1280 and 1920, each night at 03:00 UTC — the task `media.avif`, also run \ + by hand from **Tasks**. A third lighter than a WebP copy of the same width, and \ + a second or two of a processor each to make: off until it is set. See \ + [AVIF](media.md#avif).", ), ], }, diff --git a/src/settings/registry.rs b/src/settings/registry.rs index 367f146..5860977 100644 --- a/src/settings/registry.rs +++ b/src/settings/registry.rs @@ -496,6 +496,14 @@ pub const REGISTRY: &[Definition] = &[ kind: Kind::Bool, scopes: SERVER_ONLY, }, + Definition { + // The WebP copies of the pictures, made at the widths the pages ask + // for and given to them in the pictures' place: off until somebody + // turns them on, the pages being given the pictures themselves. + key: "media.webp", + kind: Kind::Bool, + scopes: SERVER_ONLY, + }, Definition { // The night's AVIF copies of the posters and the backgrounds: off // until somebody turns them on, as they cost seconds of a processor diff --git a/src/wire/plex.rs b/src/wire/plex.rs index a2c7b36..fb28043 100644 --- a/src/wire/plex.rs +++ b/src/wire/plex.rs @@ -1579,12 +1579,16 @@ fn ratings(item: &MediaItem) -> Vec { /// (`orders::client_names`). fn season_types(item: &MediaItem, orders: &[EpisodeOrder]) -> Vec { let ids = &item.external_ids; + // Without the orders — a match drawn from a search result — told as the + // catalogue tells the aired order's source: a work entered by hand is + // numbered here, `local`, unless TheTVDB lists it. let source = match orders::aired(orders) { Some(aired) => aired.source.clone(), None if ids.tvdb.is_some() => "tvdb".to_string(), + None if item.is_manual => crate::db::repo::order::LOCAL.to_string(), None if ids.tmdb.is_some() => "tmdb".to_string(), None if ids.fankai.is_some() => "fankai".to_string(), - None => "manual".to_string(), + None => crate::db::repo::order::LOCAL.to_string(), }; const AIRED_TITLE: &str = "Aired order"; let mut types = vec![PlexSeasonType { diff --git a/src/wire/sonarr_v2.rs b/src/wire/sonarr_v2.rs index 2e4bb00..8419512 100644 --- a/src/wire/sonarr_v2.rs +++ b/src/wire/sonarr_v2.rs @@ -108,7 +108,8 @@ pub struct ShowResourceV2 { /// Who numbers the episodes: `tvdb`; `tmdb` for a series TMDB numbers — /// one TheTVDB does not list, under an id this server made for it (see /// [`ids`]), or one TheTVDB stopped answering for; `fankai` for a - /// Fan-Kai. + /// Fan-Kai; `local` for a series entered by hand that nothing elsewhere + /// names. pub metadata_source: String, } @@ -603,7 +604,7 @@ fn formats(item: &MediaItem) -> Vec { /// say — TMDB for a series it numbers while TheTVDB does not answer for it — /// and, where they say nothing, where the id Sonarr holds the series under /// comes from: TheTVDB, or this server's own ranges for a work TheTVDB does -/// not list. +/// not list — the catalogue itself, `local`, for one entered by hand. fn metadata_source(client_id: i64, orders: &[EpisodeOrder]) -> &'static str { match orders::aired(orders).map(|order| order.source.as_str()) { Some("tvdb") => return "tvdb", @@ -615,6 +616,8 @@ fn metadata_source(client_id: i64, orders: &[EpisodeOrder]) -> &'static str { "fankai" } else if ids::from_synthetic(client_id).is_some() { "tmdb" + } else if ids::is_own(client_id) { + crate::db::repo::order::LOCAL } else { "tvdb" }