From b2f58b9c8d832c72da5216eb9bb3267c51b766f6 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 12:53:55 +0000 Subject: [PATCH 1/7] =?UTF-8?q?docs:=20specify=20M8=20=E2=80=94=20AI=20gen?= =?UTF-8?q?eration=20via=20fal.ai?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The contract the backend and frontend are built against in parallel: queue API with persisted request ids for restart safety, a separate generation worker queue, data-URI bases, a catalogue that describes each endpoint's parameter dialect, admins named in config, provenance columns on the asset, and exact cost from fal's billing header. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TPmZjuM4q96DFRfU9cHL33 --- docs/m8-ai-generation.md | 364 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 364 insertions(+) create mode 100644 docs/m8-ai-generation.md diff --git a/docs/m8-ai-generation.md b/docs/m8-ai-generation.md new file mode 100644 index 0000000..2ad631c --- /dev/null +++ b/docs/m8-ai-generation.md @@ -0,0 +1,364 @@ +# M8 — AI generation + +**Status:** specified; being built on `claude/keen-fermi-2rffhr`. This document is the +contract the backend and frontend are built against in parallel, so the API section is +exact rather than illustrative. It is updated with what the build actually did once it +lands. + +M8 adds three ways to make new media with fal.ai — **text → image**, **image → image** +from one or more base images, and **image → video** from a start frame (and, where the +model supports one, an end frame). Results are ordinary library assets with +`source = "ai_generated"`, carry enough provenance to be regenerated, and are costed +from fal's own billing header rather than an estimate. + +Research behind this, kept out of the repository because it is long and dated: a read of +gecko-notes' fal code at `6c6f7ac`, a read of fal's published clients +(`@fal-ai/client` 1.11.0-alpha.5, `fal-client` 1.0.3 — the sandbox cannot reach any +fal.ai host, so the generated endpoint types in the JS client are the source for every +parameter name below), and a survey of GAM's own job, ingest, usage and settings code. + +--- + +## What gecko-notes has, and what it does not + +gecko-notes generates **text → image only**, synchronously (`POST https://fal.run/` +inside the request), keeps nothing about a generation except the file and a usage row, +and "regenerate" re-sends the modal's in-memory state. There is no image → image, no +video, no multi-image input, no queue use, and nothing to port for +regenerate-from-stored-prompt. What *is* worth taking: + +- the `Authorization: Key ` raw-httpx approach (fal's clients hide response + headers, and the billing header is the point); +- `compute_fal_cost`'s shape — units from `x-fal-billable-units` × a per-endpoint unit + price, `x-fal-request-id` as the external reference; +- `ModelCatalogEntry` — an admin-managed, global table of endpoints, because fal renames + and retires endpoint ids every few months (`imagen4/preview`, all of `veo3/*` and + `kling-video/v2.1` disappeared between May and October 2026). The catalogue must be + data, not code. + +What must **not** be copied: the price cache filled from fal's *usage* API (needs a +billing-scoped key, only knows endpoints already billed, and each refresh drops prices +outside its window); downloading the whole result into memory before checking its size; +reporting every fal error as a 502 carrying fal's raw JSON; a catalogue `PUT` that cannot +clear a field because it ignores `null`; and no uniqueness on the endpoint id. + +## Decisions + +### fal's queue API for all three kinds, polled from a job +`POST https://queue.fal.run/{endpoint_id}` returns `request_id`, `status_url`, +`response_url` and `cancel_url`; GAM stores all four **in the job payload, committed, +before it polls**. The synchronous endpoint is unsuitable for video (minutes long, no +retry, a dropped connection loses a result that is still billed) and two code paths for +one feature is one too many — an image simply completes on the first or second poll. + +Persisting the request before polling is what makes restart safe: `recover_pending` +re-runs a job from the start, and a job that finds a stored `request_id` resumes polling +that request instead of submitting — and paying for — a second one. The same reasoning +applies after the download: created asset ids are written to the payload as each output +is ingested, so a restart after ingest finishes the job instead of duplicating assets, +and a `usage_recorded` flag keeps the cost from being recorded twice. + +Status `COMPLETED` is also what a *failed* run reports, with `error`/`error_type` set, so +those are checked before fetching the result. Unknown statuses are non-terminal; GAM's +own deadline (`generation_timeout_minutes`) bounds the wait, and on timeout or user +cancel GAM sends `PUT {cancel_url}` (best effort) — a remote job left running keeps +billing. + +Webhooks were rejected: GAM sits behind a reverse proxy with no public callback URL, and +polling from a worker thread needs nothing inbound. + +### Generation gets its own worker queue +Every job shares one `JobQueue` at `enrichment_concurrency = 1`. A video that takes four +minutes would hold every transcription queued behind it. `JobQueue` gains a kind filter +(`kinds` / `exclude_kinds`), applied in `recover_pending` and `sweep_stale` so two queues +over the same `EnrichmentJob` table never recover or sweep each other's rows. The +enrichment queue excludes `generate`; a second queue owns it at +`generation_concurrency = 2`. A generation is almost entirely waiting on fal, so two at +once costs the container nothing. + +### Base images go to fal as data URIs +GAM's media is not public and must not become public to serve this. fal's two options +are a `data:` URI in the request body or an upload to fal's CDN. Data URIs win: +- nothing of the user's lands on a public CDN with no expiry (gecko-notes' audio uploads + do exactly that); +- no second, separately-versioned upload API (fal has already broken the older + `storage_type=gcs` flow once). + +The cost is body size, so each base is normalised first: EXIF orientation applied, long +edge capped at `generation_base_max_edge` (2048 px), re-encoded as JPEG (PNG when it has +transparency), and refused if the result still exceeds `generation_max_data_uri_mb` +(8 MB). Bases must be image assets that own a file and that Pillow can read — not clips, +not avif/heic/svg. **Unverified:** that every seeded model accepts data URIs; fal's +schemas say "publicly accessible or base64 data URI". If one does not, CDN upload is the +fallback, not a redesign. + +### The catalogue describes each endpoint's dialect +Endpoints disagree on everything: the image field (`image_url`, `image_urls[]`, +`start_image_url`), the end-frame field (`tail_image_url`, `end_image_url`, +`last_frame_url`, or none), the type of `duration` (`"5"`, `"8s"`, `6`), and which of +`aspect_ratio` / `image_size` / `resolution` / `seed` / `negative_prompt` exist at all. +Rather than code per model, each catalogue row carries: + +- `image_field`, `image_field_is_list`, `max_images`, `end_image_field`; +- `options` — the *exact values* the endpoint accepts for each user-facing choice + (`aspect_ratios`, `image_sizes`, `durations`, `resolutions`), passed through verbatim so + `"5"` and `"8s"` and `6` all work, plus `supports_seed`, `supports_negative_prompt`, + `supports_audio` and `max_outputs`; +- `extra_params` — defaults merged into every request body first (e.g. + `{"generate_audio": false}` for Veo, whose audio default raises the price by half). + +The form offers only what a row declares, and the server rejects a value outside the +declared list. This also closes gecko-notes' worst catalogue bug: a seeded "text to +image" model that was really an edit endpoint and 422'd every call. + +### Admins are named in config +`User.is_admin` exists and `/api/me` returns it, but nothing sets it and the Notes token +carries no admin claim. `ADMIN_USERS` (comma-separated user ids or usernames, usernames +compared case-insensitively) makes a user an admin; `/api/me` reports +`is_admin = record.is_admin or listed`. An `AdminUser` dependency answers +`403 {"code": "admin_required"}`. Only admins edit the catalogue; everyone uses it. +With `ADMIN_USERS` empty nobody can edit it, and the seeded catalogue still works. + +### Provenance lives on the asset; the base is not a parent +`parent_asset_id` is wrong for a base image: anything with a parent inherits its +attribution (M10), appears in the parent's Clip tab, and one column cannot hold several +bases. New nullable columns on `Asset`: + +| column | holds | +|---|---| +| `ai_model` | the endpoint id | +| `ai_generation_type` | `text_to_image` / `image_to_image` / `image_to_video` | +| `ai_prompt` | the prompt as sent | +| `ai_parameters` | JSON — the request body exactly as sent, minus `prompt` and the image fields (catalogue defaults included, so a regeneration is reproducible after the catalogue changes) | +| `ai_source_assets` | JSON list of `{"asset_id", "role": "base" \| "end_frame", "name"}` — `name` is a snapshot so a deleted base still reads as something | +| `ai_seed` | the seed fal reports back, when it reports one | +| `ai_generated_at` | naive UTC, from `app.clock.utcnow` | + +Bases are referenced by id inside JSON, not by foreign key: deleting a base never blocks +or breaks a generated asset; it only makes regenerating it impossible, which is reported +as such. + +The generated asset's `name` is the prompt cut at a word boundary (≤ 80 chars), and its +`description` is the full prompt, both stamped `"ai"` in `field_provenance`. That makes a +generation findable by what was asked for — keyword and semantic search both read +`description` — without an FTS rebuild migration. A later Describe run overwrites the +description as it would any other; `ai_prompt` keeps the original. + +### Exact cost +fal's `x-fal-billable-units` (on the queue **result** fetch, with `x-fal-request-id`) +gives the billed quantity in the endpoint's unit. The unit price comes from fal's +pricing API, `GET https://api.fal.ai/v1/models/pricing?endpoint_id=…` with the user's key, +cached in-process per (user, endpoint) for six hours. + +| units from | price from | `cost` | `cost_estimated` | +|---|---|---|---| +| header | pricing API | units × price | `False` — the exact bill | +| header | catalogue `unit_price` | units × price | `True` | +| outputs (header absent) | either | quantity derived from what was downloaded (images, megapixels rounded up per image, probed seconds, videos) × price | `True` | +| — | no price anywhere | `None` | `None` — "not priced" | + +**Unverified:** that the queue result carries the billing header (one third-party source +says it does; gecko-notes only ever read it from the synchronous endpoint), and that the +pricing API accepts an ordinary key. Both degrade to the estimated rows above rather than +to nothing. The first real generation answers both questions. + +One `UsageEvent` per fal request, attached to the first output asset, with +`provider="fal.ai"`, `model=`, `kind="image"` or `"video"`, and fal's +request id in a new `external_ref` column. `unit_type` is `images`, `megapixels`, +`generated_seconds` or `videos` — deliberately not `seconds`, which the usage totals +already sum as transcribed audio. `units` is an integer column, so it holds the billed +quantity rounded up; `cost` is computed from the exact value first. + +### A generated video is not auto-transcribed +`_chain_transcription` would send every fal video to Deepgram and bill for it, and most +generated video is silent or music. Ingest skips it for `SOURCE_AI`; the Transcribe +button still works by hand. + +--- + +## Schema (one Alembic revision) + +1. `asset`: the seven `ai_*` columns above, all nullable, no indexes, no foreign keys. +2. `usageevent`: `external_ref` (nullable string). +3. `generationmodel` (new): + +| column | type | notes | +|---|---|---| +| `id` | str PK | uuid4 | +| `endpoint_id` | str | **unique** index | +| `kind` | str | indexed; `text_to_image` / `image_to_image` / `image_to_video` | +| `label` | str | | +| `note` | str | default `""` | +| `sort_order` | int | default 0 | +| `is_active` | bool | default true | +| `image_field` | str? | null for text → image | +| `image_field_is_list` | bool | default false | +| `max_images` | int | 0 for text → image, ≥ 1 otherwise | +| `end_image_field` | str? | image → video only | +| `options` | str | JSON, default `"{}"` | +| `extra_params` | str | JSON, default `"{}"` | +| `unit_price` | float? | list price, the fallback when the pricing API is unavailable | +| `price_unit` | str? | `image` / `megapixel` / `second` / `video` | +| `price_currency` | str? | | +| `created_at`, `updated_at` | datetime | naive UTC | + +4. Seed rows, inserted by the same revision, three per kind. Every parameter name and + allowed value is checked against the endpoint types in fal's client before seeding: + - text → image: `fal-ai/flux/schnell`, `fal-ai/flux/dev`, `fal-ai/nano-banana-2` + - image → image: `fal-ai/nano-banana/edit`, `fal-ai/bytedance/seedream/v4.5/edit` + (≤ 10 bases), `fal-ai/flux-pro/kontext` (one base) + - image → video: `fal-ai/kling-video/v2.5-turbo/pro/image-to-video` (start + end + frame), `fal-ai/minimax/hailuo-02/standard/image-to-video` (start + end, cheap), + `fal-ai/veo3.1/fast/image-to-video` (`generate_audio: false` by default) + +--- + +## API + +Envelopes and errors follow the house contract (`{"data": …}`, +`{"detail": {"code", "message"}}`). + +### Catalogue +`GenerationModelRead`: +```json +{"id": "…", "endpoint_id": "fal-ai/flux/dev", "kind": "text_to_image", + "label": "FLUX.1 [dev]", "note": "", "sort_order": 10, "is_active": true, + "image_field": null, "image_field_is_list": false, "max_images": 0, + "end_image_field": null, + "options": {"aspect_ratios": [], "image_sizes": ["square_hd", "landscape_16_9"], + "durations": [], "resolutions": [], "supports_seed": true, + "supports_negative_prompt": false, "supports_audio": false, + "max_outputs": 4}, + "extra_params": {}, "unit_price": 0.025, "price_unit": "megapixel", + "price_currency": "USD", "created_at": "…", "updated_at": "…"} +``` +- `GET /api/generate/models?include_inactive=false` → list envelope, ordered by + `kind, sort_order, label`. `include_inactive` is honoured for admins only. +- `POST /api/generate/models` *(admin)* → `201`, `GenerationModelRead`. + `409 endpoint_exists` on a duplicate `endpoint_id`. +- `PATCH /api/generate/models/{id}` *(admin)* → updates exactly the fields present in + the body; an explicit `null` clears a nullable field. +- `DELETE /api/generate/models/{id}` *(admin)* → `204`. +- Validation (`422 invalid_model_entry`): `kind` in the three values; `max_images` 0 and + no `image_field` for text → image, `max_images ≥ 1` and an `image_field` otherwise; + `end_image_field` only for image → video; `max_outputs` 1–4 (1 for video); `options` + and `extra_params` are objects. + +### The fal key +`GET /api/settings/generation` → `{"data": {"fal_key_configured": bool}}`. +`PUT /api/settings/generation` with `{"fal_api_key": "…"}` stores it encrypted (setting +key `fal_api_key`, in `settings_store.SECRET_KEYS`); `""` removes it; omitted leaves it +alone. The key is never returned, masked or otherwise. + +### Generating +`POST /api/generate` → `202 {"data": ActivityJobRead}` +```json +{"model_id": "…", "prompt": "…", + "base_asset_ids": ["…"], "end_frame_asset_id": null, + "params": {"aspect_ratio": null, "image_size": null, "duration": null, + "resolution": null, "seed": null, "negative_prompt": null, + "generate_audio": null, "num_outputs": 1}} +``` +Checked at submit time, in this order: +- `400 fal_key_missing` — no key stored. +- `404 model_not_found`; `409 model_inactive`. +- `422 invalid_generation` with a specific message, for: + - an empty prompt or one over 4,000 characters; + - the wrong number of bases for the kind (none for text → image, 1–`max_images` for + image → image, exactly one for image → video); + - an end frame on a model without `end_image_field`; + - a choice outside the row's declared list; + - `seed` / `negative_prompt` / `generate_audio` on a model that does not declare it; + - `num_outputs` outside 1–`max_outputs`. +- `404 asset_not_found` for a base the user does not own. +- `422 invalid_base` for a base that is not an image, owns no file, or is not readable. + +The job's `asset_name` is the prompt, truncated. `ActivityJobRead` gains +`result_asset_ids: list[str]` (every created asset, in output order); `result_asset_id` +stays the first. + +`POST /api/generate/{asset_id}/regenerate` → `202 {"data": ActivityJobRead}`, body +`{"prompt": null, "reuse_seed": false}`. Re-submits the stored endpoint, parameters and +sources (optionally with a new prompt, and with `ai_seed` as `seed` when `reuse_seed` and +the model supports seeds). +- `404` when the asset is not owned. +- `409 not_generated` when the asset has no stored generation. +- `409 model_unavailable` when no active catalogue row has that endpoint id. +- `409 source_asset_missing` when a base was deleted, naming it. + +### Asset read model +`AssetRead` gains `generation` — `null`, or: +```json +{"model": "fal-ai/flux/dev", "kind": "text_to_image", "prompt": "…", + "parameters": {…}, "sources": [{"asset_id": "…", "role": "base", "name": "…"}], + "seed": 1234, "generated_at": "…"} +``` + +### Activity +Action `generate`. Stages, in order: `Preparing the base images`, `Submitting to fal.ai`, +`Waiting in fal.ai's queue` (detail: queue position), `Generating`, `Downloading`, +`Saving`, `Recording the cost`. Errors are sentences a user can act on — +`fal.ai rejected the API key — check it in Settings`, `fal.ai refused the request: +image_urls: field required`, `fal.ai did not finish within 30 minutes`, `A base image +was deleted before this could run`, `The generated file is larger than the 1024 MB limit`. + +--- + +## Configuration (`app/config.py`) + +| setting | default | | +|---|---|---| +| `admin_users` | `""` | comma-separated user ids or usernames | +| `fal_queue_base_url` | `https://queue.fal.run` | | +| `fal_api_base_url` | `https://api.fal.ai` | pricing | +| `generation_concurrency` | `2` | the generation queue's workers | +| `generation_timeout_minutes` | `30` | then cancel at fal and fail | +| `generation_poll_seconds` | `3` | | +| `generation_max_download_mb` | `1024` | per output, enforced while streaming | +| `generation_base_max_edge` | `2048` | base image long edge, px | +| `generation_max_data_uri_mb` | `8` | per base, after normalising | + +Result downloads stream to a temp file through a new capped helper: SSRF check +(`safe_url.require_safe_external_url`) on the URL and on each redirect hop (at most +three), `Content-Length` checked up front, bytes counted while streaming, the job's +progress callback called between chunks so a cancel lands mid-download. The extension +comes from the URL path when it is a known media type, else from `Content-Type`, else +`.png` / `.mp4` by kind. + +## Frontend + +- `api/generate.ts` — types beside it, `generateApi = { models, createModel, updateModel, + deleteModel, start, regenerate }`. `api/settings.ts` gains `generationSettingsApi`. + `Asset` gains `generation`; `User` gains `is_admin`. +- `stores/generation.ts` — the catalogue, loaded once, with `reset()` fanned out on + logout. +- `components/GenerateForm.tsx` — one form, three entry points. Kind is chosen from what + the bases allow (none → text → image; one or more → image → image, or image → video + with the second base as the end frame when the model has one). The model list is + filtered by kind, with its note and list price ("≈ $0.025 per megapixel, list price"). + Every option control renders only when the selected model declares it. Submitting + queues the job and says to follow it under background activity, as `UrlImport` does; + `fal_key_missing` says to add a key in Settings. +- Entry points: a third section in `AddPanel` (text → image); "Generate from these" in + `SelectionBar` when every selected asset is an image that owns a file; and a + **Generate** tab in `AssetDetail` for any image asset with a file and any asset with a + `generation`. On a generated asset that tab leads with the provenance (model, prompt, + parameters, sources linked to `/a/{id}`, seed, cost) and **Regenerate**, + **Regenerate with the same seed** (when there is one) and **Edit and generate**, which + opens the form pre-filled. +- `LibraryView` adds every `result_asset_ids` entry of a finished `generate` job to the + grid, the way it does an import. `ActivityIndicator` labels it "Generating". +- Settings: a **Generation** panel with the fal key (the `SpeechPanel` pattern) and the + model catalogue — editable for admins, read-only for everyone else. +- The asset Info tab's "AI cost (est.)" and `UsagePanel`'s "an estimate, not a bill" + respect `estimated`, now that an exact cost exists. + +## Known limitations at the outset + +- No picker for bases beyond "what is selected" and "this asset" — the embeddable picker + is M9. +- A video's frame cannot be a base; only image assets can. +- Generated outputs keep fal's copy for fal's default retention (≥ 7 days). Deleting it + sooner needs fal's payload-deletion API and a billing-scoped key. +- Cost is reconciled against nothing after the fact; fal's billing-events API would give + the discounted final amount and is the natural follow-up. From 90316b061b4f160eb41cb5ee55dceb5f7a761e65 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 13:11:56 +0000 Subject: [PATCH 2/7] fix(clips): keep a poster a surviving row still shows; delete a parent with its clips A clip copies its parent's thumb_key, a URL import's chapter clips all copy the site thumbnail, and a promote whose own thumbnail failed keeps the copy. delete_asset unlinked whatever keys the deleted row held, so deleting one clip deleted the poster of the parent and every sibling clip, and deleting a parent took the poster of a promoted child it had just let go. A key is now unlinked only once no surviving row of that user references it. DELETE /api/assets/{id}?with_clips=true deletes the parent's live clips and then the parent in one transaction; without the flag the 409 still stands, so an older client never cascades by accident. A delete is refused with 409 extraction_in_progress while any row it would remove has a queued or running extract_subvideo job, which would otherwise finish into a missing row after ffmpeg had written an unreferenced file. The 409 message now says "1 clip depends". Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TPmZjuM4q96DFRfU9cHL33 --- backend/app/routers/assets.py | 42 +++++- backend/app/services/assets.py | 111 +++++++++++---- backend/tests/test_clips_api.py | 234 +++++++++++++++++++++++++++++++- 3 files changed, 356 insertions(+), 31 deletions(-) diff --git a/backend/app/routers/assets.py b/backend/app/routers/assets.py index 06e6e20..35865f1 100644 --- a/backend/app/routers/assets.py +++ b/backend/app/routers/assets.py @@ -15,7 +15,7 @@ from app.jobs import enrichment as enrichment_jobs from app.jobs.registry import KINDS from app.models.asset import Asset -from app.models.job import KIND_HARVEST_ATTRIBUTION +from app.models.job import KIND_EXTRACT_SUBVIDEO, KIND_HARVEST_ATTRIBUTION from app.models.tag import Tag from app.schemas import DataResponse, ListResponse from app.schemas_assets import AssetRead, AssetUpdate, UploadRejection, UploadResult @@ -346,28 +346,58 @@ def update_asset( def delete_asset( asset_id: str, user: CurrentUser, + with_clips: bool = Query(default=False), session: Session = Depends(get_session), storage: LocalStorage = Depends(get_storage), ) -> None: + """Delete an asset. For a clip that is only the clip: it owns no bytes, and the + poster it shows is its parent's file (see `services/assets.delete_asset`). + + `with_clips` is the guard's third answer, beside "promote them" and "cancel": the + live clips cut from this asset go with it, in one transaction. It fails safe — + FastAPI ignores a query parameter it does not declare, so a frontend ahead of an + older backend gets the 409 below, never a silent cascade. + """ asset = _owned(asset_id, user.id, session) # Checked here, before `delete_asset` touches anything: a live clip (M7) is meant # to block this, and reporting how many rather than just refusing is what lets the - # frontend offer "promote them, then delete" instead of a dead end. + # frontend offer "promote them, or delete them too" instead of a dead end. blocking = service.blocking_clips(session, asset.id) - if blocking: + if blocking and not with_clips: + n = len(blocking) raise HTTPException( status_code=status.HTTP_409_CONFLICT, detail={ "code": "asset_has_dependent_clips", "message": ( - f"{len(blocking)} clip{'' if len(blocking) == 1 else 's'} depend on " - "this asset. Extract them as sub-videos first." + f"{n} clip{'' if n == 1 else 's'} depend{'s' if n == 1 else ''} on " + "this asset. Extract them as files first, or delete them with it." + ), + }, + ) + clips = blocking if with_clips else [] + + # A promote or an extract still queued or running against a row about to go would + # finish into a row that no longer exists: `promote_clip` updates nothing, or + # `create_subvideo_asset`'s insert trips the parent foreign key — either way after + # ffmpeg has already written a file nothing will ever reference. + if any( + enrichment_jobs.active_job(session, row.id, KIND_EXTRACT_SUBVIDEO) is not None + for row in (asset, *clips) + ): + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail={ + "code": "extraction_in_progress", + "message": ( + "An extraction is still running on this asset or one of its clips. " + "Let it finish, or cancel it, then delete." ), }, ) - service.delete_asset(session, storage, asset) + service.delete_asset(session, storage, asset, clips=clips) # ─── tagging ───────────────────────────────────────────────────────────────── diff --git a/backend/app/services/assets.py b/backend/app/services/assets.py index 7e28627..9742037 100644 --- a/backend/app/services/assets.py +++ b/backend/app/services/assets.py @@ -11,7 +11,7 @@ import logging import mimetypes from pathlib import Path -from typing import Any, AsyncIterator, Iterable, Optional +from typing import Any, AsyncIterator, Iterable, Optional, Sequence from sqlalchemy import and_, func, not_, or_ from sqlalchemy.orm import aliased @@ -358,12 +358,55 @@ def blocking_clips(session: Session, asset_id: str) -> list[Asset]: ) -def delete_asset(session: Session, storage: LocalStorage, asset: Asset) -> None: - """Remove the row, everything that hangs off it, and the bytes it owns. +def _delete_dependents(session: Session, asset_id: str) -> None: + """The rows hanging off one asset that must go before it does (see `delete_asset`). + + Does not commit: the caller deletes the asset itself in the same transaction, and a + commit here would leave a half-stripped row behind if that delete then failed. + """ + tags.detach_all_from_asset(session, asset_id) + session.exec(delete(Suggestion).where(col(Suggestion.asset_id) == asset_id)) + session.exec(delete(TranscriptSegment).where(col(TranscriptSegment.asset_id) == asset_id)) + session.exec(delete(DocumentPage).where(col(DocumentPage.asset_id) == asset_id)) + + +def _still_referenced(session: Session, user_id: str, key: str) -> bool: + """Whether any surviving row still points at this storage object. + + A key is not owned by exactly one row. A clip's `thumb_key` is a copy of its + parent's (`create_clip`), every chapter clip of a URL import copies the site + thumbnail written over that same key (`import_url._apply_thumbnail`), and a promote + whose own thumbnail failed keeps the copy (`promote_clip`). So "this row is gone, + unlink its keys" deleted the poster the parent and every sibling clip were still + showing — or, deleting the parent, the poster of a sub-video it had just let go. + + Asked per key rather than special-casing clips as owning no keys, because that + rule is wrong the other way round: the promoted survivor above is not a clip, and + it is the parent's delete that has to leave the file alone. Scoped by `user_id` + because keys are per-user-prefixed and that column is indexed; neither key column + is. + """ + return bool( + session.exec( + select(func.count()) + .select_from(Asset) + .where( + Asset.user_id == user_id, + or_(col(Asset.storage_key) == key, col(Asset.thumb_key) == key), + ) + ).one() + ) + + +def delete_asset( + session: Session, storage: LocalStorage, asset: Asset, *, clips: Sequence[Asset] = () +) -> None: + """Remove the row, everything that hangs off it, and the bytes only it was using. The row goes first. If the unlink fails the asset is still gone from the user's view, and an orphaned file is recoverable by a sweep; the reverse — a row pointing - at nothing — is what produces broken images. + at nothing — is what produces broken images. "Only it was using" is + `_still_referenced`: a poster is routinely shared between a parent and its clips. The dependent rows have to go before it, for two different reasons. Tag attachments are a hard constraint: `AssetTag.asset_id` is a foreign key with no @@ -381,15 +424,25 @@ def delete_asset(session: Session, storage: LocalStorage, asset: Asset) -> None: expected to have already called `blocking_clips` and refused the delete if any *live* clip depends on this asset, but a survivor's mere breadcrumb still has to be cleared, or the same foreign key raises on it the moment this row is gone. - """ - storage_key, thumb_key = asset.storage_key, asset.thumb_key + `clips` are live clips of `asset` to delete along with it — the guard's "Delete + clips too". They go in the same transaction, and *first*: their `parent_asset_id` + is the foreign key that would otherwise refuse the parent's delete. A bulk statement + rather than `session.delete` per row, because nothing tells the unit of work that + rows of one table depend on each other, so it is free to flush the parent first. + The router only ever passes `blocking_clips(asset)`, so a promoted or extracted + child is never among them; those keep their bytes and only lose the breadcrumb, as + before. + """ + doomed = [*clips, asset] + doomed_ids = [row.id for row in doomed] + keys = {key for row in doomed for key in (row.storage_key, row.thumb_key) if key} asset_id, user_id = asset.id, asset.user_id - tags.detach_all_from_asset(session, asset_id) - session.exec(delete(Suggestion).where(col(Suggestion.asset_id) == asset_id)) - session.exec(delete(TranscriptSegment).where(col(TranscriptSegment.asset_id) == asset_id)) - session.exec(delete(DocumentPage).where(col(DocumentPage.asset_id) == asset_id)) + for row_id in doomed_ids: + _delete_dependents(session, row_id) + if clips: + session.exec(delete(Asset).where(col(Asset.id).in_([clip.id for clip in clips]))) session.exec(update(Asset).where(col(Asset.parent_asset_id) == asset_id).values(parent_asset_id=None)) session.delete(asset) session.commit() @@ -398,21 +451,31 @@ def delete_asset(session: Session, storage: LocalStorage, asset: Asset) -> None: # is already gone, so raising here would abort the rest of the cleanup and leave # more behind than it removed. A stale index entry is recoverable; a half-finished # delete is the thing this function exists to avoid. - try: - # Drops its own transaction and invalidates the per-user vector cache, which - # must not happen while the asset could still come back. - vectors.remove_for_asset(session, user_id, asset_id) - except Exception: # noqa: BLE001 - see above - logger.warning("Could not drop embeddings for asset %s", asset_id, exc_info=True) - - try: - fts.remove_asset(session, asset_id) - except Exception: # noqa: BLE001 - logger.warning("Could not un-index asset %s", asset_id, exc_info=True) + for row_id in doomed_ids: + try: + # Drops its own transaction and invalidates the per-user vector cache, which + # must not happen while the asset could still come back. + vectors.remove_for_asset(session, user_id, row_id) + except Exception: # noqa: BLE001 - see above + logger.warning("Could not drop embeddings for asset %s", row_id, exc_info=True) - for key in (storage_key, thumb_key): - if key: - storage.delete(key) + try: + fts.remove_asset(session, row_id) + except Exception: # noqa: BLE001 + logger.warning("Could not un-index asset %s", row_id, exc_info=True) + + # After the commit, so the check sees the rows that survived and not the ones just + # deleted — a poster only these rows shared goes, one a survivor still shows stays. + # A check that cannot run keeps the file: an orphan is what a sweep is for, and an + # unlinked file that something still shows is not recoverable at all. + for key in keys: + try: + if _still_referenced(session, user_id, key): + continue + except Exception: # noqa: BLE001 - see above + logger.warning("Could not check whether %s is still in use", key, exc_info=True) + continue + storage.delete(key) def apply_metadata(session: Session, asset: Asset, changes: dict) -> Asset: diff --git a/backend/tests/test_clips_api.py b/backend/tests/test_clips_api.py index 0909098..72620cb 100644 --- a/backend/tests/test_clips_api.py +++ b/backend/tests/test_clips_api.py @@ -3,6 +3,7 @@ from pathlib import Path import pytest +from sqlalchemy import text from sqlmodel import select from app.enrichment import source @@ -11,7 +12,10 @@ from app.jobs import enrichment as enrichment_jobs from app.models.asset import Asset from app.models.job import EnrichmentJob, KIND_EXTRACT_SUBVIDEO +from app.models.tag import AssetTag +from app.search import fts from app.services import assets as asset_service +from app.storage import thumb_key_for from app.storage.base import StoredFile from app.media_tools import ffmpeg_available @@ -49,6 +53,36 @@ def _fake_probe(duration: float = 9.0) -> ProbeResult: return ProbeResult(duration_seconds=duration, width=320, height=240, codec="h264") +def _clip(client, parent_id: str, in_point: float = 0.0, out_point: float = 0.5, **fields): + return client.post( + f"/api/assets/{parent_id}/clips", + json={"in_point": in_point, "out_point": out_point, **fields}, + ).json()["data"] + + +def _files(media_dir) -> set[str]: + return {str(p.relative_to(media_dir)) for p in Path(media_dir).rglob("*") if p.is_file()} + + +def _give_a_poster(session, media_dir, asset_id: str) -> Path: + """Put a poster on disk for this asset, so clips cut afterwards copy its key. + + By hand, so the shared-key path runs whether or not ffmpeg is on PATH — the same + reason `test_clip_inherits_the_parents_thumbnail` sets one. Written at the key the + upload's own poster uses, not a made-up one: a poster ffmpeg did generate is then + overwritten rather than orphaned beside it, where a "nothing left on disk" + assertion would trip over it. + """ + row = session.get(Asset, asset_id) + row.thumb_key = thumb_key_for(row.storage_key) + path = Path(media_dir) / row.thumb_key + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(b"\xff\xd8poster") + session.add(row) + session.commit() + return path + + # ─── creating a clip (synchronous, no ffmpeg needed) ────────────────────────── @@ -388,7 +422,15 @@ def test_delete_message_counts_more_than_one_clip(library): library.post(f"/api/assets/{parent['id']}/clips", json={"in_point": 0.3, "out_point": 0.5}) body = library.delete(f"/api/assets/{parent['id']}").json()["detail"] - assert "2" in body["message"] + assert "2 clips depend on" in body["message"] + + +def test_delete_message_says_depends_for_one_clip(library): + parent = _upload_video(library) + _clip(library, parent["id"]) + + body = library.delete(f"/api/assets/{parent['id']}").json()["detail"] + assert "1 clip depends on" in body["message"] def test_delete_succeeds_with_no_clips(library): @@ -438,6 +480,196 @@ def test_delete_still_blocked_if_only_some_clips_are_promoted(library, session): assert "1" in response.json()["detail"]["message"] +def test_deleting_a_parent_keeps_its_promoted_childs_poster(library, session, media_dir): + """A promote whose own thumbnail failed keeps the parent's key (`promote_clip`'s + fallback), so the survivor's poster is the parent's file. Deleting the parent used + to unlink it, leaving the sub-video it just let through showing nothing.""" + parent = _upload_video(library) + poster = _give_a_poster(session, media_dir, parent["id"]) + promoted = _clip(library, parent["id"]) + asset_service.promote_clip( + session, + session.get(Asset, promoted["id"]), + stored=_fake_stored(), + thumb_key=None, + probe_result=_fake_probe(), + ) + + assert library.delete(f"/api/assets/{parent['id']}").status_code == 204 + + assert poster.exists() + survivor = library.get(f"/api/assets/{promoted['id']}").json()["data"] + assert poster.name in survivor["thumb_url"] + + +# ─── deleting a parent with its clips ───────────────────────────────────────── + + +def test_delete_with_clips_takes_the_live_clips_and_the_parent(library, session, media_dir): + parent = _upload_video(library) + _give_a_poster(session, media_dir, parent["id"]) + first = _clip(library, parent["id"], 0.0, 0.5) + second = _clip(library, parent["id"], 0.5, 1.0) + # Tagged, because `AssetTag` is a foreign key: a clip deleted without its tags + # detached first is a 500, not a 204. + library.post(f"/api/assets/{first['id']}/tags", json={"names": ["chapter"]}) + + response = library.delete(f"/api/assets/{parent['id']}", params={"with_clips": "true"}) + assert response.status_code == 204 + + for row in (parent, first, second): + assert library.get(f"/api/assets/{row['id']}").status_code == 404 + # The file and the shared poster both went: once the parent and every clip were + # gone, nothing referenced either. + assert _files(media_dir) == set() + + +def test_delete_with_clips_keeps_a_promoted_child(library, session, media_dir): + """`with_clips` deletes exactly what `blocking_clips` returns. A promoted child owns + its own bytes and only loses the breadcrumb — and keeps the poster it shares.""" + parent = _upload_video(library) + poster = _give_a_poster(session, media_dir, parent["id"]) + live = _clip(library, parent["id"], 0.0, 0.5) + promoted = _clip(library, parent["id"], 0.5, 1.0) + asset_service.promote_clip( + session, + session.get(Asset, promoted["id"]), + stored=_fake_stored(), + thumb_key=None, + probe_result=_fake_probe(), + ) + + response = library.delete(f"/api/assets/{parent['id']}", params={"with_clips": "true"}) + assert response.status_code == 204 + + assert library.get(f"/api/assets/{live['id']}").status_code == 404 + session.expire_all() + survivor = session.get(Asset, promoted["id"]) + assert survivor is not None + assert survivor.parent_asset_id is None + assert survivor.source == "sub_video" + assert poster.exists() + + +def test_delete_with_clips_is_a_plain_delete_when_there_are_none(library): + asset = _upload_video(library) + + response = library.delete(f"/api/assets/{asset['id']}", params={"with_clips": "true"}) + assert response.status_code == 204 + assert library.get(f"/api/assets/{asset['id']}").status_code == 404 + + +def test_delete_with_clips_is_refused_while_one_is_being_promoted(library): + """A promote that finished into a deleted row would write a file nothing points at. + Refused whole: the parent and both clips are still there afterwards.""" + parent = _upload_video(library) + first = _clip(library, parent["id"], 0.0, 0.5) + second = _clip(library, parent["id"], 0.5, 1.0) + # Stays queued: the autouse `no_background_workers` fixture keeps the worker off. + assert library.post(f"/api/assets/{first['id']}/promote").status_code == 202 + + response = library.delete(f"/api/assets/{parent['id']}", params={"with_clips": "true"}) + assert response.status_code == 409 + assert response.json()["detail"]["code"] == "extraction_in_progress" + + for row in (parent, first, second): + assert library.get(f"/api/assets/{row['id']}").status_code == 200 + + +def test_delete_is_refused_while_a_sub_video_is_being_cut_from_it(library): + asset = _upload_video(library) + library.post(f"/api/assets/{asset['id']}/subvideo", json={"in_point": 0.0, "out_point": 0.5}) + + response = library.delete(f"/api/assets/{asset['id']}") + assert response.status_code == 409 + assert response.json()["detail"]["code"] == "extraction_in_progress" + assert library.get(f"/api/assets/{asset['id']}").status_code == 200 + + +def test_delete_with_clips_of_someone_elses_asset_is_a_404(library, session): + other = Asset(user_id="somebody-else", name="Theirs", asset_type="video", source="local_upload") + session.add(other) + session.commit() + their_clip = Asset( + user_id="somebody-else", + name="Their clip", + asset_type="video", + source="clip", + parent_asset_id=other.id, + in_point=0.0, + out_point=1.0, + ) + session.add(their_clip) + session.commit() + + response = library.delete(f"/api/assets/{other.id}", params={"with_clips": "true"}) + assert response.status_code == 404 + assert session.get(Asset, other.id) is not None + assert session.get(Asset, their_clip.id) is not None + + +# ─── deleting a clip ─────────────────────────────────────────────────────────── + + +def test_deleting_a_clip_removes_only_the_clip(library, session, media_dir): + """A clip owns no bytes, and the poster it shows is its parent's: `create_clip` + copies the key. Deleting one used to unlink that key, blanking the poster of the + parent and of every sibling clip with it.""" + parent = _upload_video(library) + poster = _give_a_poster(session, media_dir, parent["id"]) + doomed = _clip(library, parent["id"], 0.0, 0.5) + sibling = _clip(library, parent["id"], 0.5, 1.0) + library.post(f"/api/assets/{doomed['id']}/tags", json={"names": ["chapter"]}) + before = _files(media_dir) + + assert library.delete(f"/api/assets/{doomed['id']}").status_code == 204 + + assert library.get(f"/api/assets/{doomed['id']}").status_code == 404 + assert library.get(f"/api/assets/{parent['id']}").status_code == 200 + assert not session.exec(select(AssetTag).where(AssetTag.asset_id == doomed["id"])).all() + # Nothing on disk changed: the parent's file and its poster are both still there. + assert _files(media_dir) == before + fetched = library.get(f"/api/assets/{sibling['id']}").json()["data"] + assert poster.name in fetched["thumb_url"] + assert poster.exists() + + +def test_deleting_the_last_clip_unblocks_the_parent(library): + parent = _upload_video(library) + clip = _clip(library, parent["id"]) + assert library.delete(f"/api/assets/{parent['id']}").status_code == 409 + + assert library.delete(f"/api/assets/{clip['id']}").status_code == 204 + assert library.delete(f"/api/assets/{parent['id']}").status_code == 204 + + +def test_delete_is_refused_while_the_clip_is_being_promoted(library): + parent = _upload_video(library) + clip = _clip(library, parent["id"]) + assert library.post(f"/api/assets/{clip['id']}/promote").status_code == 202 + + response = library.delete(f"/api/assets/{clip['id']}") + assert response.status_code == 409 + assert response.json()["detail"]["code"] == "extraction_in_progress" + assert library.get(f"/api/assets/{clip['id']}").status_code == 200 + + +def test_a_deleted_clip_leaves_search(library, session): + """Asserted on the index itself, not only on the results: search already drops a + hit whose asset cannot be loaded, so a stale entry would not show there.""" + parent = _upload_video(library) + clip = _clip(library, parent["id"], name="The nano weapons moment") + hits = library.get("/api/search", params={"q": "nano weapons"}).json()["data"] + assert clip["id"] in {hit["asset"]["id"] for hit in hits} + + assert library.delete(f"/api/assets/{clip['id']}").status_code == 204 + + indexed = session.execute( + text(f"SELECT count(*) FROM {fts.ASSET_FTS} WHERE asset_id = :id"), {"id": clip["id"]} + ).one()[0] + assert indexed == 0 + + # ─── the gather() guard (M6 enrichment must refuse a clip) ──────────────────── From 17ebf9537a231afd23a610e3804193bd00d7fa67 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 13:12:09 +0000 Subject: [PATCH 3/7] feat(clips): delete clips from the Clip tab and from the parent's delete guard The Clip tab listed a video's clips with no way to remove one, and could not even open a live clip; a freshly saved clip only reached the grid on the next reload; and the parent's guard offered only to promote its clips. - ClipEditor: every row's name links to /a/{id}; a live-clip row gets a trash button with an inline "Delete clip / Cancel" confirm (sub-video rows own real bytes and get none); a saved clip is prepended to the grid. - AssetDetail: "Delete clips too" beside "Promote and delete", through a new non-optimistic removeWithClips so a failure leaves the guard mounted to say why (the remount bug M7 recorded); clip-specific wording in the Info-tab confirm, and no pointless clip lookup when deleting a clip. - library store: removeWithClips, prepend, and remove restoring the server's total on failure (it restored the loaded-row count) and only counting down a row it actually had. - docs: the poster bug and its fix in m7-clips-and-subvideos.md; the chapter-clip limitation in url-import.md now that the follow-up exists. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TPmZjuM4q96DFRfU9cHL33 --- docs/m7-clips-and-subvideos.md | 87 +++++++++++++ docs/url-import.md | 11 +- frontend/src/api/assets.ts | 13 +- frontend/src/components/AssetDetail.tsx | 88 +++++++++---- frontend/src/components/ClipEditor.test.tsx | 133 +++++++++++++++++++- frontend/src/components/ClipEditor.tsx | 79 +++++++++++- frontend/src/stores/library.test.ts | 98 +++++++++++++++ frontend/src/stores/library.ts | 50 +++++++- frontend/src/views/AssetView.test.tsx | 87 +++++++++++++ 9 files changed, 604 insertions(+), 42 deletions(-) diff --git a/docs/m7-clips-and-subvideos.md b/docs/m7-clips-and-subvideos.md index 1efb2c1..e19f838 100644 --- a/docs/m7-clips-and-subvideos.md +++ b/docs/m7-clips-and-subvideos.md @@ -7,6 +7,10 @@ sandbox had none at first; `apt-get install ffmpeg` got real coverage instead of on the `needs_ffmpeg` skip marker, and it caught two test bugs the skip would have hidden — see "What real ffmpeg caught" below). +A follow-up made clips deletable where they actually appear — per row in the Clip tab, +and together with their parent from the delete guard — and fixed the bug that made +deleting one unsafe in the first place; see "A third bug" below. + Two research passes preceded the code: a read-only survey of the actual codebase (not `docs/plan-of-attack.md`'s aspirational file sketch, which turned out to be stale in three places — see "Where this diverges from the plan doc" below), and an independent @@ -76,6 +80,11 @@ manifests once the promote flow is actually used, which is exactly the kind of g ships clean and breaks in the field — it's covered by `test_delete_after_promoting_succeeds`-shaped tests in `test_clips_api.py`. +The guard's "Delete clips too" (`DELETE /api/assets/{id}?with_clips=true`) deletes +exactly `blocking_clips`, by the same physical test — never a promoted or extracted +child. Those own their bytes, so they survive the parent's delete exactly as they do a +plain one: breadcrumb cleared, file and poster kept. + For the constraint to exist at all, the migration needed an explicit `batch_op.create_foreign_key(...)`, not just `add_column` — a bare nullable column gets SQLModel's ORM-level awareness of the relationship but not SQLite's actual enforcement @@ -173,6 +182,58 @@ created between the check and the delete, vanishingly unlikely for anything but multi-tab session) falls back to the plain "could not delete" path rather than the promote UI, which is an acceptable edge case for something this rare. +### A third bug: deleting a clip deleted its parent's poster + +Shipped with M7, and found while making clips deletable from the places they are +listed. A storage key is not owned by exactly one row. `create_clip` copies its parent's +`thumb_key` rather than resolving it later (see its comment — that is what lets a clip's +thumbnail sign like any other asset's); a URL import's `_apply_thumbnail` writes the +site's thumbnail over the generated poster *at the same key* precisely so the chapter +clips made next copy it; and `promote_clip` keeps the copied key when thumbnailing the +freshly extracted file fails (`thumb_key or clip.thumb_key`). `delete_asset`, though, +unlinked whatever `storage_key` and `thumb_key` the deleted row held, unconditionally. + +So deleting one clip deleted the poster file the parent and every sibling clip were +still pointing at. Each kept signing a `thumb_url` for it — the grid fell back to an +icon on the 404, the player lost its `poster` — and for a URL import it was the site +thumbnail, which nothing regenerates. The same bug ran the other way: deleting a parent +whose promoted child had kept the copied key took the survivor's poster with it, on the +very delete the promote existed to let through. Nothing caught it because nothing had +deleted a clip with a poster: the UI only offered it from the clip's own Info tab, which +a freshly saved clip had no way into short of a search hit or a library reload, and no +test deleted a clip at all. Two now do (`test_deleting_a_clip_removes_only_the_clip`, +`test_deleting_a_parent_keeps_its_promoted_childs_poster`); both failed against the old +code before the fix. + +The fix is per key, after the commit: `_still_referenced` asks whether any surviving row +of the same user still names the key in either column, and only an unreferenced key is +unlinked. Deliberately not "a clip owns no keys, so skip it" — that special case fixes +the clip direction and leaves the promoted survivor broken, because there it is the +*parent's* delete that has to leave the file alone. A check that cannot run keeps the +file: an orphan is what a sweep is for; an unlinked file something still shows is not +recoverable. + +Two smaller pieces went in with it: + +- **`?with_clips=true`.** The guard's third answer, beside promote and cancel. The live + clips are deleted in the same transaction as the parent and *before* it — their + `parent_asset_id` is the foreign key that would otherwise refuse the parent's delete — + as one bulk statement, since nothing tells the unit of work that rows of one table + depend on each other. It fails safe: FastAPI ignores an undeclared query parameter, so + a newer frontend against an older backend gets the 409, never a silent cascade. +- **`extraction_in_progress`.** A delete is refused with a 409 while any row it would + remove has a queued or running `extract_subvideo` job. A promote finishing into a + deleted clip updates nothing, and an extract finishing into a deleted parent trips the + foreign key on insert — either way after ffmpeg has written a file nothing will ever + reference. + +On the frontend, `stores/library.ts`'s `removeWithClips` is **not** optimistic, for +exactly the reason the second bug above records: dropping the parent before the server +answers unmounts `AssetDetail`, and a failure would then remount a fresh one with no +guard and no message. It waits, then drops the parent and the clips the guard listed. The +Clip tab's per-row delete does use the optimistic `remove` — the id it drops is the +clip's, while the panel it lives in renders the parent, so nothing unmounts beneath it. + ### The ffmpeg fallback heuristic, verified against real output `enrichment/subvideo.py`'s two strategies: input-side seek + stream copy @@ -225,6 +286,14 @@ under `self.root`. promoted. - AI enrichment (describe/summarize/autotag/generate_all) is unavailable on clips — see the `gather()` guard above. +- **No bulk delete from the selection bar.** There is none for any asset type yet; it + needs its own ownership-filtered endpoint (the shape of `/tags/bulk`) and a per-row + answer for a guarded parent. A general library feature, not a clip fix — "Delete + clips too" already covers the twenty-chapter import that made deleting clips matter. +- **No "delete every clip, keep the video"** in the Clip tab. It would be + `DELETE /api/assets/{id}/clips`: `blocking_clips`, the same dependents loop and the + same reference-checked unlink. Deferred until someone asks; per-row delete covers a + handful. --- @@ -258,6 +327,17 @@ than reimplementing its running/error/polling states), `components/AssetDetail.t test file, it's exercised through the view that renders it, matching the existing convention for that component). +**Deleting clips (follow-up):** `services/assets.py` (`_delete_dependents`, +`_still_referenced`, `delete_asset`'s `clips`), `routers/assets.py` (`with_clips`, +`extraction_in_progress`, the 409's grammar); `api/assets.ts` (`remove`'s `withClips`), +`stores/library.ts` (`removeWithClips`, `prepend`, and `remove` restoring the server's +total and only counting down a row it had), `components/ClipEditor.tsx` (each row's name +links to `/a/{id}`, a live clip's trash button with an inline confirm, a saved clip +prepended to the grid), `components/AssetDetail.tsx` ("Delete clips too", clip wording +in the Info-tab confirm). Tests in `test_clips_api.py`'s delete-guard, with-clips and +deleting-a-clip sections, `components/ClipEditor.test.tsx`, the `clips (M7)` block and +`stores/library.test.ts`. + --- ## Verification @@ -271,3 +351,10 @@ convention for that component). the same range as a sub-video, confirm the file is standalone and the parent is untouched; attempt to delete a parent with clips, confirm the guard fires with the dependent list, and the promote path clears it and lets the delete through. +- The clip-delete follow-up: `pytest -q` — 1038 passed (twelve new in + `test_clips_api.py`); `npm run format:check && npm run lint && npm test && npm run + build` — 318 passed (seventeen new), clean. +- Still to run by hand for the follow-up, in a real dev environment: delete one clip + from the Clip tab and confirm the parent's poster and file are untouched and the + sibling clips still show it; delete a parent with clips via **Delete clips too** and + confirm neither its file nor its poster is left on disk. diff --git a/docs/url-import.md b/docs/url-import.md index f87752a..6b2071a 100644 --- a/docs/url-import.md +++ b/docs/url-import.md @@ -43,7 +43,7 @@ preserve them. | `retrieved_at` | now | The one attribution field only an import can know — M10 left it with no automatic writer until this. | | thumbnail | the site's | The picture people remember the video by, and the only one an audio import gets. | | tags | `tags` | Whitespace collapsed, case-insensitive duplicates and anything over 80 characters dropped, 50 at most. | -| clips | `chapters` | Named "Chapter — Video", clamped to the file's real length; a lone chapter spanning the whole video is skipped. | +| clips | `chapters` | Named "Chapter — Video", clamped to the file's real length; a lone chapter spanning the whole video is skipped. Every chapter shows the site thumbnail — the same file as the video's — so deleting one chapter leaves the video's file and its thumbnail alone. | | transcript | captions, VTT | Only without a Deepgram key. Hand-written beats automatic; the video's own language beats YouTube's auto-translations. Recorded as model `youtube-captions` or `youtube-auto-captions`, so it is visible which kind a search is matching. | Everything from the site is stamped `embedded` in `field_provenance`: a statement of fact @@ -248,10 +248,11 @@ size free while that happens. Recorded so a later session can tell a gap from a decision. -- **A video with chapter clips cannot be deleted directly.** M7's delete guard blocks a - parent with live clips, so a twenty-chapter import needs its clips removed or promoted - first. Untick "Chapters as clips" for anything you may not keep. A "delete with its - clips" option is the natural follow-up. +- **A video with chapter clips asks before it goes.** M7's delete guard still stops a + plain delete of a parent with live clips, so twenty curated chapters are never lost by + accident; it now offers **Delete clips too** beside **Promote and delete**, which + removes the video and every chapter in one step and one transaction. A single chapter + can be deleted from the video's Clip tab. - **Downloads share the enrichment worker.** With the default `enrichment_concurrency` of 1, a long download holds up transcriptions queued behind it, and a playlist queues its videos one after another. diff --git a/frontend/src/api/assets.ts b/frontend/src/api/assets.ts index 3a840b7..27c2a5a 100644 --- a/frontend/src/api/assets.ts +++ b/frontend/src/api/assets.ts @@ -201,8 +201,17 @@ export const assetsApi = { .then((r) => r.data.data) }, - remove(id: string): Promise { - return client.delete(`/assets/${id}`).then(() => undefined) + /** + * `withClips` also deletes the live clips cut from this asset, in the same + * transaction: the delete guard's "Delete clips too". Without it, a parent with live + * clips is refused with `asset_has_dependent_clips`. + */ + remove(id: string, options: { withClips?: boolean } = {}): Promise { + return client + .delete(`/assets/${id}`, { + params: options.withClips ? { with_clips: true } : undefined, + }) + .then(() => undefined) }, /** diff --git a/frontend/src/components/AssetDetail.tsx b/frontend/src/components/AssetDetail.tsx index 053fb1c..b868627 100644 --- a/frontend/src/components/AssetDetail.tsx +++ b/frontend/src/components/AssetDetail.tsx @@ -116,6 +116,7 @@ export default function AssetDetail({ }: Props) { const update = useLibraryStore((s) => s.update) const remove = useLibraryStore((s) => s.remove) + const removeWithClips = useLibraryStore((s) => s.removeWithClips) const refreshAsset = useLibraryStore((s) => s.refreshAsset) const setAssetTags = useLibraryStore((s) => s.setAssetTags) const suggestions = useTagStore((s) => s.tags) @@ -128,11 +129,13 @@ export default function AssetDetail({ const [saving, setSaving] = useState(false) const [tagError, setTagError] = useState(null) const [confirmingDelete, setConfirmingDelete] = useState(false) - // Set once `remove()` reports `asset_has_dependent_clips` — the confirm block - // switches from "delete this?" to "promote these, then delete" while this is set. + // Set once `confirmDelete` finds live clips cut from this asset — the confirm block + // switches from "delete this?" to "promote them, or delete them too" while this is set. const [dependentClips, setDependentClips] = useState(null) const [promoting, setPromoting] = useState(false) - const [promoteError, setPromoteError] = useState(null) + const [deletingClips, setDeletingClips] = useState(false) + // Shared by the guard's two actions: whichever ran last says why it failed. + const [guardError, setGuardError] = useState(null) const [copied, setCopied] = useState(false) const descriptionRef = useAutoGrow(description) @@ -189,7 +192,7 @@ export default function AssetDetail({ setSummary(asset.summary ?? '') setConfirmingDelete(false) setDependentClips(null) - setPromoteError(null) + setGuardError(null) setTagError(null) setCurrentTime(startAt ?? asset.in_point ?? 0) }, [asset.id, asset.name, asset.description, asset.summary, asset.in_point, startAt]) @@ -298,16 +301,21 @@ export default function AssetDetail({ // silently discard `setDependentClips` below, called on a component instance that // no longer exists by the time the guard's 409 comes back. Checking first means // the guarded path never calls `remove()` at all, so it never hits that cycle. - try { - const children = await clipsApi.list(asset.id) - const blocking = children.filter((c) => c.source === 'clip') - if (blocking.length > 0) { - setDependentClips(blocking) - return + // + // Skipped for a clip, which can never have clips of its own (`create_clip` refuses + // to clip one), so the answer is known without asking. + if (!ownsNoFile(asset)) { + try { + const children = await clipsApi.list(asset.id) + const blocking = children.filter((c) => c.source === 'clip') + if (blocking.length > 0) { + setDependentClips(blocking) + return + } + } catch { + // Could not even check — fall through and let the real delete attempt, and its + // own error handling below, be the source of truth. } - } catch { - // Could not even check — fall through and let the real delete attempt, and its - // own error handling below, be the source of truth. } try { @@ -321,7 +329,7 @@ export default function AssetDetail({ const promoteAllAndDelete = async () => { if (!dependentClips || dependentClips.length === 0) return setPromoting(true) - setPromoteError(null) + setGuardError(null) try { const jobs = await Promise.all(dependentClips.map((c) => clipsApi.promote(c.id))) await waitForJobs(jobs.map((j) => j.id)) @@ -329,12 +337,30 @@ export default function AssetDetail({ await remove(asset.id) onClose() } catch (err) { - setPromoteError(apiErrorMessage(err, 'Could not promote all of them')) + setGuardError(apiErrorMessage(err, 'Could not promote all of them')) } finally { setPromoting(false) } } + // The store's non-optimistic delete, not `remove`: see `removeWithClips` for why the + // guard must stay mounted until the server has answered. + const deleteWithClips = async () => { + if (!dependentClips) return + setDeletingClips(true) + setGuardError(null) + try { + await removeWithClips( + asset.id, + dependentClips.map((c) => c.id) + ) + onClose() + } catch (err) { + setGuardError(apiErrorMessage(err, 'Could not delete it and its clips')) + setDeletingClips(false) + } + } + const visual = VISUAL_TYPES.has(asset.asset_type) const details = ( @@ -527,9 +553,9 @@ export default function AssetDetail({ {dependentClips ? (

- {dependentClips.length} clip{dependentClips.length === 1 ? '' : 's'} depend on - this asset. Extract {dependentClips.length === 1 ? 'it' : 'them'} as sub-video - {dependentClips.length === 1 ? '' : 's'} first, then this can be deleted. + {dependentClips.length === 1 + ? '1 clip depends on this asset. Extract it as a file to keep it, or delete it too.' + : `${dependentClips.length} clips depend on this asset. Extract them as files to keep them, or delete them too.`}

    {dependentClips.map((clip) => ( @@ -538,26 +564,34 @@ export default function AssetDetail({ ))}
- {promoteError && ( -

{promoteError}

+ {guardError && ( +

{guardError}

)} -
+
+ + + + ) : ( + + ))} ) })} diff --git a/frontend/src/stores/library.test.ts b/frontend/src/stores/library.test.ts index ca3e221..239fe93 100644 --- a/frontend/src/stores/library.test.ts +++ b/frontend/src/stores/library.test.ts @@ -386,3 +386,101 @@ describe('refreshAsset', () => { expect(useLibraryStore.getState().error).toBeNull() }) }) + +describe('remove', () => { + it('puts the server total back, not the loaded count, when a delete fails', async () => { + // Two rows loaded of five hundred: restoring `assets.length` would report 2. + useLibraryStore.setState({ assets: [makeAsset('a1'), makeAsset('a2')], total: 500 }) + vi.spyOn(assetsApi, 'remove').mockRejectedValue(new Error('nope')) + + await expect(useLibraryStore.getState().remove('a1')).rejects.toThrow() + + expect(useLibraryStore.getState().total).toBe(500) + expect(useLibraryStore.getState().assets.map((a) => a.id)).toEqual(['a1', 'a2']) + }) + + it('does not change the count for an id that was never loaded', async () => { + // A clip deleted from the Clip tab while filtered out of the grid, or on a page + // that has not been fetched. + useLibraryStore.setState({ assets: [makeAsset('a1')], total: 40 }) + vi.spyOn(assetsApi, 'remove').mockResolvedValue(undefined) + + await useLibraryStore.getState().remove('clip-not-in-the-grid') + + expect(useLibraryStore.getState().total).toBe(40) + expect(useLibraryStore.getState().assets.map((a) => a.id)).toEqual(['a1']) + }) +}) + +describe('removeWithClips', () => { + it('waits for the server before dropping anything', async () => { + useLibraryStore.setState({ + assets: [makeAsset('c1'), makeAsset('c2'), makeAsset('a1'), makeAsset('a2')], + total: 100, + }) + let resolve: () => void = () => {} + const remove = vi.spyOn(assetsApi, 'remove').mockReturnValue( + new Promise((r) => { + resolve = r + }) + ) + + const pending = useLibraryStore.getState().removeWithClips('a1', ['c1', 'c2']) + + // Still on screen while the request is in flight: dropping the parent now would + // unmount the panel whose guard has to report a failure. + expect(useLibraryStore.getState().assets).toHaveLength(4) + expect(remove).toHaveBeenCalledWith('a1', { withClips: true }) + + resolve() + await pending + + expect(useLibraryStore.getState().assets.map((a) => a.id)).toEqual(['a2']) + expect(useLibraryStore.getState().total).toBe(97) + }) + + it('counts down only the rows that were actually loaded', async () => { + useLibraryStore.setState({ assets: [makeAsset('a1'), makeAsset('a2')], total: 100 }) + vi.spyOn(assetsApi, 'remove').mockResolvedValue(undefined) + + await useLibraryStore.getState().removeWithClips('a1', ['c-unloaded']) + + expect(useLibraryStore.getState().assets.map((a) => a.id)).toEqual(['a2']) + expect(useLibraryStore.getState().total).toBe(99) + }) + + it('leaves every row in place and rethrows when the delete fails', async () => { + const original = [makeAsset('c1'), makeAsset('a1')] + useLibraryStore.setState({ assets: original, total: 2 }) + vi.spyOn(assetsApi, 'remove').mockRejectedValue(new Error('nope')) + + await expect( + useLibraryStore.getState().removeWithClips('a1', ['c1']) + ).rejects.toThrow('nope') + + expect(useLibraryStore.getState().assets).toEqual(original) + expect(useLibraryStore.getState().total).toBe(2) + // The guard shows the message where it was asked; a banner as well would say it twice. + expect(useLibraryStore.getState().error).toBeNull() + }) +}) + +describe('prepend', () => { + it('puts the asset at the top and counts it', () => { + useLibraryStore.setState({ assets: [makeAsset('a1')], total: 10 }) + + useLibraryStore.getState().prepend(makeAsset('c1')) + + expect(useLibraryStore.getState().assets.map((a) => a.id)).toEqual(['c1', 'a1']) + expect(useLibraryStore.getState().total).toBe(11) + }) + + it('does nothing for an asset already in the grid', () => { + useLibraryStore.setState({ assets: [makeAsset('a1')], total: 10 }) + + useLibraryStore.getState().prepend(makeAsset('a1')) + + expect(useLibraryStore.getState().assets).toHaveLength(1) + expect(useLibraryStore.getState().total).toBe(10) + }) +}) diff --git a/frontend/src/stores/library.ts b/frontend/src/stores/library.ts index 82c66db..bbdeded 100644 --- a/frontend/src/stores/library.ts +++ b/frontend/src/stores/library.ts @@ -75,6 +75,13 @@ interface LibraryState { upload: (files: File[]) => Promise update: (id: string, changes: AssetUpdate) => Promise remove: (id: string) => Promise + /** Delete an asset together with the live clips cut from it — the delete guard's + * "Delete clips too". `clipIds` are the clips the guard listed, dropped from the + * grid alongside it. */ + removeWithClips: (id: string, clipIds: string[]) => Promise + /** Put an asset made somewhere other than the grid — a clip saved from the Clip + * tab — at the top of it. */ + prepend: (asset: Asset) => void /** Replace one asset's tags with the authoritative set the server just returned. */ setAssetTags: (assetId: string, tags: Tag[]) => void /** Re-read one asset from the server, for when something other than the user @@ -303,21 +310,58 @@ export const useLibraryStore = create((set, get) => ({ }, async remove(id) { - const previous = get().assets + const { assets: previous, total: previousTotal } = get() + // Only counted down when the row was actually loaded. A clip deleted from the Clip + // tab need not be — it may be filtered out, or on a page not fetched yet — and + // taking one off the count anyway would make the total drift. + const present = previous.some((a) => a.id === id) set((state) => ({ assets: state.assets.filter((a) => a.id !== id), - total: Math.max(0, state.total - 1), + total: present ? Math.max(0, state.total - 1) : state.total, })) try { await assetsApi.remove(id) } catch (error) { - set({ assets: previous, total: previous.length }) + // The server's total, not `previous.length`: that is only the rows loaded so + // far, and after a few pages of scrolling the count would drop to a page's worth. + set({ assets: previous, total: previousTotal }) set({ error: apiErrorMessage(error, 'Could not delete that asset') }) throw error } }, + async removeWithClips(id, clipIds) { + // Waits for the server, unlike `remove`. Dropping the parent first would unmount + // AssetView's panel (it renders from `assets.find`) and mount a fresh one when a + // failure put it back, discarding the guard's own error state — the remount bug + // docs/m7-clips-and-subvideos.md records. No store `error` either: the guard + // shows its message in place, and staying mounted is what lets it. + // + // No staleness token: filtering out ids the server just deleted is right whatever + // loaded in the meantime, and after a sign-out there is nothing left to filter. + await assetsApi.remove(id, { withClips: true }) + const gone = new Set([id, ...clipIds]) + set((state) => { + const assets = state.assets.filter((a) => !gone.has(a.id)) + return { + assets, + total: Math.max(0, state.total - (state.assets.length - assets.length)), + } + }) + }, + + prepend(asset) { + // At the top regardless of the active filters, the same rule `upload` and + // `addImported` follow: it is what the user just made, and seeing it where they are + // looking is the confirmation. + set((state) => + state.assets.some((a) => a.id === asset.id) + ? state + : { assets: [asset, ...state.assets], total: state.total + 1 } + ) + }, + async openById(id) { // Fetches one asset *and puts it in `assets`*, which is the whole point. `update`, // `remove`, `setAssetTags` and `refreshAsset` all work by mapping over that list, so diff --git a/frontend/src/views/AssetView.test.tsx b/frontend/src/views/AssetView.test.tsx index 572da78..d1bd1b7 100644 --- a/frontend/src/views/AssetView.test.tsx +++ b/frontend/src/views/AssetView.test.tsx @@ -255,6 +255,93 @@ describe('clips (M7)', () => { expect(guard.getByText('Clip two')).toBeInTheDocument() expect(removeAsset).not.toHaveBeenCalled() expect(screen.getByRole('button', { name: 'Promote and delete' })).toBeInTheDocument() + expect(screen.getByRole('button', { name: 'Delete clips too' })).toBeInTheDocument() + }) + + it('says "depends" for a single clip', async () => { + vi.spyOn(assetsApi, 'get').mockResolvedValue(asset()) + vi.spyOn(clipsApi, 'list').mockResolvedValue([clip()]) + + renderAt('/a/a1') + await screen.findByRole('heading', { name: 'Giordano interview' }) + + await userEvent.click(screen.getByRole('tab', { name: 'Info' })) + await userEvent.click(screen.getByRole('button', { name: 'Delete' })) + await userEvent.click(screen.getByRole('button', { name: 'Delete' })) + + expect(await screen.findByText(/^1 clip depends on this asset/i)).toBeInTheDocument() + }) + + it('offers to delete the clips along with the asset', async () => { + vi.spyOn(assetsApi, 'get').mockResolvedValue(asset()) + const removeAsset = vi.spyOn(assetsApi, 'remove').mockResolvedValue(undefined) + const promote = vi.spyOn(clipsApi, 'promote') + vi.spyOn(clipsApi, 'list').mockResolvedValue([ + clip({ id: 'clip1', name: 'Clip one' }), + clip({ id: 'clip2', name: 'Clip two' }), + ]) + + renderAt('/a/a1') + await screen.findByRole('heading', { name: 'Giordano interview' }) + + await userEvent.click(screen.getByRole('tab', { name: 'Info' })) + await userEvent.click(screen.getByRole('button', { name: 'Delete' })) + await userEvent.click(screen.getByRole('button', { name: 'Delete' })) + await userEvent.click(await screen.findByRole('button', { name: 'Delete clips too' })) + + // One request: the server deletes the clips and the asset in one transaction. + expect(removeAsset).toHaveBeenCalledTimes(1) + expect(removeAsset).toHaveBeenCalledWith('a1', { withClips: true }) + expect(promote).not.toHaveBeenCalled() + expect(await screen.findByText('The library')).toBeInTheDocument() + }) + + it('keeps the guard open and says why when deleting with clips fails', async () => { + // The store waits for the server before dropping anything, so a failure leaves this + // very panel mounted to say so. An optimistic drop would unmount it and mount a + // fresh one with no guard and no message — the remount bug M7 already hit once. + vi.spyOn(assetsApi, 'get').mockResolvedValue(asset()) + vi.spyOn(assetsApi, 'remove').mockRejectedValue(new Error('The server fell over')) + vi.spyOn(clipsApi, 'list').mockResolvedValue([clip()]) + + renderAt('/a/a1') + await screen.findByRole('heading', { name: 'Giordano interview' }) + + await userEvent.click(screen.getByRole('tab', { name: 'Info' })) + await userEvent.click(screen.getByRole('button', { name: 'Delete' })) + await userEvent.click(screen.getByRole('button', { name: 'Delete' })) + await userEvent.click(await screen.findByRole('button', { name: 'Delete clips too' })) + + expect(await screen.findByText('The server fell over')).toBeInTheDocument() + expect(screen.getByRole('button', { name: 'Delete clips too' })).toBeEnabled() + expect( + screen.getByRole('heading', { name: 'Giordano interview' }) + ).toBeInTheDocument() + expect(screen.queryByText('The library')).not.toBeInTheDocument() + expect(useLibraryStore.getState().assets.map((a) => a.id)).toEqual(['a1']) + }) + + it('talks about a clip, not a file, when deleting a clip', async () => { + vi.spyOn(assetsApi, 'get').mockResolvedValue(clip()) + const removeAsset = vi.spyOn(assetsApi, 'remove').mockResolvedValue(undefined) + const listClips = vi.spyOn(clipsApi, 'list') + + renderAt('/a/clip1') + await screen.findByRole('heading', { name: /clip of giordano/i }) + + await userEvent.click(screen.getByRole('tab', { name: 'Info' })) + await userEvent.click(screen.getByRole('button', { name: 'Delete' })) + + expect(screen.getByText(/delete this clip\?/i)).toBeInTheDocument() + expect(screen.getByText(/video it was cut from is not affected/i)).toBeInTheDocument() + expect(screen.queryByText(/and its file/i)).not.toBeInTheDocument() + + await userEvent.click(screen.getByRole('button', { name: 'Delete' })) + + expect(removeAsset).toHaveBeenCalledWith('clip1') + // A clip cannot have clips of its own, so there is nothing for the guard to ask. + expect(listClips).not.toHaveBeenCalled() + expect(await screen.findByText('The library')).toBeInTheDocument() }) it('promoting every dependent clip retries the delete, which then succeeds', async () => { From b5630862304af76a5da9e9a114cdee77898f7723 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 13:36:13 +0000 Subject: [PATCH 4/7] =?UTF-8?q?feat(m8):=20AI=20generation=20via=20fal.ai?= =?UTF-8?q?=20=E2=80=94=20backend?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Text → image, image → image from up to ten bases, and image → video with an optional end frame, as specified in docs/m8-ai-generation.md. Results are ordinary assets with source "ai_generated", carry their provenance, and are costed from fal's billing header. - One Alembic revision: seven ai_* columns on asset, usageevent.external_ref, the generationmodel catalogue (unique endpoint id) and nine seed rows whose field names and allowed values were checked against fal's generated endpoint types. - ADMIN_USERS and an AdminUser dependency; /api/me resolves is_admin. - Catalogue CRUD at /api/generate/models (admin writes, PATCH by fields present so null clears); the fal key at /api/settings/generation. - A raw-httpx fal queue client (submit/status/result/cancel) with readable errors; bases normalised to data URIs; a streaming, size-capped, SSRF-checked download that re-checks every redirect hop. - A second JobQueue for generation over the same table, with a kind filter in recover_pending and sweep_stale, routing and cancel by kind. - The job persists fal's request ids before polling, resumes on restart, records created assets per output and usage once, and cancels at fal on a user cancel or the deadline. - Exact cost from x-fal-billable-units × fal's pricing API, falling back to the catalogue list price and to units derived from the files; a mixed usage total now reports itself as an estimate. - POST /api/generate and /regenerate; AssetRead.generation; generated video is not auto-transcribed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TPmZjuM4q96DFRfU9cHL33 --- .env.example | 36 + .../20261007_0900_add_ai_generation.py | 335 ++++++++ backend/app/auth.py | 46 ++ backend/app/config.py | 34 + backend/app/generation/__init__.py | 1 + backend/app/generation/bases.py | 70 ++ backend/app/generation/catalogue.py | 200 +++++ backend/app/generation/download.py | 198 +++++ backend/app/generation/errors.py | 19 + backend/app/generation/fal.py | 280 +++++++ backend/app/generation/pricing.py | 254 ++++++ backend/app/generation/request.py | 196 +++++ backend/app/generation/run.py | 499 ++++++++++++ backend/app/jobs/enrichment.py | 73 +- backend/app/jobs/registry.py | 51 +- backend/app/jobs/runner.py | 34 +- backend/app/main.py | 10 +- backend/app/models/__init__.py | 2 + backend/app/models/asset.py | 29 +- backend/app/models/generation.py | 87 +++ backend/app/models/job.py | 19 +- backend/app/models/usage.py | 20 +- backend/app/providers/_upstream.py | 45 +- backend/app/routers/generate.py | 469 ++++++++++++ backend/app/routers/settings.py | 31 + backend/app/schemas_assets.py | 29 +- backend/app/schemas_generate.py | 124 +++ backend/app/schemas_jobs.py | 8 +- backend/app/services/assets.py | 54 +- backend/app/settings_store.py | 11 +- backend/app/usage/events.py | 14 +- backend/tests/fal_stub.py | 290 +++++++ backend/tests/test_generation_api.py | 473 ++++++++++++ backend/tests/test_generation_catalogue.py | 356 +++++++++ backend/tests/test_generation_job.py | 721 ++++++++++++++++++ backend/tests/test_job_runner.py | 81 ++ backend/tests/test_migrations.py | 86 +++ docker-compose.yml | 9 + docs/deployment.md | 21 + 39 files changed, 5257 insertions(+), 58 deletions(-) create mode 100644 backend/alembic/versions/20261007_0900_add_ai_generation.py create mode 100644 backend/app/generation/__init__.py create mode 100644 backend/app/generation/bases.py create mode 100644 backend/app/generation/catalogue.py create mode 100644 backend/app/generation/download.py create mode 100644 backend/app/generation/errors.py create mode 100644 backend/app/generation/fal.py create mode 100644 backend/app/generation/pricing.py create mode 100644 backend/app/generation/request.py create mode 100644 backend/app/generation/run.py create mode 100644 backend/app/models/generation.py create mode 100644 backend/app/routers/generate.py create mode 100644 backend/app/schemas_generate.py create mode 100644 backend/tests/fal_stub.py create mode 100644 backend/tests/test_generation_api.py create mode 100644 backend/tests/test_generation_catalogue.py create mode 100644 backend/tests/test_generation_job.py diff --git a/.env.example b/.env.example index c5b9679..d6a1dc9 100644 --- a/.env.example +++ b/.env.example @@ -83,6 +83,42 @@ URL_IMPORT_MAX_PLAYLIST_ITEMS=200 # is a path on your own machine. The file is a login: treat it like a password. URL_IMPORT_COOKIES_FILE= +# ─── Administrators ────────────────────────────────────────────────────────── + +# Who may edit the AI generation model catalogue: Gecko Notes user ids or usernames, +# comma-separated. Ids match exactly, usernames case-insensitively. Notes' token carries +# no admin flag, so this is the only way anyone becomes an admin here. Empty means +# nobody — the catalogue still works, as seeded, for everyone. +ADMIN_USERS= + +# ─── AI generation (fal.ai) ────────────────────────────────────────────────── +# +# Nothing here is a credential: each user adds their own fal.ai key under Settings → +# Generation, and fal bills that account. See docs/m8-ai-generation.md. + +# Generations running at once. Each is almost entirely waiting on fal, so this costs the +# container next to nothing; it has its own workers, apart from transcription's. +GENERATION_CONCURRENCY=2 + +# How long to wait for fal before cancelling the request there and failing the job. A +# request abandoned without that cancel keeps running, and billing, at fal. +GENERATION_TIMEOUT_MINUTES=30 + +# Seconds between status checks while fal works. +GENERATION_POLL_SECONDS=3 + +# Largest file one output may be, in MB. Enforced while it downloads, not after. +GENERATION_MAX_DOWNLOAD_MB=1024 + +# Base images are sent inside the request, so they are resized first: long edge in +# pixels, then a size limit per image in MB that a resized one is refused above. +GENERATION_BASE_MAX_EDGE=2048 +GENERATION_MAX_DATA_URI_MB=8 + +# fal's hosts. Only worth changing if fal moves them. +FAL_QUEUE_BASE_URL=https://queue.fal.run +FAL_API_BASE_URL=https://api.fal.ai + # ─── Development only ──────────────────────────────────────────────────────── # "development" enables DEV_AUTH_USER below. Anything else, and it is ignored. diff --git a/backend/alembic/versions/20261007_0900_add_ai_generation.py b/backend/alembic/versions/20261007_0900_add_ai_generation.py new file mode 100644 index 0000000..4fb8062 --- /dev/null +++ b/backend/alembic/versions/20261007_0900_add_ai_generation.py @@ -0,0 +1,335 @@ +"""add ai generation: asset provenance, usage external_ref, the model catalogue + +M8, in one revision, as docs/m8-ai-generation.md specifies: + +1. Seven nullable `ai_*` columns on `asset` — what made a generated file, kept so it can + be made again. No indexes and no foreign keys: the bases live as ids inside + `ai_source_assets`, so deleting one never blocks or breaks what was made from it. + Adding columns is plain `ALTER TABLE ADD COLUMN` on SQLite, so the upgrade does not + recreate `asset`; the downgrade's drops do, hence the PRAGMA dance `7d4b9c1a6f28` + explains. +2. `usageevent.external_ref` — fal's `x-fal-request-id`, so a recorded cost can be found + in fal's own billing. +3. `generationmodel` — the admin-managed endpoint catalogue, with its endpoint id unique. +4. Nine seed rows, three per kind, so a fresh install can generate before anybody has + been made an admin. + +The seed is a literal copy rather than an import of app code: a migration describes the +data as it was when it ran, and must not change meaning when the app does. Every field +name and allowed value below was checked against the endpoint input types generated +from fal's OpenAPI schemas in `@fal-ai/client` 1.11.0-alpha.5 (`src/types/endpoints.d.ts`, +2026-10-06); the evidence for each row is beside it. Prices are list prices and only the +fallback — the job asks fal's pricing API first. + +Revision ID: a8e1c0f4d2b6 +Revises: 9c2e08b4a1f7 +Create Date: 2026-10-07 09:00:00.000000+00:00 +""" +import json +import uuid +from datetime import datetime, timezone +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +import sqlmodel + + +revision: str = 'a8e1c0f4d2b6' +down_revision: Union[str, None] = '9c2e08b4a1f7' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def _options( + *, + aspect_ratios=(), + image_sizes=(), + durations=(), + resolutions=(), + supports_seed=False, + supports_negative_prompt=False, + supports_audio=False, + max_outputs=1, +) -> str: + return json.dumps( + { + "aspect_ratios": list(aspect_ratios), + "image_sizes": list(image_sizes), + "durations": list(durations), + "resolutions": list(resolutions), + "supports_seed": supports_seed, + "supports_negative_prompt": supports_negative_prompt, + "supports_audio": supports_audio, + "max_outputs": max_outputs, + } + ) + + +# FluxDevInput (FluxSchnellInput is an alias of it): `image_size` is a preset or +# {width, height}; `num_images`, `seed`; no `aspect_ratio`, no `negative_prompt`. +_FLUX_SIZES = ( + "square_hd", "square", "portrait_4_3", "portrait_16_9", "landscape_4_3", "landscape_16_9", +) + +SEEDS = [ + # ─── text → image ──────────────────────────────────────────────────────── + dict( + endpoint_id="fal-ai/flux/schnell", + kind="text_to_image", + label="FLUX.1 [schnell]", + note="Fastest and cheapest. Good for drafts.", + sort_order=10, + options=_options(image_sizes=_FLUX_SIZES, supports_seed=True, max_outputs=4), + # Price: LiteLLM's fal table. + unit_price=0.003, + price_unit="megapixel", + ), + dict( + endpoint_id="fal-ai/flux/dev", + kind="text_to_image", + label="FLUX.1 [dev]", + note="Good quality at a low price.", + sort_order=20, + options=_options(image_sizes=_FLUX_SIZES, supports_seed=True, max_outputs=4), + # Price: the example in fal's own pricing API reference. + unit_price=0.025, + price_unit="megapixel", + ), + dict( + endpoint_id="fal-ai/nano-banana-2", + kind="text_to_image", + label="Nano Banana 2", + note="Google's model. Strong at text in images. Price rises with resolution.", + sort_order=30, + # NanoBanana2Input: `aspect_ratio` incl. auto and the extreme 4:1…1:8, + # `resolution` "0.5K"|"1K"|"2K"|"4K", `num_images`, `seed`. Its output reports no + # seed back. + options=_options( + aspect_ratios=( + "auto", "21:9", "16:9", "3:2", "4:3", "5:4", "1:1", "4:5", "3:4", "2:3", + "9:16", "4:1", "1:4", "8:1", "1:8", + ), + resolutions=("0.5K", "1K", "2K", "4K"), + supports_seed=True, + max_outputs=4, + ), + # Price: LiteLLM, at 1K. + unit_price=0.08, + price_unit="image", + ), + # ─── image → image ─────────────────────────────────────────────────────── + dict( + endpoint_id="fal-ai/nano-banana/edit", + kind="image_to_image", + label="Nano Banana (edit)", + note="Edits or combines images from an instruction.", + sort_order=10, + # NanoBananaEditInput: `image_urls: Array` (required), `aspect_ratio` + # incl. auto, `num_images`, `seed`. The schema states no cap on the list; three + # is what Google advises for this model, and an admin can raise it. + image_field="image_urls", + image_field_is_list=True, + max_images=3, + options=_options( + aspect_ratios=( + "auto", "21:9", "16:9", "3:2", "4:3", "5:4", "1:1", "4:5", "3:4", "2:3", + "9:16", + ), + supports_seed=True, + max_outputs=4, + ), + # Price: LiteLLM's text-to-image figure for the same model. + unit_price=0.039, + price_unit="image", + ), + dict( + endpoint_id="fal-ai/bytedance/seedream/v4.5/edit", + kind="image_to_image", + label="Seedream 4.5 (edit)", + note="Takes up to ten reference images.", + sort_order=20, + # SeedDream45EditInput: `image_urls` — "up to 10 image inputs are allowed. If over + # 10 images are sent, only the last 10 will be used" — so ten, enforced here + # rather than silently truncated there. `image_size` accepts the presets in its + # type, but its doc says width and height must be 1920–4096 (or 2560×1440 to + # 4096² in total), which every preset is below; only the auto sizes are safe. + # `num_images`, `seed`. + image_field="image_urls", + image_field_is_list=True, + max_images=10, + options=_options( + image_sizes=("auto_2K", "auto_4K"), supports_seed=True, max_outputs=4 + ), + # Price: unverified third-party figure. + unit_price=0.04, + price_unit="image", + ), + dict( + endpoint_id="fal-ai/flux-pro/kontext", + kind="image_to_image", + label="FLUX.1 Kontext [pro]", + note="Precise edits to one image.", + sort_order=30, + # FluxKontextInput: `image_url` (single, required), `aspect_ratio` 21:9…9:21 with + # no auto, `num_images`, `seed`. The edit endpoint gecko-notes seeded as text → + # image, where it 422'd every call; here its kind is right. + image_field="image_url", + image_field_is_list=False, + max_images=1, + options=_options( + aspect_ratios=("21:9", "16:9", "4:3", "3:2", "1:1", "2:3", "3:4", "9:16", "9:21"), + supports_seed=True, + max_outputs=4, + ), + unit_price=0.04, + price_unit="image", + ), + # ─── image → video ─────────────────────────────────────────────────────── + dict( + endpoint_id="fal-ai/kling-video/v2.5-turbo/pro/image-to-video", + kind="image_to_video", + label="Kling 2.5 Turbo Pro", + note="Start and end frame. 5 or 10 seconds.", + sort_order=10, + # KlingVideoV25TurboProImageToVideoInput: `image_url`, `tail_image_url` (the pro + # variant only), `duration` "5"|"10" as strings, `negative_prompt`, `cfg_scale`. + # No `seed`, no `aspect_ratio`, no `resolution`, no audio. + image_field="image_url", + max_images=1, + end_image_field="tail_image_url", + options=_options(durations=("5", "10"), supports_negative_prompt=True), + # Price: third-party, $0.35 per five seconds. + unit_price=0.07, + price_unit="second", + ), + dict( + endpoint_id="fal-ai/minimax/hailuo-02/standard/image-to-video", + kind="image_to_video", + label="MiniMax Hailuo 02 Standard", + note="Start and end frame. The cheapest video here.", + sort_order=20, + # StandardImageToVideoHailuo02Input: `image_url`, `end_image_url`, `duration` + # "6"|"10", `resolution` "512P"|"768P" (uppercase P), `prompt_optimizer`. No + # `seed`, no `negative_prompt`, no `aspect_ratio`. + image_field="image_url", + max_images=1, + end_image_field="end_image_url", + options=_options(durations=("6", "10"), resolutions=("512P", "768P")), + # Price: third-party, at 768P. + unit_price=0.045, + price_unit="second", + ), + dict( + endpoint_id="fal-ai/veo3.1/fast/image-to-video", + kind="image_to_video", + label="Veo 3.1 Fast", + note="Google's model. Audio is off unless asked for — it costs half as much again.", + sort_order=30, + # Veo31ImageToVideoInput: `image_url` and no end frame (that is the separate + # first-last-frame endpoint), `duration` "4s"|"6s"|"8s" with the suffix, + # `resolution` "720p"|"1080p"|"4k", `aspect_ratio` "auto"|"16:9"|"9:16", + # `generate_audio` (default true), `negative_prompt`, `seed`. + image_field="image_url", + max_images=1, + options=_options( + aspect_ratios=("auto", "16:9", "9:16"), + durations=("4s", "6s", "8s"), + resolutions=("720p", "1080p", "4k"), + supports_seed=True, + supports_negative_prompt=True, + supports_audio=True, + ), + extra_params=json.dumps({"generate_audio": False}), + # Price: fal's Veo 3.1 Fast page, without audio, at 720p/1080p. + unit_price=0.10, + price_unit="second", + ), +] + + +def upgrade() -> None: + with op.batch_alter_table('asset', schema=None) as batch_op: + batch_op.add_column(sa.Column('ai_model', sqlmodel.sql.sqltypes.AutoString(), nullable=True)) + batch_op.add_column(sa.Column('ai_generation_type', sqlmodel.sql.sqltypes.AutoString(), nullable=True)) + batch_op.add_column(sa.Column('ai_prompt', sqlmodel.sql.sqltypes.AutoString(), nullable=True)) + batch_op.add_column(sa.Column('ai_parameters', sqlmodel.sql.sqltypes.AutoString(), nullable=True)) + batch_op.add_column(sa.Column('ai_source_assets', sqlmodel.sql.sqltypes.AutoString(), nullable=True)) + batch_op.add_column(sa.Column('ai_seed', sa.Integer(), nullable=True)) + batch_op.add_column(sa.Column('ai_generated_at', sa.DateTime(), nullable=True)) + + with op.batch_alter_table('usageevent', schema=None) as batch_op: + batch_op.add_column(sa.Column('external_ref', sqlmodel.sql.sqltypes.AutoString(), nullable=True)) + + generationmodel = op.create_table('generationmodel', + sa.Column('id', sqlmodel.sql.sqltypes.AutoString(), nullable=False), + sa.Column('endpoint_id', sqlmodel.sql.sqltypes.AutoString(), nullable=False), + sa.Column('kind', sqlmodel.sql.sqltypes.AutoString(), nullable=False), + sa.Column('label', sqlmodel.sql.sqltypes.AutoString(), nullable=False), + sa.Column('note', sqlmodel.sql.sqltypes.AutoString(), nullable=False), + sa.Column('sort_order', sa.Integer(), nullable=False), + sa.Column('is_active', sa.Boolean(), nullable=False), + sa.Column('image_field', sqlmodel.sql.sqltypes.AutoString(), nullable=True), + sa.Column('image_field_is_list', sa.Boolean(), nullable=False), + sa.Column('max_images', sa.Integer(), nullable=False), + sa.Column('end_image_field', sqlmodel.sql.sqltypes.AutoString(), nullable=True), + sa.Column('options', sqlmodel.sql.sqltypes.AutoString(), nullable=False), + sa.Column('extra_params', sqlmodel.sql.sqltypes.AutoString(), nullable=False), + sa.Column('unit_price', sa.Float(), nullable=True), + sa.Column('price_unit', sqlmodel.sql.sqltypes.AutoString(), nullable=True), + sa.Column('price_currency', sqlmodel.sql.sqltypes.AutoString(), nullable=True), + sa.Column('created_at', sa.DateTime(), nullable=False), + sa.Column('updated_at', sa.DateTime(), nullable=False), + sa.PrimaryKeyConstraint('id') + ) + with op.batch_alter_table('generationmodel', schema=None) as batch_op: + batch_op.create_index(batch_op.f('ix_generationmodel_endpoint_id'), ['endpoint_id'], unique=True) + batch_op.create_index(batch_op.f('ix_generationmodel_kind'), ['kind'], unique=False) + + # Naive UTC, the convention every timestamp column here follows (see app/clock.py), + # spelled out rather than imported for the reason the module docstring gives. + now = datetime.now(timezone.utc).replace(tzinfo=None) + op.bulk_insert( + generationmodel, + [ + { + "id": str(uuid.uuid4()), + "note": "", + "is_active": True, + "image_field": None, + "image_field_is_list": False, + "max_images": 0, + "end_image_field": None, + "extra_params": "{}", + "price_currency": "USD", + "created_at": now, + "updated_at": now, + **seed, + } + for seed in SEEDS + ], + ) + + +def downgrade() -> None: + with op.batch_alter_table('generationmodel', schema=None) as batch_op: + batch_op.drop_index(batch_op.f('ix_generationmodel_kind')) + batch_op.drop_index(batch_op.f('ix_generationmodel_endpoint_id')) + + op.drop_table('generationmodel') + + with op.batch_alter_table('usageevent', schema=None) as batch_op: + batch_op.drop_column('external_ref') + + # Dropping columns recreates `asset`, and SQLite checks foreign keys on that DROP + # TABLE — see `7d4b9c1a6f28`, whose upgrade hit exactly this on a populated library. + op.execute('PRAGMA foreign_keys=OFF') + with op.batch_alter_table('asset', schema=None) as batch_op: + batch_op.drop_column('ai_generated_at') + batch_op.drop_column('ai_seed') + batch_op.drop_column('ai_source_assets') + batch_op.drop_column('ai_parameters') + batch_op.drop_column('ai_prompt') + batch_op.drop_column('ai_generation_type') + batch_op.drop_column('ai_model') + op.execute('PRAGMA foreign_keys=ON') diff --git a/backend/app/auth.py b/backend/app/auth.py index 6d23cab..a848056 100644 --- a/backend/app/auth.py +++ b/backend/app/auth.py @@ -35,8 +35,11 @@ from jose import JWTError, jwt from jose.exceptions import ExpiredSignatureError from pydantic import BaseModel +from sqlmodel import Session from app.config import settings +from app.database import get_session +from app.models.user import User logger = logging.getLogger(__name__) @@ -210,6 +213,49 @@ def current_user( CurrentUser = Annotated[UserCtx, Depends(current_user)] +# ─── administrators ────────────────────────────────────────────────────────── +# +# The token carries no admin claim and `User.is_admin` has no writer, so an admin is +# whoever `ADMIN_USERS` names — or a row somebody set by hand, which still counts. Today +# the only thing an admin may do that others may not is edit the generation catalogue. + + +def is_admin(user: UserCtx, record: Optional[User]) -> bool: + """Whether this caller is an administrator. + + An entry in `ADMIN_USERS` matches an id exactly and a username case-insensitively, + against either the token's username or the shadow row's, so a configured name keeps + working whichever one has caught up with a rename in Notes. + """ + if record is not None and record.is_admin: + return True + + usernames = { + name.lower() + for name in (user.username, record.username if record is not None else "") + if name + } + return any( + entry == user.id or entry.lower() in usernames for entry in settings.admin_user_list + ) + + +def admin_user(user: CurrentUser, session: Session = Depends(get_session)) -> UserCtx: + """The caller, or 403 if they are not an administrator.""" + if not is_admin(user, session.get(User, user.id)): + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail={ + "code": "admin_required", + "message": "Only an administrator can change this", + }, + ) + return user + + +AdminUser = Annotated[UserCtx, Depends(admin_user)] + + # ─── secrets at rest ───────────────────────────────────────────────────────── diff --git a/backend/app/config.py b/backend/app/config.py index 5a713b8..8824059 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -83,6 +83,36 @@ class Settings(BaseSettings): # age-restricted or members-only videos; mount the file read-only and point here. url_import_cookies_file: str = "" + # ─── administration ────────────────────────────────────────────────────── + # Comma-separated user ids or usernames. The Notes token carries no admin claim and + # nothing in GAM can grant one, so this is the only way anybody becomes an admin — + # which today means being allowed to edit the generation model catalogue. Ids match + # exactly; usernames case-insensitively, because people type them from memory. + admin_users: str = "" + + # ─── AI generation (M8) ────────────────────────────────────────────────── + # Settings, not constants, so a test can point both at a stub and an operator can + # follow fal if it moves a host. Each user's own key is what authenticates. + fal_queue_base_url: str = "https://queue.fal.run" + # The platform API, used here only for unit prices. + fal_api_base_url: str = "https://api.fal.ai" + # Its own worker pool, apart from `enrichment_concurrency`: a generation is almost + # entirely waiting on fal, and a four-minute video on the shared queue would hold + # every transcription behind it. + generation_concurrency: int = 2 + # GAM's own deadline on a fal request. fal reports no terminal "failed" status a + # client can rely on, so without one an unknown status would be waited on forever. + # Past it the request is cancelled at fal — a remote job left running keeps billing. + generation_timeout_minutes: int = 30 + generation_poll_seconds: float = 3 + # Per output, enforced while streaming rather than after: a video arrives as one + # file and a size checked afterwards has already filled the disk. + generation_max_download_mb: int = 1024 + # Base images travel inside the request as data URIs, so their size is request + # size. 2048 px on the long edge is more than any seeded model reads. + generation_base_max_edge: int = 2048 + generation_max_data_uri_mb: int = 8 + # ─── development ───────────────────────────────────────────────────────── # Bypasses token verification and acts as this user id. Refused unless # `environment` is "development", so it cannot be switched on in production by @@ -117,6 +147,10 @@ def resolved_media_dir(self) -> Path: def cors_origins(self) -> List[str]: return [o.strip() for o in self.cors_origin.split(",") if o.strip()] + @property + def admin_user_list(self) -> List[str]: + return [u.strip() for u in self.admin_users.split(",") if u.strip()] + @property def is_development(self) -> bool: return self.environment.strip().lower() in {"development", "dev", "local"} diff --git a/backend/app/generation/__init__.py b/backend/app/generation/__init__.py new file mode 100644 index 0000000..3345129 --- /dev/null +++ b/backend/app/generation/__init__.py @@ -0,0 +1 @@ +"""M8: making new media with fal.ai. Specified in docs/m8-ai-generation.md.""" diff --git a/backend/app/generation/bases.py b/backend/app/generation/bases.py new file mode 100644 index 0000000..9e1eba2 --- /dev/null +++ b/backend/app/generation/bases.py @@ -0,0 +1,70 @@ +"""A base image as the data URI fal is sent. + +Data URIs rather than an upload to fal's CDN, because GAM's media is private and must +not become public to serve this: an upload lands on a CDN with no expiry (gecko-notes' +audio uploads do exactly that), and it is a second, separately versioned API fal has +already broken once. The price is request size, which is why each base is normalised +here before it is encoded: + +- EXIF orientation applied, since a model reads pixels, not tags — a portrait phone + photo would otherwise arrive on its side; +- the long edge capped (`generation_base_max_edge`), which no seeded model reads past; +- re-encoded as JPEG, or PNG when it really has transparency — an edit model told to + "put this on a beach" needs to know where the subject ends; +- refused if it is still over `generation_max_data_uri_mb`. +""" + +from __future__ import annotations + +import base64 +import io +from pathlib import Path + +from PIL import Image, ImageOps, UnidentifiedImageError + +from app.generation.errors import GenerationError + +JPEG_QUALITY = 90 + + +def data_uri(path: Path, *, max_edge: int, max_bytes: int) -> str: + """Normalise the image at `path` and return it as a `data:` URI.""" + try: + with Image.open(path) as opened: + opened.load() + image = ImageOps.exif_transpose(opened) + except (OSError, UnidentifiedImageError, Image.DecompressionBombError, ValueError) as exc: + raise GenerationError("A base image could not be read as a picture") from exc + + if max(image.size) > max_edge: + image.thumbnail((max_edge, max_edge), Image.Resampling.LANCZOS) + + buffer = io.BytesIO() + if _has_transparency(image): + image.convert("RGBA").save(buffer, format="PNG", optimize=True) + media_type = "image/png" + else: + image.convert("RGB").save(buffer, format="JPEG", quality=JPEG_QUALITY) + media_type = "image/jpeg" + + uri = f"data:{media_type};base64,{base64.b64encode(buffer.getvalue()).decode('ascii')}" + if len(uri) > max_bytes: + raise GenerationError( + f"A base image is larger than the {max_bytes // (1024 * 1024)} MB limit " + "even after resizing" + ) + return uri + + +def _has_transparency(image: Image.Image) -> bool: + """Whether any pixel is actually see-through, not merely whether it could be. + + Many PNGs carry an alpha channel that is opaque everywhere; sending those as PNG + would cost several times the bytes of a JPEG for nothing. + """ + if image.mode == "P": + return "transparency" in image.info + if image.mode not in ("RGBA", "LA", "PA"): + return False + low, _high = image.getchannel("A").getextrema() + return low < 255 diff --git a/backend/app/generation/catalogue.py b/backend/app/generation/catalogue.py new file mode 100644 index 0000000..bfc4db6 --- /dev/null +++ b/backend/app/generation/catalogue.py @@ -0,0 +1,200 @@ +"""Reading and validating catalogue rows. + +Strict on write, lenient on read. An admin saving a row is told precisely what is wrong +with it, because a row that saves and then 422s every generation is the failure this +catalogue exists to prevent. A row already stored is read with defaults filled in and +bad values dropped, so one hand-edited row cannot make the model list unreadable. +""" + +from __future__ import annotations + +import copy +import json +import re +from typing import Any, Optional + +from app.models.generation import ( + GENERATION_KINDS, + KIND_IMAGE_TO_VIDEO, + KIND_TEXT_TO_IMAGE, + PRICE_UNITS, + GenerationModel, +) + +# The user-facing choices, as `params` names them and as `options` lists their values. +CHOICES = ( + ("aspect_ratio", "aspect_ratios"), + ("image_size", "image_sizes"), + ("duration", "durations"), + ("resolution", "resolutions"), +) +OPTION_LISTS = tuple(key for _param, key in CHOICES) +OPTION_FLAGS = ("supports_seed", "supports_negative_prompt", "supports_audio") +MAX_OUTPUTS = 4 + +DEFAULT_OPTIONS: dict[str, Any] = { + **{key: [] for key in OPTION_LISTS}, + **{flag: False for flag in OPTION_FLAGS}, + "max_outputs": 1, +} + +# An endpoint id becomes a URL path, so it is held to what fal's ids look like: at +# least two segments of letters, digits, dots, dashes and underscores. That rules out +# `..`, a query string, and anything else that would put the request somewhere other +# than the endpoint named. +_ENDPOINT_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*(/[A-Za-z0-9][A-Za-z0-9._-]*)+") +# A request body key. +_FIELD_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") + + +class InvalidEntry(ValueError): + """A catalogue row that cannot be saved. The message says which field and why.""" + + +# ─── reading ───────────────────────────────────────────────────────────────── + + +def options_of(row: GenerationModel) -> dict[str, Any]: + """A stored row's options, with every key present.""" + options = copy.deepcopy(DEFAULT_OPTIONS) + stored = _json_object(row.options) + for key in OPTION_LISTS: + if isinstance(stored.get(key), list): + options[key] = [v for v in stored[key] if _is_scalar(v)] + for flag in OPTION_FLAGS: + if isinstance(stored.get(flag), bool): + options[flag] = stored[flag] + if _is_int(stored.get("max_outputs")): + options["max_outputs"] = stored["max_outputs"] + return options + + +def extra_params_of(row: GenerationModel) -> dict[str, Any]: + return _json_object(row.extra_params) + + +def _json_object(text: Optional[str]) -> dict: + try: + value = json.loads(text or "{}") + except ValueError: + return {} + return value if isinstance(value, dict) else {} + + +# ─── validating ────────────────────────────────────────────────────────────── + + +def normalise_options(value: Any) -> dict[str, Any]: + """An admin's `options`, checked, with defaults for whatever was left out. + + An unknown key is refused rather than kept: `"duration"` for `"durations"` would + otherwise save, do nothing, and leave the form offering no durations at all. + """ + if not isinstance(value, dict): + raise InvalidEntry("options must be an object") + + options = copy.deepcopy(DEFAULT_OPTIONS) + for key, item in value.items(): + if key in OPTION_LISTS: + if not isinstance(item, list) or not all(_is_scalar(v) for v in item): + raise InvalidEntry(f"options.{key} must be a list of strings or numbers") + options[key] = item + elif key in OPTION_FLAGS: + if not isinstance(item, bool): + raise InvalidEntry(f"options.{key} must be true or false") + options[key] = item + elif key == "max_outputs": + if not _is_int(item): + raise InvalidEntry("options.max_outputs must be a whole number") + options[key] = item + else: + raise InvalidEntry(f"options has no setting called {key!r}") + return options + + +def validate(entry: dict[str, Any]) -> dict[str, Any]: + """Check a whole row — after a PATCH is merged, not just the fields it sent. + + Returns the entry with `options` normalised. The rules are the spec's, plus one it + implies: a model that takes more than one base must take them as a list, since a + single-image field cannot hold two. + """ + endpoint_id = entry.get("endpoint_id") + if not isinstance(endpoint_id, str) or not _ENDPOINT_ID.fullmatch(endpoint_id): + raise InvalidEntry( + "endpoint_id must be a fal endpoint id such as fal-ai/flux/dev" + ) + + kind = entry.get("kind") + if kind not in GENERATION_KINDS: + raise InvalidEntry(f"kind must be one of {', '.join(GENERATION_KINDS)}") + + label = entry.get("label") + if not isinstance(label, str) or not label.strip(): + raise InvalidEntry("label is required") + + image_field = entry.get("image_field") + end_image_field = entry.get("end_image_field") + max_images = entry.get("max_images") + if not _is_int(max_images): + raise InvalidEntry("max_images must be a whole number") + + for name, value in (("image_field", image_field), ("end_image_field", end_image_field)): + if value is not None and (not isinstance(value, str) or not _FIELD_NAME.fullmatch(value)): + raise InvalidEntry(f"{name} must be a request field name, such as image_url") + + if kind == KIND_TEXT_TO_IMAGE: + if max_images != 0: + raise InvalidEntry( + "A text-to-image model takes no base images, so max_images must be 0" + ) + if image_field is not None: + raise InvalidEntry( + "A text-to-image model takes no base images, so it has no image_field" + ) + else: + if max_images < 1: + raise InvalidEntry( + "This kind of model needs at least one base image: max_images must be 1 or more" + ) + if image_field is None: + raise InvalidEntry( + "This kind of model needs an image_field to send its base image in" + ) + if max_images > 1 and not entry.get("image_field_is_list"): + raise InvalidEntry( + "A model that takes more than one base image needs image_field_is_list" + ) + + if end_image_field is not None and kind != KIND_IMAGE_TO_VIDEO: + raise InvalidEntry("Only an image-to-video model takes an end frame") + + options = normalise_options(entry.get("options", {})) + max_outputs = options["max_outputs"] + if kind == KIND_IMAGE_TO_VIDEO: + if max_outputs != 1: + raise InvalidEntry( + "A video model makes one video at a time: options.max_outputs must be 1" + ) + elif not 1 <= max_outputs <= MAX_OUTPUTS: + raise InvalidEntry(f"options.max_outputs must be between 1 and {MAX_OUTPUTS}") + + if not isinstance(entry.get("extra_params", {}), dict): + raise InvalidEntry("extra_params must be an object") + + unit_price = entry.get("unit_price") + if unit_price is not None and (isinstance(unit_price, bool) or unit_price < 0): + raise InvalidEntry("unit_price cannot be negative") + price_unit = entry.get("price_unit") + if price_unit is not None and price_unit not in PRICE_UNITS: + raise InvalidEntry(f"price_unit must be one of {', '.join(PRICE_UNITS)}") + + return {**entry, "options": options} + + +def _is_scalar(value: Any) -> bool: + return isinstance(value, (str, int, float)) and not isinstance(value, bool) + + +def _is_int(value: Any) -> bool: + return isinstance(value, int) and not isinstance(value, bool) diff --git a/backend/app/generation/download.py b/backend/app/generation/download.py new file mode 100644 index 0000000..d7e19d3 --- /dev/null +++ b/backend/app/generation/download.py @@ -0,0 +1,198 @@ +"""Fetching what fal made: streamed, capped, and checked at every hop. + +The first streaming download in GAM — uploads arrive streamed, and yt-dlp does its own +fetching. gecko-notes reads the whole body into memory and checks its size afterwards, +which caps nothing: the cap is enforced here while the bytes arrive, against the +`Content-Length` first when there is one, because a video is one file and a check made +after it is written has already filled the disk. + +The URL is fal's answer, not the user's, but it is still a URL this server fetches on +somebody's behalf, so it gets the same SSRF check a pasted link does — on every redirect +hop too, since a public URL that redirects inward is the classic way past a check made +only on the first. `safe_url`'s own docstring calls this defence in depth; DNS can still +change between the check and the connect. + +No credentials go with it. fal's output URLs are public CDN links, and the fal key sent +to wherever a result happened to point would be a key handed to that host. +""" + +from __future__ import annotations + +import base64 +import binascii +import logging +import mimetypes +from pathlib import Path +from typing import Callable, Optional +from urllib.parse import urljoin, urlsplit + +import httpx +from fastapi import HTTPException + +from app.generation.errors import GenerationError +from app.ingest.filetypes import IMAGE_EXTENSIONS, VIDEO_EXTENSIONS, extension_of +from app.jobs.runner import readable_error +from app.safe_url import require_safe_external_url + +logger = logging.getLogger(__name__) + +CHUNK_SIZE = 1024 * 1024 +MAX_REDIRECTS = 3 +# Per read, not per file: a large video can take minutes in total, but a CDN that sends +# nothing at all for a minute has stopped. +TIMEOUT = httpx.Timeout(60.0, connect=10.0) + +MEDIA_EXTENSIONS = IMAGE_EXTENSIONS | VIDEO_EXTENSIONS + +# `received`, and `total` when the server said. +Progress = Callable[[int, Optional[int]], None] + + +def download( + url: str, + workdir: Path, + stem: str, + *, + max_bytes: int, + default_extension: str, + content_type_hint: Optional[str] = None, + on_progress: Optional[Progress] = None, +) -> Path: + """Fetch `url` into `workdir/` and return the path. + + The extension decides the asset type on ingest, so it comes from the URL path when + that names a media type, else from the response's `Content-Type` (then fal's own + `content_type` for the file), else `default_extension`. `on_progress` is called + between chunks; the job's progress callback raises there to cancel, which closes the + stream and removes the partial file on the way out. + """ + report = on_progress or (lambda _received, _total: None) + + if url.startswith("data:"): + # `sync_mode` in a catalogue row's extra_params makes fal answer inline. Not + # something the seeded rows ask for, but nothing about it is worth refusing. + return _from_data_uri( + url, workdir, stem, max_bytes=max_bytes, default_extension=default_extension + ) + + target = url + with httpx.Client(timeout=TIMEOUT, follow_redirects=False) as client: + for _hop in range(MAX_REDIRECTS + 1): + _require_safe(target) + try: + with client.stream("GET", target) as response: + # `is_redirect` already requires a Location header. + if response.is_redirect: + target = urljoin(target, response.headers["location"]) + continue + if response.status_code >= 400: + raise GenerationError( + f"Could not download the generated file (HTTP {response.status_code})" + ) + + total = _content_length(response) + if total is not None and total > max_bytes: + raise _too_large(max_bytes) + + extension = _extension_for( + target, + response.headers.get("content-type"), + content_type_hint, + default_extension, + ) + destination = workdir / f"{stem}{extension}" + _stream_to( + response, destination, max_bytes=max_bytes, total=total, report=report + ) + return destination + except httpx.TimeoutException as exc: + raise GenerationError("The generated file did not download in time") from exc + except httpx.RequestError as exc: + # The type, not the text: a RequestError's message carries the full URL. + raise GenerationError( + f"Could not download the generated file ({type(exc).__name__})" + ) from exc + + raise GenerationError("Could not download the generated file (too many redirects)") + + +def _require_safe(url: str) -> None: + try: + require_safe_external_url(url) + except HTTPException as exc: + raise GenerationError( + f"The generated file is at an address this app will not fetch: {readable_error(exc)}" + ) from exc + + +def _stream_to( + response: httpx.Response, + destination: Path, + *, + max_bytes: int, + total: Optional[int], + report: Progress, +) -> None: + received = 0 + try: + with open(destination, "wb") as handle: + for chunk in response.iter_bytes(CHUNK_SIZE): + received += len(chunk) + # Counted, not trusted: Content-Length may be absent, or wrong. + if received > max_bytes: + raise _too_large(max_bytes) + handle.write(chunk) + report(received, total) + except BaseException: + # BaseException, so a cancel raised from `report` cleans up too. + destination.unlink(missing_ok=True) + raise + + +def _from_data_uri( + uri: str, workdir: Path, stem: str, *, max_bytes: int, default_extension: str +) -> Path: + header, _, payload = uri.partition(",") + if ";base64" not in header: + raise GenerationError("fal.ai returned an inline file this app cannot read") + # Four characters of base64 are three bytes; checked before decoding, so an + # oversized one is refused without being held in memory twice. + if len(payload) * 3 // 4 > max_bytes: + raise _too_large(max_bytes) + try: + data = base64.b64decode(payload, validate=True) + except (binascii.Error, ValueError) as exc: + raise GenerationError("fal.ai returned an inline file this app cannot read") from exc + + media_type = header[len("data:"):].split(";", 1)[0] + destination = workdir / f"{stem}{_extension_for('', media_type, None, default_extension)}" + destination.write_bytes(data) + return destination + + +def _too_large(max_bytes: int) -> GenerationError: + return GenerationError( + f"The generated file is larger than the {max_bytes // (1024 * 1024)} MB limit" + ) + + +def _content_length(response: httpx.Response) -> Optional[int]: + raw = response.headers.get("content-length") + try: + return int(raw) if raw is not None else None + except ValueError: + return None + + +def _extension_for( + url: str, content_type: Optional[str], hint: Optional[str], default: str +) -> str: + from_path = extension_of(urlsplit(url).path) if url else "" + if from_path in MEDIA_EXTENSIONS: + return from_path + for candidate in (content_type, hint): + media_type = (candidate or "").split(";", 1)[0].strip().lower() + guessed = mimetypes.guess_extension(media_type) if media_type else None + if guessed and guessed.lower() in MEDIA_EXTENSIONS: + return guessed.lower() + return default diff --git a/backend/app/generation/errors.py b/backend/app/generation/errors.py new file mode 100644 index 0000000..8a7d592 --- /dev/null +++ b/backend/app/generation/errors.py @@ -0,0 +1,19 @@ +"""The one exception a generation fails with.""" + + +class GenerationError(Exception): + """Something the user can be told about, in a sentence they can act on. + + Its own class rather than a ProviderError so the job runner can catch it by name and + record the message without a traceback: a rejected key or a refused prompt is an + answer, not a crash. + """ + + +class FalUnavailable(GenerationError): + """fal.ai answered 429 or 5xx — worth asking again rather than giving up on. + + Separate because the poll loop treats it so: a video four minutes into generating + should not be abandoned, and billed for nothing, over one bad gateway on a status + check. + """ diff --git a/backend/app/generation/fal.py b/backend/app/generation/fal.py new file mode 100644 index 0000000..408f82d --- /dev/null +++ b/backend/app/generation/fal.py @@ -0,0 +1,280 @@ +"""fal.ai's queue API, spoken over raw httpx. + +Raw rather than through fal's own clients, for gecko-notes' reason, which holds: neither +`fal-client` nor `@fal-ai/client` exposes response headers, and `x-fal-billable-units` +on the result is the whole of exact costing. The four calls are a hundred lines. + +The queue for every kind, never the synchronous `fal.run`: a video takes minutes, a +dropped connection there loses a result fal still bills for, and nothing about a +synchronous call survives a restart. Submit returns a request id and three URLs, which +the job persists before it polls — see `run.py`. + +Every failure is a GenerationError worded for the activity feed. What fal says is kept +where it helps (a 422 names the field it wanted) and replaced where it does not (a 401 +is a key problem, whatever its body says). +""" + +from __future__ import annotations + +import logging +from dataclasses import asdict, dataclass +from typing import Any, Mapping, Optional +from urllib.parse import urlsplit + +import httpx + +from app.config import settings +from app.generation.errors import FalUnavailable, GenerationError +from app.providers import _upstream +from app.providers.base import ProviderError + +logger = logging.getLogger(__name__) + +LABEL = "fal.ai" + +# Submit carries the base images inline — up to ten data URIs of up to 8 MB each — so it +# gets a long budget. The others return a small JSON document. +SUBMIT_TIMEOUT = 120.0 +CALL_TIMEOUT = 30.0 +CANCEL_TIMEOUT = 10.0 + +STATUS_IN_QUEUE = "IN_QUEUE" +STATUS_IN_PROGRESS = "IN_PROGRESS" +STATUS_COMPLETED = "COMPLETED" + +# Namespaces whose app id is three segments rather than two (fal's own clients encode +# the same exception). +_THREE_SEGMENT_NAMESPACES = frozenset({"workflows", "comfy"}) + + +@dataclass(frozen=True) +class QueuedRequest: + """What submit returned: enough to poll, fetch and cancel from another process.""" + + request_id: str + status_url: str + response_url: str + cancel_url: str + + def to_dict(self) -> dict: + return asdict(self) + + @classmethod + def from_dict(cls, data: Any) -> Optional["QueuedRequest"]: + if not isinstance(data, dict): + return None + try: + return cls(**{name: str(data[name]) for name in cls.__dataclass_fields__}) + except KeyError: + return None + + +@dataclass(frozen=True) +class QueueStatus: + status: str + queue_position: Optional[int] = None + # Set alongside COMPLETED when the run failed: fal reports a failure as completed. + error: Optional[str] = None + + +@dataclass(frozen=True) +class QueueResult: + body: dict + # A decimal string in the endpoint's priced unit, when fal sends it. + billable_units: Optional[str] + request_id: Optional[str] + + +def auth_headers(api_key: str) -> dict[str, str]: + """fal's scheme is `Key`, not `Bearer` — sent literally, `id:secret` pairs included.""" + return {"Authorization": f"Key {api_key}"} + + +# ─── the four calls ────────────────────────────────────────────────────────── + + +def submit(endpoint_id: str, body: Mapping[str, Any], *, api_key: str) -> QueuedRequest: + """Queue one request. Retries 429/503 only — nothing is billed on those.""" + url = f"{_queue_base()}/{endpoint_id}" + response = _upstream.post_json( + url, + headers=auth_headers(api_key), + json_body=body, + timeout=SUBMIT_TIMEOUT, + label=LABEL, + ) + if response.status_code >= 400: + raise error_for(response) + + data = _json(response) + request_id = data.get("request_id") + if not isinstance(request_id, str) or not request_id: + raise GenerationError("fal.ai accepted the request but did not say what it was called") + + # fal's own URLs are preferred — the path rule below has exceptions fal may add to — + # but only while they point back at fal. The key goes to whatever is stored here on + # every poll, so a URL elsewhere is replaced rather than trusted. + built = f"{_queue_base()}/{_app_id(endpoint_id)}/requests/{request_id}" + return QueuedRequest( + request_id=request_id, + status_url=_same_origin_or(data.get("status_url"), f"{built}/status"), + response_url=_same_origin_or(data.get("response_url"), built), + cancel_url=_same_origin_or(data.get("cancel_url"), f"{built}/cancel"), + ) + + +def status(request: QueuedRequest, *, api_key: str) -> QueueStatus: + response = _upstream.request( + "GET", + request.status_url, + headers=auth_headers(api_key), + params={"logs": "0"}, + timeout=CALL_TIMEOUT, + label=LABEL, + ) + if response.status_code >= 400: + raise error_for(response) + + data = _json(response) + position = data.get("queue_position") + if isinstance(position, bool) or not isinstance(position, int): + position = None + error = data.get("error") + return QueueStatus( + status=str(data.get("status") or ""), + queue_position=position, + error=str(error) if error else None, + ) + + +def result(request: QueuedRequest, *, api_key: str) -> QueueResult: + response = _upstream.request( + "GET", + request.response_url, + headers=auth_headers(api_key), + timeout=CALL_TIMEOUT, + label=LABEL, + ) + if response.status_code >= 400: + raise error_for(response) + return QueueResult( + body=_json(response), + billable_units=response.headers.get("x-fal-billable-units"), + request_id=response.headers.get("x-fal-request-id"), + ) + + +def cancel(request: QueuedRequest, *, api_key: str) -> bool: + """Ask fal to stop. Best effort, never raises; returns whether fal accepted it. + + Sent on a user's cancel and on GAM's own timeout, because a request abandoned here + keeps running — and billing — there. 202 means accepted, not stopped; a running job + may finish anyway, and nothing here waits to find out. + """ + try: + response = _upstream.request( + "PUT", + request.cancel_url, + headers=auth_headers(api_key), + timeout=CANCEL_TIMEOUT, + label=LABEL, + ) + except ProviderError: + logger.warning("Could not ask fal.ai to cancel request %s", request.request_id) + return False + if response.status_code >= 400: + logger.info( + "fal.ai declined to cancel %s: HTTP %d", request.request_id, response.status_code + ) + return False + return True + + +# ─── errors ────────────────────────────────────────────────────────────────── + + +def error_for(response: httpx.Response) -> GenerationError: + """A fal error response as a sentence somebody can act on.""" + code = response.status_code + detail = _upstream.error_detail(response) + + if code in (401, 403): + # An exhausted balance arrives as a 403 too, as far as integrators report — and + # "check your key" would send somebody to the wrong page. fal's own words are + # the better answer there. + if "balance" in detail.lower(): + return GenerationError(f"fal.ai refused the request: {detail}") + return GenerationError("fal.ai rejected the API key — check it in Settings") + + if code == 422: + return GenerationError( + f"fal.ai refused the request: {_validation_detail(response, detail)}" + ) + + if code == 429 or code >= 500: + return FalUnavailable(f"fal.ai is not available right now ({detail})") + + return GenerationError(f"fal.ai returned an error: {detail}") + + +def _validation_detail(response: httpx.Response, fallback: str) -> str: + """fal's 422 body, `{"detail": [{"loc", "msg"}]}`, as `field: message`. + + `_upstream.error_detail` reads only a string `detail`, so a list falls back to + "HTTP 422" there — which is exactly the case where fal says which field it wanted. + `body` is dropped from the location: every field in a request is in the body. + """ + try: + body = response.json() + except ValueError: + return fallback + items = body.get("detail") if isinstance(body, dict) else None + if not isinstance(items, list): + return fallback + + messages = [] + for item in items: + if not isinstance(item, dict): + continue + location = [str(part) for part in item.get("loc") or [] if part != "body"] + message = str(item.get("msg") or "invalid") + messages.append(f"{'.'.join(location)}: {message}" if location else message) + return "; ".join(messages) or fallback + + +# ─── helpers ───────────────────────────────────────────────────────────────── + + +def _queue_base() -> str: + return settings.fal_queue_base_url.rstrip("/") + + +def _app_id(endpoint_id: str) -> str: + """The part of an endpoint id the queue's request URLs use. + + Only `owner/alias`: `fal-ai/kling-video/v2.5-turbo/pro/image-to-video` polls at + `fal-ai/kling-video/requests/{id}/status`. Both of fal's clients do this. + """ + parts = endpoint_id.split("/") + keep = 3 if parts[0] in _THREE_SEGMENT_NAMESPACES else 2 + return "/".join(parts[:keep]) + + +def _same_origin_or(url: Any, fallback: str) -> str: + if not isinstance(url, str) or not url: + return fallback + given, expected = urlsplit(url), urlsplit(_queue_base()) + if (given.scheme, given.netloc) != (expected.scheme, expected.netloc): + logger.warning("fal.ai returned a queue URL on another host; using the documented one") + return fallback + return url + + +def _json(response: httpx.Response) -> dict: + try: + data = response.json() + except ValueError as exc: + raise GenerationError("fal.ai sent back something that was not JSON") from exc + if not isinstance(data, dict): + raise GenerationError("fal.ai sent back something this app cannot read") + return data diff --git a/backend/app/generation/pricing.py b/backend/app/generation/pricing.py new file mode 100644 index 0000000..e2275b2 --- /dev/null +++ b/backend/app/generation/pricing.py @@ -0,0 +1,254 @@ +"""What a fal.ai generation cost. + +fal reports the billed *quantity* (`x-fal-billable-units`, in the endpoint's own unit) +on the queue result; the *price* per unit comes from its pricing API, fetched with the +user's own key so a negotiated price is the one applied. Units × price is then the bill, +and recorded as exact. Each half degrades separately rather than to nothing: + +| units from | price from | cost | cost_estimated | +|-------------------------|---------------------|--------------|----------------| +| header | pricing API | units × price| False | +| header | catalogue list price| units × price| True | +| outputs (no header) | either | derived × price | True | +| — | no price anywhere | None | None | + +gecko-notes filled its price cache from fal's *usage* API, which needs a billing-scoped +key, only knows endpoints already billed, and dropped prices outside each refresh's +window; the pricing API has none of those problems. Neither the header on the queue +result nor the pricing API on an ordinary key could be verified from where this was +written — both are documented, neither was reachable — which is why both fall back. +""" + +from __future__ import annotations + +import logging +import math +import threading +import time +from dataclasses import dataclass +from typing import Iterable, Optional + +from app.config import settings +from app.generation.fal import LABEL, auth_headers +from app.models.usage import ( + UNIT_GENERATED_SECONDS, + UNIT_IMAGES, + UNIT_MEGAPIXELS, + UNIT_VIDEOS, +) +from app.providers import _upstream +from app.providers.base import ProviderError + +logger = logging.getLogger(__name__) + +PRICE_TTL_SECONDS = 6 * 3600 +PRICING_TIMEOUT = 10.0 + +# fal rounds each image *up* to whole megapixels, and its megapixel is 1024², not 10⁶: +# 1024×1024 bills as one, 1024×1536 as two, 512×512 as one. Computing it any other way +# under-counts (one integrator was 14% low). +PIXELS_PER_MEGAPIXEL = 1024 * 1024 + +_UNIT_TYPES = { + "image": UNIT_IMAGES, + "megapixel": UNIT_MEGAPIXELS, + "second": UNIT_GENERATED_SECONDS, + "video": UNIT_VIDEOS, +} + + +@dataclass(frozen=True) +class UnitPrice: + unit_price: float + unit: str # fal's word for it: "image", "megapixel", "second", "video", or other + currency: str + + +@dataclass(frozen=True) +class OutputFacts: + """What was probed from one downloaded output, for deriving units without a header.""" + + width: Optional[int] = None + height: Optional[int] = None + duration_seconds: Optional[float] = None + + +@dataclass(frozen=True) +class Cost: + units: int + unit_type: str + cost: Optional[float] + currency: Optional[str] + estimated: Optional[bool] + + +# ─── fal's pricing API, cached ─────────────────────────────────────────────── +# +# Per (user, endpoint): the price is fetched with the user's key and "custom pricing or +# discounts may be applied", so one user's price is not another's. In-process only — it +# is six hours of one small number, and a restart refetching it costs one request. + +_cache: dict[tuple[str, str], tuple[float, UnitPrice]] = {} +_lock = threading.Lock() + + +def list_price(user_id: str, endpoint_id: str, *, api_key: str) -> Optional[UnitPrice]: + """fal's own price for this endpoint, or None if its pricing API would not say.""" + key = (user_id, endpoint_id) + now = time.monotonic() + with _lock: + cached = _cache.get(key) + if cached and now - cached[0] < PRICE_TTL_SECONDS: + return cached[1] + + price = _fetch(endpoint_id, api_key) + # Only a found price is cached. A failure is retried on the next generation, which + # costs one small request, rather than pinning six hours of estimates on a blip. + if price is not None: + with _lock: + _cache[key] = (now, price) + return price + + +def clear_cache() -> None: + with _lock: + _cache.clear() + + +def _fetch(endpoint_id: str, api_key: str) -> Optional[UnitPrice]: + try: + response = _upstream.request( + "GET", + f"{settings.fal_api_base_url.rstrip('/')}/v1/models/pricing", + headers=auth_headers(api_key), + params={"endpoint_id": endpoint_id}, + timeout=PRICING_TIMEOUT, + label=f"{LABEL} pricing", + ) + except ProviderError as exc: + logger.info("No price from fal.ai for %s: %s", endpoint_id, exc) + return None + if response.status_code >= 400: + logger.info( + "No price from fal.ai for %s: HTTP %d (%s)", + endpoint_id, + response.status_code, + _upstream.error_detail(response), + ) + return None + + try: + body = response.json() + except ValueError: + return None + for entry in (body.get("prices") if isinstance(body, dict) else None) or []: + if not isinstance(entry, dict) or entry.get("endpoint_id") != endpoint_id: + continue + unit_price = entry.get("unit_price") + if isinstance(unit_price, bool) or not isinstance(unit_price, (int, float)): + continue + return UnitPrice( + unit_price=float(unit_price), + unit=str(entry.get("unit") or ""), + currency=str(entry.get("currency") or "USD"), + ) + return None + + +# ─── the cost of one request ───────────────────────────────────────────────── + + +def compute( + *, + is_video: bool, + billable_units: Optional[str], + api_price: Optional[UnitPrice], + catalogue_price: Optional[UnitPrice], + fallback_unit: Optional[str], + outputs: Iterable[OutputFacts], +) -> Cost: + """Apply the table in the module docstring. + + `units` is an integer column, so it holds the quantity rounded up; `cost` is worked + out from the exact quantity first. `fallback_unit` is the catalogue's `price_unit`, + used to label units when there is no price to say what they are in. + """ + outputs = list(outputs) + price = api_price or catalogue_price + unit = _unit_kind(price.unit if price else fallback_unit) + unit_type = _UNIT_TYPES.get(unit or "", UNIT_VIDEOS if is_video else UNIT_IMAGES) + + quantity = _parse_quantity(billable_units) + from_header = quantity is not None + if quantity is None and unit is not None: + quantity = _derive(unit, outputs) + + if price is None or quantity is None: + # Not priceable. The row is still written — a total that silently left out a + # generation would read as complete — with whatever count there is. + units = quantity if quantity is not None else len(outputs) + return Cost( + units=math.ceil(units), unit_type=unit_type, cost=None, currency=None, estimated=None + ) + + return Cost( + units=math.ceil(quantity), + unit_type=unit_type, + cost=round(quantity * price.unit_price, 6), + currency=price.currency, + # Exact only when both halves came from fal: its count and its price. + estimated=not (from_header and api_price is not None), + ) + + +def _parse_quantity(raw: Optional[str]) -> Optional[float]: + if raw is None: + return None + try: + value = float(raw.strip()) + except (ValueError, AttributeError): + return None + return value if math.isfinite(value) and value >= 0 else None + + +def _unit_kind(unit: Optional[str]) -> Optional[str]: + """fal's free-text unit as one of the four this app can derive and label. + + "compute seconds" and GPU units are not generated seconds and are left unmapped — + a header quantity in them still prices correctly, it just cannot be derived. + """ + text = (unit or "").strip().lower() + if "megapixel" in text: + return "megapixel" + if "compute" in text or "gpu" in text: + return None + if "second" in text: + return "second" + if "video" in text: + return "video" + if "image" in text: + return "image" + return None + + +def _derive(unit: str, outputs: list[OutputFacts]) -> Optional[float]: + """The billed quantity, worked out from the files themselves. + + None when it cannot be: a megapixel price with an image whose size was not probed + is unpriceable, not free. + """ + if not outputs: + return None + if unit in ("image", "video"): + return float(len(outputs)) + if unit == "megapixel": + if any(not o.width or not o.height for o in outputs): + return None + return float( + sum(max(1, math.ceil(o.width * o.height / PIXELS_PER_MEGAPIXEL)) for o in outputs) + ) + if unit == "second": + if any(o.duration_seconds is None for o in outputs): + return None + return float(sum(o.duration_seconds for o in outputs)) + return None diff --git a/backend/app/generation/request.py b/backend/app/generation/request.py new file mode 100644 index 0000000..6cc3a53 --- /dev/null +++ b/backend/app/generation/request.py @@ -0,0 +1,196 @@ +"""What one generation was asked to do, and the checks it passed to be asked. + +The request body is assembled at submit time, against the catalogue row as it was then, +and travels in the job's payload. The job never re-reads the row for its dialect: an +admin editing a model while a video is queued should not change what that video is. +""" + +from __future__ import annotations + +from dataclasses import asdict, dataclass, field +from typing import Any, Optional + +from app.generation.catalogue import CHOICES, extra_params_of +from app.generation.errors import GenerationError +from app.models.generation import ( + KIND_IMAGE_TO_IMAGE, + KIND_IMAGE_TO_VIDEO, + KIND_TEXT_TO_IMAGE, + GenerationModel, +) + +MAX_PROMPT_CHARS = 4000 +NAME_LIMIT = 80 + +ROLE_BASE = "base" +ROLE_END_FRAME = "end_frame" + + +class InvalidGeneration(ValueError): + """A request the catalogue row says the endpoint would refuse. The message says why.""" + + +@dataclass(frozen=True) +class GenerationRequest: + model_id: str + endpoint_id: str + kind: str + prompt: str + # The request body minus the prompt and the image fields: catalogue defaults, then + # the user's choices. Exactly what `Asset.ai_parameters` will hold. + parameters: dict + image_field: Optional[str] + image_field_is_list: bool + end_image_field: Optional[str] + # [{"asset_id", "role", "name"}], bases in order and then the end frame. `name` is + # read at submit so the finished asset can say what it was made from even if the + # base is deleted while the job waits. + sources: list = field(default_factory=list) + # The generated asset a regeneration started from, for the record. + regenerated_from: Optional[str] = None + + @property + def base_asset_ids(self) -> list[str]: + return [s["asset_id"] for s in self.sources if s.get("role") == ROLE_BASE] + + @property + def end_frame_asset_id(self) -> Optional[str]: + ends = [s["asset_id"] for s in self.sources if s.get("role") == ROLE_END_FRAME] + return ends[0] if ends else None + + def to_dict(self) -> dict: + return asdict(self) + + @classmethod + def from_dict(cls, data: Any) -> "GenerationRequest": + try: + request = cls(**{name: data[name] for name in cls.__dataclass_fields__ if name in data}) + except (TypeError, KeyError) as exc: + raise GenerationError("This generation job has no request to send") from exc + if not request.endpoint_id or not isinstance(request.parameters, dict): + raise GenerationError("This generation job has no request to send") + return request + + +def check_prompt(prompt: Optional[str]) -> str: + text = (prompt or "").strip() + if not text: + raise InvalidGeneration("Write a prompt first") + if len(text) > MAX_PROMPT_CHARS: + raise InvalidGeneration( + f"The prompt is {len(text):,} characters; the limit is {MAX_PROMPT_CHARS:,}" + ) + return text + + +def check_shape(row: GenerationModel, base_count: int, has_end_frame: bool) -> None: + """The right number of bases for the kind, and an end frame only where one fits.""" + if row.kind == KIND_TEXT_TO_IMAGE and base_count: + raise InvalidGeneration( + f"{row.label} makes an image from a prompt alone and takes no base image" + ) + if row.kind == KIND_IMAGE_TO_IMAGE and not 1 <= base_count <= row.max_images: + raise InvalidGeneration( + f"{row.label} takes one base image" + if row.max_images == 1 + else f"{row.label} takes between 1 and {row.max_images} base images" + ) + if row.kind == KIND_IMAGE_TO_VIDEO and base_count != 1: + raise InvalidGeneration(f"{row.label} animates exactly one start image") + if has_end_frame and not row.end_image_field: + raise InvalidGeneration(f"{row.label} does not take an end frame") + + +def build_parameters(row: GenerationModel, options: dict, params: dict) -> dict: + """Catalogue defaults, then the user's choices, each checked against the row. + + A choice is sent as the row declares it, not as it arrived: `"5"` and `5` are + different requests to fal, and the declared value is the one known to work. + """ + body = extra_params_of(row) + + for param, key in CHOICES: + value = params.get(param) + if value is None: + continue + allowed = options[key] + name = param.replace("_", " ") + if not allowed: + article = "an" if name[0] in "aeiou" else "a" + raise InvalidGeneration(f"{row.label} does not take {article} {name}") + match = next((option for option in allowed if _same(option, value)), _NO_MATCH) + if match is _NO_MATCH: + raise InvalidGeneration( + f"{name.capitalize()} must be one of: " + ", ".join(str(option) for option in allowed) + ) + body[param] = match + + if params.get("seed") is not None: + if not options["supports_seed"]: + raise InvalidGeneration(f"{row.label} does not take a seed") + body["seed"] = params["seed"] + + # An empty negative prompt is the form's way of saying "none"; it is not a request + # for a feature the model lacks. + negative = params.get("negative_prompt") + if negative is not None and negative.strip(): + if not options["supports_negative_prompt"]: + raise InvalidGeneration(f"{row.label} does not take a negative prompt") + body["negative_prompt"] = negative.strip() + + if params.get("generate_audio") is not None: + if not options["supports_audio"]: + raise InvalidGeneration(f"{row.label} does not generate audio") + body["generate_audio"] = params["generate_audio"] + + num_outputs = params.get("num_outputs", 1) + max_outputs = options["max_outputs"] + if not 1 <= num_outputs <= max_outputs: + raise InvalidGeneration( + f"{row.label} makes one at a time" + if max_outputs == 1 + else f"Ask {row.label} for between 1 and {max_outputs} images" + ) + # Sent only where the row declares more than one is possible: a row whose model has + # no `num_images` field has max_outputs 1, and the field is not sent to it at all. + if row.kind != KIND_IMAGE_TO_VIDEO and max_outputs > 1: + body["num_images"] = num_outputs + + return without_app_fields(body, row.image_field, row.end_image_field) + + +def without_app_fields(body: dict, *image_fields: Optional[str]) -> dict: + """The parameters with the fields the app owns taken out. + + The prompt and the images are added last when the request is sent, so a catalogue + default could not override them anyway; removing them here keeps them out of + `ai_parameters`, where a regeneration would otherwise resend a stale copy. + """ + owned = {"prompt", *(f for f in image_fields if f)} + return {key: value for key, value in body.items() if key not in owned} + + +def name_from_prompt(prompt: str, limit: int = NAME_LIMIT) -> str: + """The prompt cut at a word boundary, for an asset or activity name.""" + text = " ".join(prompt.split()) + if len(text) <= limit: + return text + cut = text[: limit - 1] + space = cut.rfind(" ") + # Only back off to a space that leaves most of the budget; one enormous word is + # cut mid-word rather than reduced to almost nothing. + if space > limit // 2: + cut = cut[:space] + return cut.rstrip(" ,.;:-") + "…" + + +_NO_MATCH = object() + + +def _same(declared: Any, given: Any) -> bool: + """Equal and of the same kind: "5" is not 5, and True is not 1.""" + if isinstance(declared, bool) or isinstance(given, bool): + return declared is given + if isinstance(declared, str) or isinstance(given, str): + return isinstance(declared, str) and isinstance(given, str) and declared == given + return declared == given diff --git a/backend/app/generation/run.py b/backend/app/generation/run.py new file mode 100644 index 0000000..d9dda6f --- /dev/null +++ b/backend/app/generation/run.py @@ -0,0 +1,499 @@ +"""The generate job: a request to fal.ai in, library assets out. + +In order, each a stage the activity feed shows: + +1. Preparing the base images — each normalised to a data URI (`bases.py`). +2. Submitting to fal.ai — and the request ids **committed to the payload before the + first poll**. `recover_pending` re-runs a job from the top; one that finds a stored + request resumes polling it rather than submitting, and paying for, a second. +3. Waiting in fal.ai's queue / Generating — polled every `generation_poll_seconds` + until fal says COMPLETED, which is also how it reports a failure, or until GAM's own + deadline. On that deadline, and on a user's cancel, fal is told to stop: a request + abandoned here keeps running, and billing, there. +4. Downloading — each output streamed under the size cap (`download.py`). +5. Saving — through `ingest_file`, the path an upload takes, so it is probed, + thumbnailed and indexed like anything else; then the `ai_*` provenance, and the + prompt as name and description so search finds it by what was asked for. Each asset + id is written to the payload as it is made, so a restart here finishes the job + instead of saving the same image twice. +6. Recording the cost — once, guarded by `usage_recorded` in the same commit as the row. + +Runs on a worker thread of the generation queue. `progress` is the job's reporter and +the cancellation checkpoint, so it is called on every poll and between download chunks. +""" + +from __future__ import annotations + +import json +import logging +import tempfile +import time +from dataclasses import dataclass, field +from datetime import datetime, timedelta +from pathlib import Path +from typing import Any, Callable, Optional + +from sqlmodel import Session, select + +from app.clock import utcnow +from app.config import settings +from app.generation import bases, fal, pricing +from app.generation.download import download +from app.generation.errors import FalUnavailable, GenerationError +from app.generation.request import GenerationRequest, name_from_prompt +from app.ingest.filetypes import SOURCE_AI +from app.jobs.runner import JobCancelled, set_fields +from app.models.asset import Asset +from app.models.generation import KIND_IMAGE_TO_VIDEO, KIND_TEXT_TO_IMAGE, GenerationModel +from app.models.job import EnrichmentJob +from app.models.usage import KIND_IMAGE, KIND_VIDEO +from app.providers.base import ProviderError +from app.services import assets as asset_service +from app.settings_store import load_fal_key +from app.storage import StorageError, build_storage +from app.usage import events as usage_events + +logger = logging.getLogger(__name__) + +Progress = Callable[[str, int, str], None] + +# Looked up on the module at call time, so a test replaces the wait between polls and +# the clock it is measured against, and a thirty-minute timeout runs in milliseconds. +sleep = time.sleep +now = utcnow + +STAGE_PREPARING = "Preparing the base images" +STAGE_SUBMITTING = "Submitting to fal.ai" +STAGE_QUEUED = "Waiting in fal.ai's queue" +STAGE_GENERATING = "Generating" +STAGE_DOWNLOADING = "Downloading" +STAGE_SAVING = "Saving" +STAGE_COST = "Recording the cost" + +# How long a generation is expected to take, for a progress bar that creeps rather than +# sits still. Only shapes the curve; nothing waits on it. +_EXPECTED_SECONDS = {KIND_IMAGE_TO_VIDEO: 180.0} +_EXPECTED_SECONDS_DEFAULT = 20.0 + +# Report a download's progress at most this often, in bytes: each report is a write. +_REPORT_EVERY_BYTES = 4 * 1024 * 1024 + +# fal's seed is a JSON number; SQLite's INTEGER is 64-bit. +_MAX_SEED = 2**63 - 1 + + +@dataclass +class GenerationResult: + detail: str + created_assets: list[Asset] = field(default_factory=list) + + +@dataclass +class _State: + """The job's payload, decoded. Written back after every step that must survive.""" + + request: GenerationRequest + queued: Optional[fal.QueuedRequest] = None + submitted_at: Optional[datetime] = None + # What fal returned, trimmed to what the rest of the job reads. Kept so a restart + # after the result was fetched does not depend on fal still serving it. + result: Optional[dict] = None + created_asset_ids: list[str] = field(default_factory=list) + usage_recorded: bool = False + + @classmethod + def load(cls, payload: Optional[str]) -> "_State": + try: + data = json.loads(payload or "{}") + except ValueError: + data = {} + if not isinstance(data, dict) or not isinstance(data.get("request"), dict): + raise GenerationError("This generation job has no request to send") + submitted = data.get("submitted_at") + try: + submitted_at = datetime.fromisoformat(submitted) if isinstance(submitted, str) else None + except ValueError: + submitted_at = None + created = data.get("created_asset_ids") + if not isinstance(created, list): + created = [] + return cls( + request=GenerationRequest.from_dict(data["request"]), + queued=fal.QueuedRequest.from_dict(data.get("fal_request")), + submitted_at=submitted_at, + result=data.get("result") if isinstance(data.get("result"), dict) else None, + created_asset_ids=[i for i in created if isinstance(i, str)], + usage_recorded=bool(data.get("usage_recorded")), + ) + + def encode(self) -> str: + data: dict[str, Any] = { + "request": self.request.to_dict(), + "created_asset_ids": self.created_asset_ids, + "usage_recorded": self.usage_recorded, + } + if self.queued is not None: + data["fal_request"] = self.queued.to_dict() + if self.submitted_at is not None: + data["submitted_at"] = self.submitted_at.isoformat() + if self.result is not None: + data["result"] = self.result + return json.dumps(data, sort_keys=True) + + +def run(session: Session, job: EnrichmentJob, progress: Progress) -> GenerationResult: + state = _State.load(job.payload) + request = state.request + + api_key = load_fal_key(session, job.user_id) + if not api_key: + raise GenerationError("No fal.ai API key is configured. Add one in Settings.") + + if state.result is None: + if state.queued is None: + body = _request_body(session, job, request, progress) + progress(STAGE_SUBMITTING, 10, "") + state.queued = fal.submit(request.endpoint_id, body, api_key=api_key) + state.submitted_at = now() + _save(session, job, state) + logger.info("Generation %s submitted to fal.ai as %s", job.id, state.queued.request_id) + else: + logger.info( + "Generation %s resuming fal.ai request %s", job.id, state.queued.request_id + ) + + fetched = _await_result(state, request, api_key, progress) + state.result = _trim(fetched) + _save(session, job, state) + + outputs = state.result.get("outputs") or [] + if not outputs: + raise GenerationError("fal.ai finished but returned nothing to download") + + created = _save_outputs(session, job, state, outputs, progress) + + progress(STAGE_COST, 97, "") + if not state.usage_recorded: + try: + _record_cost(session, job, state, created, api_key) + except Exception: # noqa: BLE001 - accounting must not undo the work it measures + # `usage_events.record` already holds this rule for the write; the price + # lookup before it needs the same, or a pricing hiccup fails a generation + # whose files are already in the library. + session.rollback() + logger.warning("Could not record the cost of generation %s", job.id, exc_info=True) + + noun = "video" if request.kind == KIND_IMAGE_TO_VIDEO else "image" + count = len(created) + return GenerationResult( + detail=f"{count} {noun}{'' if count == 1 else 's'}", + created_assets=created, + ) + + +# ─── submitting ────────────────────────────────────────────────────────────── + + +def _request_body( + session: Session, job: EnrichmentJob, request: GenerationRequest, progress: Progress +) -> dict: + """Catalogue defaults and choices first, the app's own fields last, so nothing in + `parameters` can override the prompt or the images.""" + body = dict(request.parameters) + + if request.kind != KIND_TEXT_TO_IMAGE and request.image_field: + base_ids = request.base_asset_ids + progress(STAGE_PREPARING, 3, f"{len(base_ids)} image{'' if len(base_ids) == 1 else 's'}") + storage = build_storage() + uris = [_base_uri(session, storage, job.user_id, asset_id) for asset_id in base_ids] + body[request.image_field] = uris if request.image_field_is_list else uris[0] + end_frame = request.end_frame_asset_id + if end_frame and request.end_image_field: + body[request.end_image_field] = _base_uri(session, storage, job.user_id, end_frame) + + body["prompt"] = request.prompt + return body + + +def _base_uri(session: Session, storage, user_id: str, asset_id: str) -> str: + asset = session.get(Asset, asset_id) + if asset is None or asset.user_id != user_id or not asset.storage_key: + raise GenerationError("A base image was deleted before this could run") + try: + with storage.materialise(asset.storage_key) as path: + return bases.data_uri( + path, + max_edge=settings.generation_base_max_edge, + max_bytes=settings.generation_max_data_uri_mb * 1024 * 1024, + ) + except (StorageError, OSError) as exc: + raise GenerationError(f"The file for the base image “{asset.name}” is missing") from exc + + +# ─── waiting ───────────────────────────────────────────────────────────────── + + +def _await_result( + state: _State, request: GenerationRequest, api_key: str, progress: Progress +) -> fal.QueueResult: + queued = state.queued + submitted_at = state.submitted_at or now() + minutes = settings.generation_timeout_minutes + deadline = submitted_at + timedelta(minutes=minutes) + expected = _EXPECTED_SECONDS.get(request.kind, _EXPECTED_SECONDS_DEFAULT) + stage, percent, detail = STAGE_QUEUED, 15, "" + + try: + while True: + try: + current = fal.status(queued, api_key=api_key) + if current.status == fal.STATUS_COMPLETED: + # A failed run is reported as COMPLETED too, with the reason beside + # it — checked before fetching a result that is not there. + if current.error: + raise GenerationError(f"fal.ai could not generate this: {current.error}") + progress(STAGE_GENERATING, 70, "Finished") + return fal.result(queued, api_key=api_key) + + if current.status == fal.STATUS_IN_QUEUE: + stage, percent = STAGE_QUEUED, 15 + detail = _queue_detail(current.queue_position) + else: + # IN_PROGRESS, or a status fal's clients do not know — treated as + # still running, with the deadline as the backstop. + elapsed = max(0.0, (now() - submitted_at).total_seconds()) + stage = STAGE_GENERATING + percent = 20 + int(50 * elapsed / (elapsed + expected)) + detail = f"{int(elapsed) // 60}:{int(elapsed) % 60:02d} so far" + except (FalUnavailable, ProviderError) as exc: + # A status check that failed says nothing about the generation, which + # is still running and still billed. Ask again until the deadline. + logger.info("fal.ai status check failed for %s: %s", queued.request_id, exc) + detail = "fal.ai is not answering; still trying" + + progress(stage, percent, detail) + + if now() >= deadline: + fal.cancel(queued, api_key=api_key) + raise GenerationError(f"fal.ai did not finish within {minutes} minutes") + + sleep(settings.generation_poll_seconds) + except JobCancelled: + fal.cancel(queued, api_key=api_key) + raise + + +def _queue_detail(position: Optional[int]) -> str: + if position is None: + return "" + if position <= 0: + return "Next in line" + return f"Position {position + 1} in the queue" + + +def _trim(fetched: fal.QueueResult) -> dict: + """The parts of fal's answer the rest of the job reads, in a form that persists.""" + body = fetched.body + outputs: list[dict] = [] + for key in ("images", "videos"): + for item in body.get(key) or []: + outputs.append(item) + for key in ("image", "video"): + if isinstance(body.get(key), dict): + outputs.append(body[key]) + + seed = body.get("seed") + if isinstance(seed, bool) or not isinstance(seed, int) or abs(seed) > _MAX_SEED: + seed = None + return { + "outputs": [ + {"url": item["url"], "content_type": _text_or_none(item.get("content_type"))} + for item in outputs + if isinstance(item, dict) and isinstance(item.get("url"), str) and item["url"] + ], + "seed": seed, + "billable_units": fetched.billable_units, + "fal_request_id": fetched.request_id, + } + + +def _text_or_none(value: Any) -> Optional[str]: + return value if isinstance(value, str) else None + + +# ─── saving ────────────────────────────────────────────────────────────────── + + +def _save_outputs( + session: Session, + job: EnrichmentJob, + state: _State, + outputs: list[dict], + progress: Progress, +) -> list[Asset]: + request = state.request + # Made by an earlier run of this job before a restart. One the user has deleted + # since is simply not there; it is not made again. + created = [a for a in (session.get(Asset, i) for i in state.created_asset_ids) if a is not None] + + is_video = request.kind == KIND_IMAGE_TO_VIDEO + default_extension = ".mp4" if is_video else ".png" + total = len(outputs) + storage = build_storage() + + with tempfile.TemporaryDirectory(prefix="gam-generate-") as workdir: + for index, output in enumerate(outputs): + if index < len(state.created_asset_ids): + continue + + label = f"{index + 1} of {total} · " if total > 1 else "" + start = 72 + int(16 * index / total) + span = max(1, int(16 / total)) + progress(STAGE_DOWNLOADING, start, label.rstrip(" ·")) + + path = download( + output["url"], + Path(workdir), + f"output-{index}", + max_bytes=settings.generation_max_download_mb * 1024 * 1024, + default_extension=default_extension, + content_type_hint=output.get("content_type"), + on_progress=_download_reporter(progress, label, start, span), + ) + + progress(STAGE_SAVING, 90, label.rstrip(" ·")) + asset = _ingest(session, storage, job, request, state, path) + created.append(asset) + state.created_asset_ids.append(asset.id) + _save(session, job, state) + return created + + +def _download_reporter(progress: Progress, label: str, start: int, span: int): + last = {"bytes": -_REPORT_EVERY_BYTES} + + def report(received: int, total: Optional[int]) -> None: + if received - last["bytes"] < _REPORT_EVERY_BYTES and received != total: + return + last["bytes"] = received + megabytes = received / (1024 * 1024) + if total: + percent = start + int(span * received / total) + detail = f"{label}{megabytes:.1f} of {total / (1024 * 1024):.1f} MB" + else: + percent = start + detail = f"{label}{megabytes:.1f} MB" + progress(STAGE_DOWNLOADING, percent, detail) + + return report + + +def _ingest( + session: Session, + storage, + job: EnrichmentJob, + request: GenerationRequest, + state: _State, + path: Path, +) -> Asset: + name = name_from_prompt(request.prompt) + try: + asset = asset_service.ingest_file( + session, + storage, + user_id=job.user_id, + source_path=path, + filename=_filename_for(name, path), + name=name, + source=SOURCE_AI, + ) + except asset_service.UnsupportedFile as exc: + raise GenerationError( + f"fal.ai delivered a {path.suffix or 'nameless'} file, " + "which the library does not accept" + ) from exc + + asset.ai_model = request.endpoint_id + asset.ai_generation_type = request.kind + asset.ai_prompt = request.prompt + asset.ai_parameters = json.dumps(request.parameters, sort_keys=True) + asset.ai_source_assets = json.dumps(request.sources) + asset.ai_seed = (state.result or {}).get("seed") + asset.ai_generated_at = now() + # Through the AI writer rather than set directly, for its stamp and its re-index: + # `field_provenance` records that a model, not a person, wrote these two, and the + # keyword index — built by ingest before there was a description — learns the + # prompt. The `ai_*` columns above ride in on the same commit. + asset_service.apply_ai_metadata(session, asset, {"name": name, "description": request.prompt}) + return asset + + +def _filename_for(name: str, path: Path) -> str: + """The display filename: the name, with the real extension (which decides the + type). Slashes replaced for the reason `import_url._filename_for` gives.""" + stem = name.replace("/", "-").replace("\\", "-").rstrip("…").strip() or "generated" + return f"{stem[:200]}{path.suffix.lower()}" + + +# ─── the bill ──────────────────────────────────────────────────────────────── + + +def _record_cost( + session: Session, job: EnrichmentJob, state: _State, created: list[Asset], api_key: str +) -> None: + request = state.request + result = state.result or {} + is_video = request.kind == KIND_IMAGE_TO_VIDEO + + row = session.exec( + select(GenerationModel).where(GenerationModel.endpoint_id == request.endpoint_id) + ).first() + catalogue_price = ( + pricing.UnitPrice( + unit_price=row.unit_price, + unit=row.price_unit or "", + currency=row.price_currency or "USD", + ) + if row is not None and row.unit_price is not None and row.price_unit + else None + ) + + cost = pricing.compute( + is_video=is_video, + billable_units=result.get("billable_units"), + api_price=pricing.list_price(job.user_id, request.endpoint_id, api_key=api_key), + catalogue_price=catalogue_price, + fallback_unit=row.price_unit if row is not None else None, + outputs=[ + pricing.OutputFacts(width=a.width, height=a.height, duration_seconds=a.duration_seconds) + for a in created + ], + ) + + # The flag goes into the same transaction as the row: `record` commits once, and + # that commit carries this payload change with it. Two separate commits would leave + # a window where a restart finds the row written and the flag not, and bills twice. + # If `record` fails it rolls back both, which is the consistent outcome too. + state.usage_recorded = True + job.payload = state.encode() + session.add(job) + usage_events.record( + session, + user_id=job.user_id, + asset_id=created[0].id if created else None, + kind=KIND_VIDEO if is_video else KIND_IMAGE, + units=cost.units, + unit_type=cost.unit_type, + provider="fal.ai", + model=request.endpoint_id, + cost=cost.cost, + currency=cost.currency, + cost_estimated=cost.estimated, + # The id fal bills under; the queue's own id if the result did not carry one. + external_ref=result.get("fal_request_id") + or (state.queued.request_id if state.queued else None), + ) + + +def _save(session: Session, job: EnrichmentJob, state: _State) -> None: + set_fields(session, job, payload=state.encode()) diff --git a/backend/app/jobs/enrichment.py b/backend/app/jobs/enrichment.py index 29384c5..a610070 100644 --- a/backend/app/jobs/enrichment.py +++ b/backend/app/jobs/enrichment.py @@ -3,6 +3,12 @@ One queue rather than one per action, because they contend for the same things — CPU for ffmpeg, and a rate-limited upstream — and separate queues would each honour their own cap while collectively ignoring it. + +The exception is generation (M8), which gets a second queue over the same table. It +contends for none of those things: a generation is minutes of waiting on fal.ai with +the CPU idle, and sharing the one worker would leave every transcription queued behind +a video. Both queues run the same `_run`; the kind filter on each is what stops them +recovering or sweeping each other's rows. """ from __future__ import annotations @@ -35,6 +41,8 @@ from app.enrichment.summarize import run as run_summarize from app.enrichment.transcribe import TranscriptionError from app.enrichment.transcribe import run as run_transcribe +from app.generation.errors import GenerationError +from app.generation.run import run as run_generate from app.ingest.ytdlp import UrlImportError from app.providers.base import ProviderError from app.jobs.runner import ( @@ -56,6 +64,8 @@ KIND_EMBED, KIND_EXTRACT_SUBVIDEO, KIND_EXTRACT_TEXT, + GENERATION_JOB_KINDS, + KIND_GENERATE, KIND_GENERATE_ALL, KIND_IMPORT_URL, KIND_SUMMARIZE, @@ -66,6 +76,7 @@ logger = logging.getLogger(__name__) _queue: Optional[JobQueue] = None +_generation_queue: Optional[JobQueue] = None def queue() -> JobQueue: @@ -82,16 +93,40 @@ def queue() -> JobQueue: name="enrichment", concurrency=settings.enrichment_concurrency, engine=engine, + exclude_kinds=GENERATION_JOB_KINDS, ) return _queue +def generation_queue() -> JobQueue: + """The process's generation queue, built on first use, for the same reason.""" + global _generation_queue + if _generation_queue is None: + _generation_queue = JobQueue( + EnrichmentJob, + _run_generation_job, + name="generation", + concurrency=settings.generation_concurrency, + engine=engine, + kinds=GENERATION_JOB_KINDS, + ) + return _generation_queue + + +def queue_for(kind: str) -> JobQueue: + """Whichever queue runs this kind — and therefore the one that can cancel it.""" + return generation_queue() if kind in GENERATION_JOB_KINDS else queue() + + def start() -> None: + # Both from here, so the one `start` the app lifespan calls — and the one conftest + # stubs out — governs every worker thread there is. queue().start() + generation_queue().start() -def enqueue(job_id: str) -> None: - queue().enqueue(job_id) +def enqueue(job_id: str, kind: Optional[str] = None) -> None: + (queue_for(kind) if kind else queue()).enqueue(job_id) def active_job(session: Session, asset_id: str, kind: str) -> Optional[EnrichmentJob]: @@ -157,7 +192,7 @@ def submit_library( session.commit() session.refresh(job) - enqueue(job.id) + enqueue(job.id, job.kind) return job @@ -187,18 +222,28 @@ def submit( session.commit() session.refresh(job) - enqueue(job.id) + enqueue(job.id, job.kind) return job def _run_job(job_id: str) -> None: + """The enrichment queue's worker function.""" + _run(job_id, queue()) + + +def _run_generation_job(job_id: str) -> None: + """The generation queue's. Same body; the queue is what differs — its reporter is + the one whose cancel set the API writes to for this kind.""" + _run(job_id, generation_queue()) + + +def _run(job_id: str, q: JobQueue) -> None: """Run one job to completion, recording whatever happened. Every exit writes a terminal status. A job that ends without one sits at "processing" until the stale sweeper ends it forty minutes later, which reads to the user as a hang. """ - q = queue() progress = q.reporter(job_id) with q.session() as session: @@ -242,6 +287,9 @@ def _run_job(job_id: str) -> None: # An import that stored the site's captions as a transcript, to embed afterwards # the same way a finished transcription is. captioned: Optional[Asset] = None + # What a generation made, embedded afterwards for the same reason: its prompt + # is its description, and semantic search reads that too. + generated: list[Asset] = [] try: if job.kind == KIND_TRANSCRIBE: @@ -298,6 +346,12 @@ def _run_job(job_id: str) -> None: # finished row, which is the first thing to look at when an import # produced something unexpected. job.payload = _with_created_asset(job.payload, imported.created_asset_id) + elif job.kind == KIND_GENERATE: + # The created ids are already in the payload — the job writes each as it + # is made, so a restart mid-way knows what not to make again. + outcome = run_generate(session, job, progress) + detail = outcome.detail + generated = outcome.created_assets elif job.kind == KIND_HARVEST_ATTRIBUTION: harvested = run_harvest_attribution(session, job.user_id, progress) detail = ( @@ -328,6 +382,8 @@ def _run_job(job_id: str) -> None: _chain_embedding(session, asset) elif captioned is not None: _chain_embedding(session, captioned) + for asset_made in generated: + _chain_embedding(session, asset_made) except JobCancelled: _mark_asset_failed(session, asset, job, status=None) @@ -375,6 +431,13 @@ def _run_job(job_id: str) -> None: set_fields(session, job, status="error", stage="", error_message=message) logger.info("Enrichment job %s failed: %s", job_id, message) + except GenerationError as exc: + # A rejected key, a refused prompt, a timeout: an answer from fal or from the + # job's own limits, already worded for the activity feed. No traceback. + message = str(exc) + set_fields(session, job, status="error", stage="", error_message=message) + logger.info("Generation job %s failed: %s", job_id, message) + except TranscriptionError as exc: message = str(exc) _mark_asset_failed(session, asset, job) diff --git a/backend/app/jobs/registry.py b/backend/app/jobs/registry.py index c893e3f..aad0b73 100644 --- a/backend/app/jobs/registry.py +++ b/backend/app/jobs/registry.py @@ -21,19 +21,22 @@ from sqlmodel import Session, col, select from app.jobs.runner import ACTIVE_STATUSES, JobQueue, is_stale -from app.models.job import KIND_EXTRACT_SUBVIDEO, KIND_IMPORT_URL, EnrichmentJob +from app.models.job import KIND_EXTRACT_SUBVIDEO, KIND_GENERATE, KIND_IMPORT_URL, EnrichmentJob from app.schemas_jobs import ActivityJobRead class JobKind: - """One row in the union: a table, a serialiser, and the queue that can stop it.""" + """One row in the union: a table, a serialiser, and the queue that can stop it. + + `queue_for` is handed the row, since which queue runs it can depend on the row. + """ def __init__( self, key: str, model: Type[Any], to_activity: Callable[[Any], ActivityJobRead], - queue_for: Callable[[], Optional[JobQueue]], + queue_for: Callable[[Any], Optional[JobQueue]], ) -> None: self.key = key self.model = model @@ -41,30 +44,41 @@ def __init__( self.queue_for = queue_for -# Kinds whose finished payload names the asset they made. Both create a row that did -# not exist when the job was queued, so `asset_id` cannot carry it: for an extraction it -# stays pointed at the source, and an import has no asset at all until it is done. -_CREATES_AN_ASSET = frozenset({KIND_EXTRACT_SUBVIDEO, KIND_IMPORT_URL}) +# Kinds whose payload names the asset(s) they made. Each creates rows that did not exist +# when the job was queued, so `asset_id` cannot carry them: for an extraction it stays +# pointed at the source, and an import or a generation has no asset at all until it is +# done. +_CREATES_AN_ASSET = frozenset({KIND_EXTRACT_SUBVIDEO, KIND_IMPORT_URL, KIND_GENERATE}) + +def _result_asset_ids(job: EnrichmentJob) -> List[str]: + """Every asset the job created, in the order it created them. -def _result_asset_id(job: EnrichmentJob) -> Optional[str]: - """The asset an "extract" sub-video job or a URL import created, once it has. + An extraction or an import makes one and records it as `created_asset_id`; a + generation can make up to four and records `created_asset_ids`, appended as each is + saved — so a running generation already reports the ones it has. Best-effort, the same defensive shape `routers/transcripts.py::_words_of` uses to read a JSON-as-TEXT column that might be empty, or — for every other kind, whose payload means something else entirely — simply not have this key. """ if job.kind not in _CREATES_AN_ASSET or not job.payload: - return None + return [] try: data = json.loads(job.payload) except ValueError: - return None - value = data.get("created_asset_id") if isinstance(data, dict) else None - return value if isinstance(value, str) else None + return [] + if not isinstance(data, dict): + return [] + many = data.get("created_asset_ids") + if isinstance(many, list): + return [value for value in many if isinstance(value, str)] + one = data.get("created_asset_id") + return [one] if isinstance(one, str) else [] def _enrichment_to_activity(job: EnrichmentJob) -> ActivityJobRead: + created = _result_asset_ids(job) return ActivityJobRead( id=job.id, kind="enrichment", @@ -80,19 +94,22 @@ def _enrichment_to_activity(job: EnrichmentJob) -> ActivityJobRead: asset_id=job.asset_id, asset_name=job.asset_name or "", model=job.model or "", - result_asset_id=_result_asset_id(job), + result_asset_id=created[0] if created else None, + result_asset_ids=created, error_message=job.error_message, created_at=job.created_at, updated_at=job.updated_at, ) -def _enrichment_queue() -> Optional[JobQueue]: +def _enrichment_queue(job: EnrichmentJob) -> Optional[JobQueue]: # Imported lazily: the worker pulls in ffmpeg helpers and the Deepgram client, and # the activity API has no reason to load either just to list a row. from app.jobs import enrichment - return enrichment.queue() + # By the row's kind, because one table now feeds two queues and a cancel sent to + # the one not running the job would be silently ignored. + return enrichment.queue_for(job.kind) KINDS: Dict[str, JobKind] = { @@ -162,7 +179,7 @@ def cancel_job(session: Session, user_id: str, kind_key: str, job_id: str) -> Op return None if row.status in ACTIVE_STATUSES: - queue = kind.queue_for() + queue = kind.queue_for(row) if queue is not None: queue.cancel(job_id) set_fields(session, row, status="cancelled", stage="", detail="Cancelled") diff --git a/backend/app/jobs/runner.py b/backend/app/jobs/runner.py index c6fe072..87a6f96 100644 --- a/backend/app/jobs/runner.py +++ b/backend/app/jobs/runner.py @@ -28,7 +28,7 @@ import threading import time from datetime import timedelta -from typing import Any, Callable, Optional, Set, Type +from typing import Any, Callable, Iterable, List, Optional, Set, Type from sqlalchemy.engine import Engine from sqlmodel import Session, select @@ -103,7 +103,15 @@ class JobCancelled(Exception): class JobQueue: - """One queue, its worker threads, and the cancellation set for a job table.""" + """One queue, its worker threads, and the cancellation set for a job table. + + `kinds` / `exclude_kinds` let two queues share one table (M8): generation has its + own workers so a four-minute video cannot hold every transcription behind it, but + its rows live in `EnrichmentJob` like everything else. The filter is applied + wherever a queue goes looking for rows on its own — restart recovery and the stale + sweep — because without it each queue would requeue the other's work on start and + run it on the wrong workers. Unfiltered (both None) is every row, as before. + """ def __init__( self, @@ -113,12 +121,16 @@ def __init__( name: str, concurrency: int = 1, engine: Optional[Engine] = None, + kinds: Optional[Iterable[str]] = None, + exclude_kinds: Optional[Iterable[str]] = None, ) -> None: self.model = model self.run = run self.name = name self.concurrency = max(1, concurrency) self.engine = engine or default_engine + self.kinds = frozenset(kinds) if kinds is not None else None + self.exclude_kinds = frozenset(exclude_kinds) if exclude_kinds else None self._queue: "queue.Queue[str]" = queue.Queue() self._cancelled: Set[str] = set() self._lock = threading.Lock() @@ -127,6 +139,15 @@ def __init__( def session(self) -> Session: return Session(self.engine) + def _active_rows(self, session: Session) -> List[Any]: + """Every active row this queue owns.""" + query = select(self.model).where(self.model.status.in_(ACTIVE_STATUSES)) + if self.kinds is not None: + query = query.where(self.model.kind.in_(self.kinds)) + if self.exclude_kinds is not None: + query = query.where(self.model.kind.not_in(self.exclude_kinds)) + return list(session.exec(query).all()) + # ─── control ───────────────────────────────────────────────────────────── def enqueue(self, job_id: str) -> None: @@ -176,10 +197,7 @@ def sweep_stale(self) -> int: swept = 0 try: with self.session() as session: - rows = session.exec( - select(self.model).where(self.model.status.in_(ACTIVE_STATUSES)) - ).all() - for row in rows: + for row in self._active_rows(session): if not is_stale(row): continue self.cancel(row.id) @@ -256,9 +274,7 @@ def recover_pending(self) -> None: """ try: with self.session() as session: - rows = session.exec( - select(self.model).where(self.model.status.in_(ACTIVE_STATUSES)) - ).all() + rows = self._active_rows(session) for row in rows: fields: dict = {"status": "queued"} if hasattr(row, "stage"): diff --git a/backend/app/main.py b/backend/app/main.py index b879522..ff6e948 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -16,7 +16,7 @@ from starlette.exceptions import HTTPException as StarletteHTTPException from uvicorn.middleware.proxy_headers import ProxyHeadersMiddleware -from app.auth import CurrentUser +from app.auth import CurrentUser, is_admin from app.config import settings from app.media_tools import ffmpeg_available, ffmpeg_version from fastapi import Depends @@ -28,6 +28,7 @@ from app.routers import clips as clips_router from app.routers import embeddings as embeddings_router from app.routers import enrichment as enrichment_router +from app.routers import generate as generate_router from app.routers import imports as imports_router from app.routers import media as media_router from app.routers import providers as providers_router @@ -160,7 +161,9 @@ def me( data={ "id": record.id, "username": record.username or user.username, - "is_admin": record.is_admin, + # Resolved, not the raw column: `ADMIN_USERS` is how anybody becomes one, and + # the column alone would tell the frontend nobody ever is. + "is_admin": is_admin(user, record), "first_seen": record.created_at.isoformat() if record.created_at else None, } ) @@ -174,6 +177,9 @@ def me( app.include_router(clips_router.router, prefix="/api/assets", tags=["clips"]) app.include_router(imports_router.router, prefix="/api/assets", tags=["imports"]) app.include_router(activity_router.router, prefix="/api/activity", tags=["activity"]) +# Its own tree rather than under /api/assets: the catalogue is not about any asset, and a +# generation has none until it has run. +app.include_router(generate_router.router, prefix="/api/generate", tags=["generate"]) app.include_router(search_router.router, prefix="/api/search", tags=["search"]) app.include_router(embeddings_router.router, prefix="/api/embeddings", tags=["embeddings"]) app.include_router(settings_router.router, prefix="/api/settings", tags=["settings"]) diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.py index 8d7e3ec..04ed2c2 100644 --- a/backend/app/models/__init__.py +++ b/backend/app/models/__init__.py @@ -12,6 +12,7 @@ from app.models.asset import Asset from app.models.document import DocumentPage from app.models.embedding import Embedding +from app.models.generation import GenerationModel from app.models.job import EnrichmentJob from app.models.provider import AIProvider from app.models.setting import UserSetting @@ -28,6 +29,7 @@ "DocumentPage", "Embedding", "EnrichmentJob", + "GenerationModel", "Suggestion", "Tag", "TagCategory", diff --git a/backend/app/models/asset.py b/backend/app/models/asset.py index 68d5bd6..98a1a40 100644 --- a/backend/app/models/asset.py +++ b/backend/app/models/asset.py @@ -34,9 +34,6 @@ class Asset(SQLModel, table=True): generation metadata. Splitting them into separate tables would mean every list, search, filter and tag query became a union — and "show me everything about the Giordano interview" is the query this product exists to answer. - - M8's fields are not declared yet — that is still one Alembic migration away, and - carrying unused columns for a milestone not yet started is not worth it. """ # The library listing is "my assets, newest first", and it is the hottest query in @@ -128,6 +125,32 @@ class Asset(SQLModel, table=True): # clip's inherited attribution one level down. credit_line: Optional[str] = None + # ─── AI generation (M8) ────────────────────────────────────────────────── + # What made this file, kept so it can be made again. Specified in + # docs/m8-ai-generation.md. All null on anything that was not generated. + # + # The bases are *not* `parent_asset_id`, though that column is right there: a parent + # lends its attribution (M10), puts the child in its Clip tab, and is one column for + # what may be ten images. They are ids inside `ai_source_assets` instead, with no + # foreign key, so deleting a base never blocks or breaks what was made from it — it + # only makes regenerating impossible, which the regenerate endpoint says so. + ai_model: Optional[str] = None # the fal endpoint id + ai_generation_type: Optional[str] = None # text_to_image|image_to_image|image_to_video + # As sent. `description` starts as the same text so search finds a generation by + # what was asked for, but a later Describe run overwrites that; this does not change. + ai_prompt: Optional[str] = None + # JSON: the request body exactly as sent, minus the prompt and the image fields. + # Catalogue defaults included, so a regeneration reproduces this one even after an + # admin has edited the catalogue row. + ai_parameters: Optional[str] = None + # JSON: [{"asset_id", "role": "base"|"end_frame", "name"}]. `name` is a snapshot, so + # a deleted base still reads as something rather than as a bare uuid. + ai_source_assets: Optional[str] = None + # The seed fal reported back, when the model reports one (FLUX does, nano-banana + # does not). + ai_seed: Optional[int] = None + ai_generated_at: Optional[datetime] = None + # ─── provenance of the metadata, not the file ─────────────────────────── # JSON, {"description": "ai"|"human"|"embedded", ...}. FR 8.1.3 requires that a later AI run # never silently overwrites something a person wrote; without recording who last diff --git a/backend/app/models/generation.py b/backend/app/models/generation.py new file mode 100644 index 0000000..20778c1 --- /dev/null +++ b/backend/app/models/generation.py @@ -0,0 +1,87 @@ +"""The catalogue of fal.ai endpoints a user can generate with (M8). + +Global rather than per user, and data rather than code, because fal renames and retires +endpoint ids every few months — `imagen4/preview`, all of `veo3/*` and +`kling-video/v2.1` disappeared between May and October 2026. A list compiled into the +app would be wrong within one release; a table an admin can edit is wrong only until +somebody notices. Specified in docs/m8-ai-generation.md. + +Each row also records its endpoint's *dialect*, because fal's endpoints agree on almost +nothing: the image goes in `image_url`, `image_urls[]` or `start_image_url`; the end +frame in `tail_image_url`, `end_image_url` or nowhere; a duration is `"5"`, `"8s"` or +`6`. Describing that per row is what lets one request builder serve every model rather +than growing a branch per endpoint. + +Ported in spirit from gecko-notes' `ModelCatalogEntry`, minus its two defects: no +uniqueness on the endpoint id (here a unique index), and a "text to image" seed that was +really an edit endpoint and 422'd every call (here `kind` is checked against the image +fields, so that row cannot be saved). +""" + +from __future__ import annotations + +import uuid +from datetime import datetime +from typing import Optional + +from sqlmodel import Field, SQLModel + +from app.clock import utcnow + +KIND_TEXT_TO_IMAGE = "text_to_image" +KIND_IMAGE_TO_IMAGE = "image_to_image" +KIND_IMAGE_TO_VIDEO = "image_to_video" + +GENERATION_KINDS = (KIND_TEXT_TO_IMAGE, KIND_IMAGE_TO_IMAGE, KIND_IMAGE_TO_VIDEO) + +# What a row's `unit_price` is per. The same words fal's pricing API uses. +PRICE_UNITS = ("image", "megapixel", "second", "video") + + +def new_generation_model_id() -> str: + return str(uuid.uuid4()) + + +class GenerationModel(SQLModel, table=True): + id: str = Field(default_factory=new_generation_model_id, primary_key=True) + # fal's id, all segments: `fal-ai/kling-video/v2.5-turbo/pro/image-to-video`. Unique, + # so a duplicate is a 409 rather than two rows the form cannot tell apart. + endpoint_id: str = Field(index=True, unique=True) + kind: str = Field(index=True) + label: str + # Shown beside the model in the form: what it is good at, what it costs. + note: str = Field(default="") + sort_order: int = Field(default=0) + # Hidden from the form rather than deleted, so a model fal has retired can be + # switched off without losing the row an admin may want back. + is_active: bool = Field(default=True) + + # The field the base image(s) go in. Null for text → image. + image_field: Optional[str] = None + # `image_urls: [...]` rather than `image_url: "..."`. + image_field_is_list: bool = Field(default=False) + # How many bases the endpoint takes. Enforced here because several models silently + # truncate instead of refusing — Seedream keeps the *last* ten. + max_images: int = Field(default=0) + # Image → video only, and only for models that take a last frame. + end_image_field: Optional[str] = None + + # JSON: the exact values the endpoint accepts for each user-facing choice — + # {"aspect_ratios", "image_sizes", "durations", "resolutions"} — passed through + # verbatim so `"5"`, `"8s"` and `6` all reach fal as the type it expects; plus + # {"supports_seed", "supports_negative_prompt", "supports_audio", "max_outputs"}. + # The form offers only what a row declares, and the server refuses anything else. + options: str = Field(default="{}") + # JSON: merged into every request body first, so a user's choices and the app's own + # fields both override it. `{"generate_audio": false}` on Veo is the reason it + # exists — its default is on, at half as much again per second. + extra_params: str = Field(default="{}") + + # The list price, for when fal's pricing API cannot be reached with the user's key. + # A cost computed from it is recorded as an estimate. + unit_price: Optional[float] = None + price_unit: Optional[str] = None # one of PRICE_UNITS + price_currency: Optional[str] = None + + created_at: datetime = Field(default_factory=utcnow) + updated_at: datetime = Field(default_factory=utcnow) diff --git a/backend/app/models/job.py b/backend/app/models/job.py index 7b40069..5f72005 100644 --- a/backend/app/models/job.py +++ b/backend/app/models/job.py @@ -60,6 +60,17 @@ # `created_asset_id` there, exactly as an "extract" sub-video's does. Outside # `ENRICHMENT_KINDS`: it calls no paid provider. See docs/url-import.md. KIND_IMPORT_URL = "import_url" +# M8. Making new media with fal.ai — text → image, image → image, image → video. A +# library kind for the same reason as an import: nothing it makes exists until it has +# run, and its bases (there may be ten) are inputs, not the asset the job is about. The +# request, fal's request ids and the assets made all ride in `payload`. Outside +# `ENRICHMENT_KINDS`: it enriches nothing. See docs/m8-ai-generation.md. +KIND_GENERATE = "generate" + +# Kinds that run on the generation queue rather than the enrichment one. A generation +# is minutes of waiting on fal, and on the shared queue at concurrency 1 a video would +# hold every transcription behind it — see `jobs/enrichment.py::generation_queue`. +GENERATION_JOB_KINDS = frozenset({KIND_GENERATE}) # Per-asset actions. Everything in here requires an `asset_id`. ENRICHMENT_KINDS = frozenset( @@ -80,7 +91,13 @@ # change to the dispatch. A bulk run over a selection is here too: it has many assets, # which for the purposes of `asset_id` is the same as having none. LIBRARY_KINDS = frozenset( - {KIND_BACKFILL_EMBEDDINGS, KIND_BULK_ENRICH, KIND_HARVEST_ATTRIBUTION, KIND_IMPORT_URL} + { + KIND_BACKFILL_EMBEDDINGS, + KIND_BULK_ENRICH, + KIND_HARVEST_ATTRIBUTION, + KIND_IMPORT_URL, + KIND_GENERATE, + } ) diff --git a/backend/app/models/usage.py b/backend/app/models/usage.py index a4e80ea..353ecc0 100644 --- a/backend/app/models/usage.py +++ b/backend/app/models/usage.py @@ -27,12 +27,22 @@ from app.clock import utcnow # What was consumed. Kept as GAM's own vocabulary rather than gecko-notes' — it has no -# `tts`, and `image` waits for M8. +# `tts`. KIND_AI = "ai" KIND_STT = "stt" +# M8: what fal.ai made. +KIND_IMAGE = "image" +KIND_VIDEO = "video" UNIT_TOKENS = "tokens" UNIT_SECONDS = "seconds" +# fal bills each endpoint in its own unit. `generated_seconds` is deliberately not +# `seconds`: the usage totals already sum that as audio transcribed, and four seconds of +# generated video counted as four seconds of Deepgram would corrupt both figures. +UNIT_IMAGES = "images" +UNIT_MEGAPIXELS = "megapixels" +UNIT_GENERATED_SECONDS = "generated_seconds" +UNIT_VIDEOS = "videos" def new_usage_id() -> str: @@ -62,8 +72,12 @@ class UsageEvent(SQLModel, table=True): # table does not know. Storing a zero there would read as "this was free". cost: Optional[float] = None currency: Optional[str] = None - # True for a list-price estimate. False would mean a provider-billed exact amount, - # which nothing produces yet — fal.ai's billing headers arrive with M8. + # True for a list-price estimate. False is a provider-billed exact amount, which only + # fal.ai produces: its `x-fal-billable-units` header times its own pricing API's + # unit price (M8). cost_estimated: Optional[bool] = None + # The provider's id for the call, so a figure here can be found in the provider's + # own billing — fal's `x-fal-request-id`. Not indexed: nothing looks it up from here. + external_ref: Optional[str] = None created_at: datetime = Field(default_factory=utcnow, index=True) diff --git a/backend/app/providers/_upstream.py b/backend/app/providers/_upstream.py index 6052673..ad7a344 100644 --- a/backend/app/providers/_upstream.py +++ b/backend/app/providers/_upstream.py @@ -76,15 +76,8 @@ def post_json( an HTTP error status is returned as-is, because only the caller knows how to read its body. `sleep` is injected so tests do not spend the backoff. """ - client_timeout = httpx.Timeout( - timeout, - connect=min(timeout, 10.0), - write=min(timeout, 30.0), - pool=min(timeout, 10.0), - ) - try: - with httpx.Client(timeout=client_timeout) as client: + with httpx.Client(timeout=_split_timeout(timeout)) as client: for attempt in range(MAX_ATTEMPTS): response = client.post(url, headers=dict(headers or {}), json=dict(json_body)) if response.status_code not in RETRY_STATUS_CODES: @@ -111,6 +104,42 @@ def post_json( raise ProviderError(f"Could not reach {label} ({type(exc).__name__})") from exc +def request( + method: str, + url: str, + *, + headers: Optional[Mapping[str, str]] = None, + params: Optional[Mapping[str, str]] = None, + timeout: float, + label: str, +) -> httpx.Response: + """One call with headers and no retry, for anything that is not a POST of JSON. + + fal's queue (M8) is the reason: its status, result and cancel calls are a GET and a + PUT that carry the key in a header, which `get_json` cannot send. Not retried — + a status poll is retried by being polled again, and a cancel is best-effort — but + connection failures read the same way `post_json`'s do. + """ + try: + with httpx.Client(timeout=_split_timeout(timeout)) as client: + return client.request( + method, url, headers=dict(headers or {}), params=dict(params or {}) + ) + except httpx.TimeoutException as exc: + raise ProviderError(f"{label} did not respond in time") from exc + except httpx.RequestError as exc: + raise ProviderError(f"Could not reach {label} ({type(exc).__name__})") from exc + + +def _split_timeout(timeout: float) -> httpx.Timeout: + return httpx.Timeout( + timeout, + connect=min(timeout, 10.0), + write=min(timeout, 30.0), + pool=min(timeout, 10.0), + ) + + def get_json(url: str, *, timeout: float, label: str) -> httpx.Response: """The read-only counterpart, for probing whether a local daemon is up.""" try: diff --git a/backend/app/routers/generate.py b/backend/app/routers/generate.py new file mode 100644 index 0000000..6b4b665 --- /dev/null +++ b/backend/app/routers/generate.py @@ -0,0 +1,469 @@ +"""Generating media with fal.ai (M8): the model catalogue, and starting a generation. + +Everyone reads the catalogue; only admins (`ADMIN_USERS`) change it. A generation is +checked here, in the order docs/m8-ai-generation.md fixes, and then queued — nothing in +a request touches fal. Whether fal accepts it is the job's first network call, and its +answer arrives the way every job's does: in the activity feed. + +Every check that can be made without fal is made here, because a refusal now costs a +red line under the form and the same refusal from fal costs a queued job, a wait, and — +for some endpoints — a bill. +""" + +from __future__ import annotations + +import json +from typing import Any, Optional + +from fastapi import APIRouter, Depends, HTTPException, Query, Response, status +from PIL import Image, UnidentifiedImageError +from sqlalchemy.exc import IntegrityError +from sqlmodel import Session, col, select + +from app.auth import AdminUser, CurrentUser, is_admin +from app.clock import utcnow +from app.database import get_session +from app.generation import catalogue +from app.generation.catalogue import InvalidEntry, extra_params_of, options_of +from app.generation.request import ( + ROLE_BASE, + ROLE_END_FRAME, + GenerationRequest, + InvalidGeneration, + build_parameters, + check_prompt, + check_shape, + name_from_prompt, + without_app_fields, +) +from app.ingest.filetypes import PILLOW_READABLE, TYPE_IMAGE, extension_of +from app.jobs import enrichment as enrichment_jobs +from app.jobs.registry import KINDS +from app.models.asset import Asset +from app.models.generation import GenerationModel, new_generation_model_id +from app.models.job import KIND_GENERATE +from app.models.user import User +from app.routers.assets import get_storage +from app.schemas import DataResponse, ListResponse +from app.schemas_generate import ( + GenerationCreate, + GenerationModelCreate, + GenerationModelRead, + GenerationModelUpdate, + RegenerateRequest, +) +from app.schemas_jobs import ActivityJobRead +from app.services.assets import generation_of +from app.settings_store import FAL_API_KEY, has_setting +from app.storage import LocalStorage, StorageError + +router = APIRouter() + +# Catalogue columns a PATCH may set to null, and the ones whose null means "back to the +# default". Anything else sent as null is refused rather than guessed at. +_NULLABLE = frozenset( + {"image_field", "end_image_field", "unit_price", "price_unit", "price_currency"} +) +_RESET_ON_NULL = {"note": "", "options": {}, "extra_params": {}} + + +def _error(status_code: int, code: str, message: str) -> HTTPException: + return HTTPException(status_code=status_code, detail={"code": code, "message": message}) + + +# ─── the catalogue ─────────────────────────────────────────────────────────── + + +def _read(row: GenerationModel) -> GenerationModelRead: + return GenerationModelRead( + id=row.id, + endpoint_id=row.endpoint_id, + kind=row.kind, + label=row.label, + note=row.note or "", + sort_order=row.sort_order, + is_active=row.is_active, + image_field=row.image_field, + image_field_is_list=row.image_field_is_list, + max_images=row.max_images, + end_image_field=row.end_image_field, + options=options_of(row), + extra_params=extra_params_of(row), + unit_price=row.unit_price, + price_unit=row.price_unit, + price_currency=row.price_currency, + created_at=row.created_at, + updated_at=row.updated_at, + ) + + +def _entry_of(row: GenerationModel) -> dict[str, Any]: + """A stored row as the dict `catalogue.validate` checks, for merging a PATCH into.""" + return { + "endpoint_id": row.endpoint_id, + "kind": row.kind, + "label": row.label, + "note": row.note, + "sort_order": row.sort_order, + "is_active": row.is_active, + "image_field": row.image_field, + "image_field_is_list": row.image_field_is_list, + "max_images": row.max_images, + "end_image_field": row.end_image_field, + "options": options_of(row), + "extra_params": extra_params_of(row), + "unit_price": row.unit_price, + "price_unit": row.price_unit, + "price_currency": row.price_currency, + } + + +def _validated(entry: dict[str, Any]) -> dict[str, Any]: + # Trimmed before checking, so a pasted id with a trailing space is not refused for + # a character nobody can see. + for name in ("endpoint_id", "label", "image_field", "end_image_field", "price_unit"): + if isinstance(entry.get(name), str): + entry[name] = entry[name].strip() or None + try: + return catalogue.validate(entry) + except InvalidEntry as exc: + raise _error(422, "invalid_model_entry", str(exc)) from exc + + +def _apply(row: GenerationModel, entry: dict[str, Any]) -> None: + for name, value in entry.items(): + if name in ("options", "extra_params"): + value = json.dumps(value, sort_keys=True) + elif name == "note": + value = value or "" + setattr(row, name, value) + + +def _require_unique(session: Session, endpoint_id: str, *, except_id: Optional[str] = None) -> None: + existing = session.exec( + select(GenerationModel).where(GenerationModel.endpoint_id == endpoint_id) + ).first() + if existing is not None and existing.id != except_id: + raise _endpoint_exists(endpoint_id) + + +def _endpoint_exists(endpoint_id: str) -> HTTPException: + return _error(409, "endpoint_exists", f"{endpoint_id} is already in the catalogue") + + +def _commit(session: Session, row: GenerationModel) -> None: + """Commit, turning the unique index's refusal into the 409 the pre-check gives. + + The pre-check is what normally answers; this is for two admins saving the same id at + once, where only the database can say who was first. + """ + session.add(row) + try: + session.commit() + except IntegrityError as exc: + session.rollback() + raise _endpoint_exists(row.endpoint_id) from exc + session.refresh(row) + + +def _row(session: Session, model_id: str) -> GenerationModel: + row = session.get(GenerationModel, model_id) + if row is None: + raise _error(404, "model_not_found", "No such model in the catalogue") + return row + + +@router.get("/models", response_model=ListResponse[GenerationModelRead]) +def list_models( + user: CurrentUser, + include_inactive: bool = Query(default=False), + session: Session = Depends(get_session), +) -> ListResponse[GenerationModelRead]: + """The catalogue, ordered as the form lists it. + + `include_inactive` is for the admin editor; anyone else asking for it gets the active + rows, since a switched-off model is not one they can use. + """ + query = select(GenerationModel) + if not (include_inactive and is_admin(user, session.get(User, user.id))): + query = query.where(col(GenerationModel.is_active).is_(True)) + rows = session.exec( + query.order_by( + col(GenerationModel.kind), col(GenerationModel.sort_order), col(GenerationModel.label) + ) + ).all() + return ListResponse(data=[_read(r) for r in rows], total=len(rows), limit=len(rows), offset=0) + + +@router.post("/models", response_model=DataResponse[GenerationModelRead], status_code=201) +def create_model( + payload: GenerationModelCreate, + user: AdminUser, + session: Session = Depends(get_session), +) -> DataResponse[GenerationModelRead]: + entry = _validated(payload.model_dump()) + _require_unique(session, entry["endpoint_id"]) + + row = GenerationModel( + id=new_generation_model_id(), + endpoint_id=entry["endpoint_id"], + kind=entry["kind"], + label=entry["label"], + ) + _apply(row, entry) + _commit(session, row) + return DataResponse(data=_read(row)) + + +@router.patch("/models/{model_id}", response_model=DataResponse[GenerationModelRead]) +def update_model( + model_id: str, + payload: GenerationModelUpdate, + user: AdminUser, + session: Session = Depends(get_session), +) -> DataResponse[GenerationModelRead]: + """Change exactly the fields the body names; an explicit null clears. + + Validated as the whole row it would leave behind, not field by field: changing + `kind` to text → image while the row still has an `image_field` must be refused even + though neither field alone is wrong. + """ + row = _row(session, model_id) + entry = _entry_of(row) + for name in payload.model_fields_set: + value = getattr(payload, name) + if value is None: + if name in _NULLABLE: + entry[name] = None + elif name in _RESET_ON_NULL: + entry[name] = _RESET_ON_NULL[name] + else: + raise _error(422, "invalid_model_entry", f"{name} cannot be cleared") + else: + entry[name] = value + + entry = _validated(entry) + if entry["endpoint_id"] != row.endpoint_id: + _require_unique(session, entry["endpoint_id"], except_id=row.id) + + _apply(row, entry) + row.updated_at = utcnow() + _commit(session, row) + return DataResponse(data=_read(row)) + + +@router.delete( + "/models/{model_id}", + status_code=status.HTTP_204_NO_CONTENT, + response_model=None, + response_class=Response, +) +def delete_model( + model_id: str, + user: AdminUser, + session: Session = Depends(get_session), +) -> None: + """Remove a row. Assets made with it keep their `ai_model` — it is a string, not a + reference — and only lose the ability to be regenerated, which says so.""" + row = _row(session, model_id) + session.delete(row) + session.commit() + + +# ─── generating ────────────────────────────────────────────────────────────── + + +def _require_fal_key(session: Session, user_id: str) -> None: + if not has_setting(session, user_id, FAL_API_KEY): + raise _error(400, "fal_key_missing", "Add a fal.ai API key in Settings to generate") + + +def _owned_asset(session: Session, user_id: str, asset_id: str) -> Asset: + asset = session.get(Asset, asset_id) + if asset is None or asset.user_id != user_id: + raise _error(404, "asset_not_found", "No such asset") + return asset + + +def _require_usable_base(storage: LocalStorage, asset: Asset) -> None: + """An image that owns a file Pillow can open — which is what the job will do. + + Opening reads only the header, so this is cheap; it catches a truncated upload or a + file that was never really an image before a job is queued to fail on it. + """ + if asset.asset_type != TYPE_IMAGE: + raise _error(422, "invalid_base", f"“{asset.name}” is not an image") + if not asset.storage_key: + raise _error(422, "invalid_base", f"“{asset.name}” is a clip and has no file of its own") + extension = extension_of(asset.storage_key) + if extension not in PILLOW_READABLE: + raise _error( + 422, + "invalid_base", + f"“{asset.name}” is a {extension.lstrip('.').upper() or 'nameless'} image, " + "which cannot be used as a base", + ) + try: + with storage.materialise(asset.storage_key) as path, Image.open(path): + pass + except (StorageError, OSError, UnidentifiedImageError, Image.DecompressionBombError) as exc: + raise _error(422, "invalid_base", f"“{asset.name}” could not be read as an image") from exc + + +def _checked_sources( + session: Session, + storage: LocalStorage, + user_id: str, + base_ids: list[str], + end_frame_id: Optional[str], +) -> list[dict]: + """Every base owned (404), then every base usable (422) — ownership first throughout, + so a request is never told something about an asset it could not see.""" + roles = [(asset_id, ROLE_BASE) for asset_id in base_ids] + if end_frame_id: + roles.append((end_frame_id, ROLE_END_FRAME)) + + owned = [(_owned_asset(session, user_id, asset_id), role) for asset_id, role in roles] + for asset, _role in owned: + _require_usable_base(storage, asset) + return [{"asset_id": asset.id, "role": role, "name": asset.name} for asset, role in owned] + + +def _queue(session: Session, user_id: str, request: GenerationRequest) -> ActivityJobRead: + job = enrichment_jobs.submit_library( + session, + user_id, + KIND_GENERATE, + model=request.endpoint_id, + payload=json.dumps({"request": request.to_dict()}, sort_keys=True), + asset_name=name_from_prompt(request.prompt), + ) + return KINDS["enrichment"].to_activity(job) + + +@router.post("", response_model=DataResponse[ActivityJobRead], status_code=status.HTTP_202_ACCEPTED) +def start_generation( + payload: GenerationCreate, + user: CurrentUser, + session: Session = Depends(get_session), + storage: LocalStorage = Depends(get_storage), +) -> DataResponse[ActivityJobRead]: + """Queue a generation. Returns the job, which the activity feed then follows.""" + _require_fal_key(session, user.id) + + row = session.get(GenerationModel, payload.model_id) if payload.model_id else None + if row is None: + raise _error(404, "model_not_found", "No such model in the catalogue") + if not row.is_active: + raise _error( + 409, "model_inactive", f"{row.label} has been switched off by an administrator" + ) + + try: + prompt = check_prompt(payload.prompt) + check_shape(row, len(payload.base_asset_ids), payload.end_frame_asset_id is not None) + parameters = build_parameters(row, options_of(row), payload.params.model_dump()) + except InvalidGeneration as exc: + raise _error(422, "invalid_generation", str(exc)) from exc + + sources = _checked_sources( + session, storage, user.id, payload.base_asset_ids, payload.end_frame_asset_id + ) + + request = GenerationRequest( + model_id=row.id, + endpoint_id=row.endpoint_id, + kind=row.kind, + prompt=prompt, + parameters=parameters, + image_field=row.image_field, + image_field_is_list=row.image_field_is_list, + end_image_field=row.end_image_field, + sources=sources, + ) + return DataResponse(data=_queue(session, user.id, request)) + + +@router.post( + "/{asset_id}/regenerate", + response_model=DataResponse[ActivityJobRead], + status_code=status.HTTP_202_ACCEPTED, +) +def regenerate( + asset_id: str, + user: CurrentUser, + payload: Optional[RegenerateRequest] = None, + session: Session = Depends(get_session), + storage: LocalStorage = Depends(get_storage), +) -> DataResponse[ActivityJobRead]: + """Make a generated asset again from what it recorded about itself. + + The stored parameters are resent as they are — catalogue defaults included, which is + why they were stored — through the endpoint's *current* row for its field names. The + seed is the exception: plain regenerate drops it so fal picks a new one, since a + regeneration that reproduced the same picture would not be one; `reuse_seed` sends + the one fal reported back. + """ + options = payload or RegenerateRequest() + asset = _owned_asset(session, user.id, asset_id) + generation = generation_of(asset) + if generation is None or not generation.kind: + raise _error( + 409, "not_generated", "This asset was not generated, so there is nothing to regenerate" + ) + + _require_fal_key(session, user.id) + + row = session.exec( + select(GenerationModel).where( + GenerationModel.endpoint_id == generation.model, + col(GenerationModel.is_active).is_(True), + ) + ).first() + if row is None or row.kind != generation.kind: + raise _error( + 409, + "model_unavailable", + f"{generation.model} is no longer an active model in the catalogue", + ) + + for source in generation.sources: + found = session.get(Asset, source.asset_id) + if found is None or found.user_id != user.id: + raise _error( + 409, + "source_asset_missing", + f"The base image “{source.name or source.asset_id}” has been deleted, " + "so this cannot be made again", + ) + + base_ids = [s.asset_id for s in generation.sources if s.role == ROLE_BASE] + end_ids = [s.asset_id for s in generation.sources if s.role == ROLE_END_FRAME] + try: + prompt = check_prompt(options.prompt if options.prompt is not None else generation.prompt) + check_shape(row, len(base_ids), bool(end_ids)) + except InvalidGeneration as exc: + raise _error(422, "invalid_generation", str(exc)) from exc + + sources = _checked_sources(session, storage, user.id, base_ids, end_ids[0] if end_ids else None) + + parameters = dict(generation.parameters) + stored_seed = parameters.pop("seed", None) + if options.reuse_seed and options_of(row)["supports_seed"]: + seed = generation.seed if generation.seed is not None else stored_seed + if seed is not None: + parameters["seed"] = seed + + request = GenerationRequest( + model_id=row.id, + endpoint_id=row.endpoint_id, + kind=row.kind, + prompt=prompt, + parameters=without_app_fields(parameters, row.image_field, row.end_image_field), + image_field=row.image_field, + image_field_is_list=row.image_field_is_list, + end_image_field=row.end_image_field, + sources=sources, + regenerated_from=asset.id, + ) + return DataResponse(data=_queue(session, user.id, request)) diff --git a/backend/app/routers/settings.py b/backend/app/routers/settings.py index 1e0e729..c0fa4f9 100644 --- a/backend/app/routers/settings.py +++ b/backend/app/routers/settings.py @@ -20,6 +20,7 @@ from app.enrichment.deepgram import DEFAULT_MODEL from app.safe_url import require_safe_external_url from app.schemas import DataResponse +from app.schemas_generate import GenerationSettings, GenerationSettingsUpdate from app.settings_store import ( DEEPGRAM_API_KEY, DEEPGRAM_MODEL, @@ -27,6 +28,7 @@ EMBEDDING_DIMENSIONS, EMBEDDING_MODEL, EMBEDDING_PROVIDER, + FAL_API_KEY, OLLAMA_BASE_URL, OPENAI_API_KEY, clear_setting, @@ -88,6 +90,35 @@ def write_speech_settings( return read_speech_settings(user, session) +# ─── generation (M8) ───────────────────────────────────────────────────────── +# +# Just the key. The model catalogue is global and lives at /api/generate/models; what +# each user brings is the account fal bills. + + +@router.get("/generation", response_model=DataResponse[GenerationSettings]) +def read_generation_settings( + user: CurrentUser, session: Session = Depends(get_session) +) -> DataResponse[GenerationSettings]: + return DataResponse( + data=GenerationSettings(fal_key_configured=has_setting(session, user.id, FAL_API_KEY)) + ) + + +@router.put("/generation", response_model=DataResponse[GenerationSettings]) +def write_generation_settings( + payload: GenerationSettingsUpdate, + user: CurrentUser, + session: Session = Depends(get_session), +) -> DataResponse[GenerationSettings]: + if payload.fal_api_key is not None: + if payload.fal_api_key.strip(): + set_setting(session, user.id, FAL_API_KEY, payload.fal_api_key.strip()) + else: + clear_setting(session, user.id, FAL_API_KEY) + return read_generation_settings(user, session) + + # ─── embeddings ────────────────────────────────────────────────────────────── # # Semantic search is off until one of these is configured, and until this endpoint diff --git a/backend/app/schemas_assets.py b/backend/app/schemas_assets.py index 999097c..48d7d55 100644 --- a/backend/app/schemas_assets.py +++ b/backend/app/schemas_assets.py @@ -6,7 +6,7 @@ from datetime import datetime from typing import List, Optional -from pydantic import BaseModel, Field, field_validator +from pydantic import BaseModel, ConfigDict, Field, field_validator # A whole year, a year and month, or a full date. Months 01-12 and days 01-31 are # enforced here so the string column cannot hold "2019-13" and sort between "2019-12" @@ -75,11 +75,38 @@ class AssetRead(BaseModel): # their own tags is sixty queries that grow with the page. tags: List["AssetTagRead"] = Field(default_factory=list) + # M8. How this asset was made, when fal.ai made it; null for everything else. + generation: Optional["AssetGenerationRead"] = None + upload_date: datetime modified_date: datetime metadata_modified_date: datetime +class AssetGenerationSource(BaseModel): + """One image a generation was made from.""" + + asset_id: str + role: str # "base" | "end_frame" + # A snapshot from when it was generated, so a base deleted since still reads as + # something rather than as a bare id. + name: str = "" + + +class AssetGenerationRead(BaseModel): + """The `ai_*` columns, decoded. Enough to show the provenance and to regenerate.""" + + model_config = ConfigDict(protected_namespaces=()) + + model: str # the fal endpoint id + kind: str # text_to_image | image_to_image | image_to_video + prompt: str + parameters: dict = Field(default_factory=dict) + sources: List[AssetGenerationSource] = Field(default_factory=list) + seed: Optional[int] = None + generated_at: Optional[datetime] = None + + class AssetTagRead(BaseModel): """A tag as it appears on an asset. diff --git a/backend/app/schemas_generate.py b/backend/app/schemas_generate.py new file mode 100644 index 0000000..9ecd53e --- /dev/null +++ b/backend/app/schemas_generate.py @@ -0,0 +1,124 @@ +"""Wire shapes for generation (M8): the model catalogue and a generation request. + +The contract is docs/m8-ai-generation.md's API section, which the frontend is built +against field for field. + +Most fields on the write shapes are typed loosely on purpose. A missing label or an +`options` that is a string should come back as `invalid_model_entry` with a sentence +naming the field — the house error shape — rather than as FastAPI's validation list, +which the frontend's one error handler cannot read. The real checks are in +`app/generation/catalogue.py` and `app/generation/request.py`. +""" + +from __future__ import annotations + +from datetime import datetime +from typing import Any, Optional + +from pydantic import BaseModel, ConfigDict, Field + + +class GenerationModelRead(BaseModel): + id: str + endpoint_id: str + kind: str + label: str + note: str = "" + sort_order: int = 0 + is_active: bool = True + image_field: Optional[str] = None + image_field_is_list: bool = False + max_images: int = 0 + end_image_field: Optional[str] = None + # Decoded from the JSON-as-TEXT columns, every option key present. + options: dict[str, Any] = Field(default_factory=dict) + extra_params: dict[str, Any] = Field(default_factory=dict) + unit_price: Optional[float] = None + price_unit: Optional[str] = None + price_currency: Optional[str] = None + created_at: datetime + updated_at: datetime + + +class GenerationModelCreate(BaseModel): + endpoint_id: Optional[str] = None + kind: Optional[str] = None + label: Optional[str] = None + note: str = "" + sort_order: int = 0 + is_active: bool = True + image_field: Optional[str] = None + image_field_is_list: bool = False + max_images: int = 0 + end_image_field: Optional[str] = None + options: Any = Field(default_factory=dict) + extra_params: Any = Field(default_factory=dict) + unit_price: Optional[float] = None + price_unit: Optional[str] = None + price_currency: Optional[str] = None + + +class GenerationModelUpdate(BaseModel): + """Every field optional, and *present* distinguished from *absent*. + + The router reads `model_fields_set`, so `{"note": null}` clears the note while `{}` + leaves it alone — the distinction gecko-notes' catalogue PUT lost by skipping every + None, which is why nothing there could ever be cleared. + """ + + endpoint_id: Optional[str] = None + kind: Optional[str] = None + label: Optional[str] = None + note: Optional[str] = None + sort_order: Optional[int] = None + is_active: Optional[bool] = None + image_field: Optional[str] = None + image_field_is_list: Optional[bool] = None + max_images: Optional[int] = None + end_image_field: Optional[str] = None + options: Any = None + extra_params: Any = None + unit_price: Optional[float] = None + price_unit: Optional[str] = None + price_currency: Optional[str] = None + + +class GenerationParams(BaseModel): + # Any, not str: a choice is matched against the row's declared values with its + # type, and Veo's "8s", Kling's "5" and Wan's 6 are all real durations. + aspect_ratio: Any = None + image_size: Any = None + duration: Any = None + resolution: Any = None + seed: Optional[int] = None + negative_prompt: Optional[str] = None + generate_audio: Optional[bool] = None + num_outputs: int = 1 + + +class GenerationCreate(BaseModel): + model_config = ConfigDict(protected_namespaces=()) + + model_id: str = "" + prompt: str = "" + base_asset_ids: list[str] = Field(default_factory=list) + end_frame_asset_id: Optional[str] = None + params: GenerationParams = Field(default_factory=GenerationParams) + + +class RegenerateRequest(BaseModel): + # None keeps the stored prompt. + prompt: Optional[str] = None + # Resend the seed fal reported, so the same model makes the same picture again + # with only what was changed changed. Off, the seed is dropped and fal picks one. + reuse_seed: bool = False + + +class GenerationSettings(BaseModel): + fal_key_configured: bool + + +class GenerationSettingsUpdate(BaseModel): + # None leaves the stored key alone; "" removes it — the contract every credential + # in this app follows, since without it a key could never be cleared. + fal_api_key: Optional[str] = None diff --git a/backend/app/schemas_jobs.py b/backend/app/schemas_jobs.py index 007dea5..40f5e63 100644 --- a/backend/app/schemas_jobs.py +++ b/backend/app/schemas_jobs.py @@ -3,9 +3,9 @@ from __future__ import annotations from datetime import datetime -from typing import Optional +from typing import List, Optional -from pydantic import BaseModel +from pydantic import BaseModel, Field class ActivityJobRead(BaseModel): @@ -37,6 +37,10 @@ class ActivityJobRead(BaseModel): # job's life, so this is the one place the frontend can learn the result's id — # there is nowhere else to put it in a deliberately flat, kind-agnostic shape. result_asset_id: Optional[str] = None + # M8. Every asset the job created, in output order — a generation can make up to + # four. `result_asset_id` stays, as the first of these, so nothing that already + # reads it has to change. + result_asset_ids: List[str] = Field(default_factory=list) error_message: Optional[str] = None created_at: datetime diff --git a/backend/app/services/assets.py b/backend/app/services/assets.py index 9742037..6d8c273 100644 --- a/backend/app/services/assets.py +++ b/backend/app/services/assets.py @@ -23,6 +23,7 @@ from app.config import settings from app.ingest import embedded_metadata, thumbnails from app.ingest.filetypes import ( + SOURCE_AI, SOURCE_CLIP, SOURCE_SUBVIDEO, SOURCE_UPLOAD, @@ -44,7 +45,12 @@ from app.models.suggestion import Suggestion from app.models.document import DocumentPage from app.models.transcript import TranscriptSegment -from app.schemas_assets import AssetRead, AssetTagRead +from app.schemas_assets import ( + AssetGenerationRead, + AssetGenerationSource, + AssetRead, + AssetTagRead, +) from app.search import fts, vectors from app.services import tags from app.storage import LocalStorage, StorageError, StoredFile, new_key, thumb_key_for @@ -284,6 +290,12 @@ def _chain_transcription(session: Session, asset: Asset) -> None: if not can_transcribe(asset): return + # Not a generated video (M8). Most of what fal makes is silent or music, so this + # would bill Deepgram for every one to transcribe nothing. The Transcribe button + # still works on one by hand. + if asset.source == SOURCE_AI: + return + try: if not load_deepgram_key(session, asset.user_id): logger.info( @@ -886,12 +898,52 @@ def to_read_model( AssetTagRead(id=t.id, name=t.name, category_id=t.category_id) for t in (asset_tags or []) ], + generation=generation_of(asset), upload_date=asset.upload_date, modified_date=asset.modified_date, metadata_modified_date=asset.metadata_modified_date, ) +def generation_of(asset: Asset) -> Optional[AssetGenerationRead]: + """The `ai_*` columns as the API returns them, or None for anything not generated. + + Decoded defensively, the way every JSON-as-TEXT column here is read: a value that + will not parse reads as empty rather than making the asset unreadable. + """ + if not asset.ai_model: + return None + + sources = [] + for entry in _json_or(asset.ai_source_assets, []): + if isinstance(entry, dict) and isinstance(entry.get("asset_id"), str): + sources.append( + AssetGenerationSource( + asset_id=entry["asset_id"], + role=str(entry.get("role") or "base"), + name=str(entry.get("name") or ""), + ) + ) + + return AssetGenerationRead( + model=asset.ai_model, + kind=asset.ai_generation_type or "", + prompt=asset.ai_prompt or "", + parameters=_json_or(asset.ai_parameters, {}), + sources=sources, + seed=asset.ai_seed, + generated_at=asset.ai_generated_at, + ) + + +def _json_or(text: Optional[str], default: Any) -> Any: + try: + value = json.loads(text) if text else default + except ValueError: + return default + return value if isinstance(value, type(default)) else default + + def _signed_url(key: Optional[str]) -> Optional[str]: if not key: return None diff --git a/backend/app/settings_store.py b/backend/app/settings_store.py index 42e1e6e..f0956b6 100644 --- a/backend/app/settings_store.py +++ b/backend/app/settings_store.py @@ -34,8 +34,13 @@ EMBEDDING_BASE_URL = "embedding_base_url" OLLAMA_BASE_URL = "ollama_base_url" +# M8. One key, used for generation and for fal's pricing API alike. gecko-notes keeps a +# second "admin" fal key for billing scope; GAM reads only list prices, which an ordinary +# key is documented to reach, and a second credential nobody needs is one more to leak. +FAL_API_KEY = "fal_api_key" + # Keys whose value is encrypted at rest. Anything not in here is stored as plain JSON. -SECRET_KEYS = frozenset({DEEPGRAM_API_KEY, OPENAI_API_KEY}) +SECRET_KEYS = frozenset({DEEPGRAM_API_KEY, OPENAI_API_KEY, FAL_API_KEY}) def get_setting(session: Session, user_id: str, key: str, default: Any = None) -> Any: @@ -106,3 +111,7 @@ def load_deepgram_key(session: Session, user_id: str) -> Optional[str]: def load_openai_key(session: Session, user_id: str) -> Optional[str]: return get_setting(session, user_id, OPENAI_API_KEY) or None + + +def load_fal_key(session: Session, user_id: str) -> Optional[str]: + return get_setting(session, user_id, FAL_API_KEY) or None diff --git a/backend/app/usage/events.py b/backend/app/usage/events.py index c16bb42..d2403a2 100644 --- a/backend/app/usage/events.py +++ b/backend/app/usage/events.py @@ -31,6 +31,7 @@ def record( cost: Optional[float] = None, currency: Optional[str] = None, cost_estimated: Optional[bool] = None, + external_ref: Optional[str] = None, ) -> None: """Write one usage row. Never raises.""" try: @@ -46,6 +47,7 @@ def record( cost=cost, currency=currency, cost_estimated=cost_estimated, + external_ref=external_ref, ) ) session.commit() @@ -94,8 +96,8 @@ def _record_completion(session: Session, asset_id: Optional[str], user_id: str, model=completion.model, cost=estimate[0] if estimate else None, currency=estimate[1] if estimate else None, - # Only ever True here. An exact, provider-billed figure needs fal.ai's response - # headers, which arrive with M8. + # Only ever True here: a token count times a list price. The one exact figure in + # the app is fal.ai's, from its billing header (M8). cost_estimated=True if estimate else None, ) @@ -119,9 +121,11 @@ def totals_for(session: Session, user_id: str, asset_id: Optional[str] = None) - "priced_events": len(priced), "cost": round(sum(r.cost or 0.0 for r in priced), 6) if priced else 0.0, "currency": next((r.currency for r in priced if r.currency), "USD"), - # True when every figure in the total came from the list-price table rather than - # a bill. Everything does today; the flag exists so that stops being assumed. - "estimated": all(r.cost_estimated for r in priced) if priced else True, + # False only when every figure in the total is a bill. Since M8 some are — fal's — + # and a total mixing a bill with list-price estimates is still an estimate, so one + # estimated row is enough to say so. (`all()` over the flags read the other way + # round, and called a mixed total exact.) + "estimated": any(r.cost_estimated is not False for r in priced) if priced else True, "tokens": sum(r.units for r in rows if r.unit_type == UNIT_TOKENS), "seconds": sum(r.units for r in rows if r.unit_type == "seconds"), } diff --git a/backend/tests/fal_stub.py b/backend/tests/fal_stub.py new file mode 100644 index 0000000..f117c73 --- /dev/null +++ b/backend/tests/fal_stub.py @@ -0,0 +1,290 @@ +"""A stand-in for fal.ai, plugged in at the httpx seam. + +Every request GAM makes for a generation — submit, status, result, cancel, the pricing +API, and the download of each output — goes through an `httpx.Client`. The `fal` +fixture swaps that class for one wired to an `httpx.MockTransport` whose handler is a +`FakeFal`, so the real client code runs unchanged — headers, status codes, streaming, +redirects — and only the far end is pretend. The same seam `test_provider_upstream.py` +uses, with real httpx semantics instead of a hand-written client. + +DNS is faked too, so the SSRF guard resolves names without touching the network, and +the job's clock and sleep are replaced so a thirty-minute deadline takes milliseconds. +""" + +from __future__ import annotations + +import json +import socket +from datetime import timedelta +from pathlib import Path +from typing import Any, Callable, Optional + +import httpx +import pytest + +from app.clock import utcnow +from app.generation import pricing +from app.generation import run as run_module + +FIXTURES = Path(__file__).parent / "fixtures" + +QUEUE = "https://queue.fal.run" +CDN = "https://v3.fal.media/files" +PUBLIC_ADDRESS = "93.184.216.34" + + +class ProcessDied(BaseException): + """Raised from inside a request to stop a job dead, the way a restart would. + + A BaseException, so neither httpx, the job, nor the worker's catch-all catches it: + the row is left exactly as a killed process leaves it. + """ + + +class FakeClock: + def __init__(self) -> None: + self.current = utcnow() + self.slept: list[float] = [] + # Called on each sleep, for a test that acts while the job waits. + self.on_sleep: Optional[Callable[[], None]] = None + + def now(self): + return self.current + + def sleep(self, seconds: float) -> None: + self.slept.append(seconds) + self.current += timedelta(seconds=seconds) + if self.on_sleep is not None: + self.on_sleep() + + +class FakeFal: + def __init__(self) -> None: + self.calls: list[httpx.Request] = [] + self.request_id = "req-1" + # Replaces the whole submit answer when set. + self.submit_response: Optional[httpx.Response] = None + # Answered in order; the last repeats. A dict is a 200 body; an httpx.Response + # is sent as it is; a BaseException class is raised. + self.statuses: list[Any] = [{"status": "COMPLETED"}] + self.result_response: Optional[httpx.Response] = None + self.result_body: dict = { + "images": [{"url": f"{CDN}/out-0.png", "content_type": "image/png"}], + "seed": 1234, + } + self.result_headers: dict = { + "x-fal-billable-units": "1", + "x-fal-request-id": "fal-req-1", + } + # fal's pricing API: a price for whatever endpoint is asked about, or a status. + self.price: Optional[tuple[float, str]] = (0.025, "megapixel") + self.pricing_status = 200 + self.pricing_raises: Optional[type] = None + # url (without query) -> file to serve, or a ready response. + self.files: dict[str, Any] = {f"{CDN}/out-0.png": FIXTURES / "sample_image.png"} + # Host -> address, for the SSRF guard. Anything else is public. + self.addresses: dict[str, str] = {} + self.clock = FakeClock() + + # ─── the transport ─────────────────────────────────────────────────────── + + def __call__(self, request: httpx.Request) -> httpx.Response: + self.calls.append(request) + host, path, method = request.url.host, request.url.path, request.method + + if host == "queue.fal.run": + if method == "POST": + return self._submit(request) + if method == "PUT" and path.endswith("/cancel"): + return httpx.Response(202, json={"status": "CANCELLATION_REQUESTED"}) + if path.endswith("/status"): + return self._status() + return self.result_response or httpx.Response( + 200, json=self.result_body, headers=self.result_headers + ) + + if host == "api.fal.ai" and path == "/v1/models/pricing": + if self.pricing_raises is not None: + raise self.pricing_raises() + if self.pricing_status != 200 or self.price is None: + return httpx.Response(self.pricing_status if self.pricing_status != 200 else 404, json={ + "error": {"type": "not_found", "message": "No price"} + }) + endpoint_id = request.url.params.get("endpoint_id") + unit_price, unit = self.price + return httpx.Response(200, json={ + "prices": [{"endpoint_id": endpoint_id, "unit_price": unit_price, "unit": unit, "currency": "USD"}], + "next_cursor": None, + "has_more": False, + }) + + target = str(request.url.copy_with(query=None)) + served = self.files.get(target) + if isinstance(served, httpx.Response): + return served + if isinstance(served, Path): + return httpx.Response(200, content=served.read_bytes(), headers={"content-type": _media_type(served)}) + return httpx.Response(404, json={"detail": "Not Found"}) + + def _submit(self, request: httpx.Request) -> httpx.Response: + if self.submit_response is not None: + return self.submit_response + app = "/".join(request.url.path.strip("/").split("/")[:2]) + base = f"{QUEUE}/{app}/requests/{self.request_id}" + return httpx.Response(200, json={ + "status": "IN_QUEUE", + "request_id": self.request_id, + "queue_position": 0, + "response_url": base, + "status_url": f"{base}/status", + "cancel_url": f"{base}/cancel", + }) + + def _status(self) -> httpx.Response: + answer = self.statuses[0] if len(self.statuses) == 1 else self.statuses.pop(0) + if isinstance(answer, type) and issubclass(answer, BaseException): + raise answer() + if isinstance(answer, httpx.Response): + return answer + return httpx.Response(200, json={"request_id": self.request_id, **answer}) + + def getaddrinfo(self, host, port, *args, **kwargs): + address = self.addresses.get(host, PUBLIC_ADDRESS) + return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (address, 0))] + + # ─── what was asked ────────────────────────────────────────────────────── + + def made(self, method: str, *, host: Optional[str] = None, suffix: str = "") -> list[httpx.Request]: + return [ + r for r in self.calls + if r.method == method + and (host is None or r.url.host == host) + and r.url.path.endswith(suffix) + ] + + def submissions(self) -> list[dict]: + return [json.loads(r.content) for r in self.made("POST", host="queue.fal.run")] + + def cancels(self) -> list[httpx.Request]: + return self.made("PUT", host="queue.fal.run", suffix="/cancel") + + def downloads(self) -> list[httpx.Request]: + return [r for r in self.calls if r.url.host not in ("queue.fal.run", "api.fal.ai")] + + +def _media_type(path: Path) -> str: + return { + ".png": "image/png", + ".jpg": "image/jpeg", + ".mp4": "video/mp4", + }.get(path.suffix, "application/octet-stream") + + +@pytest.fixture +def fal(monkeypatch): + fake = FakeFal() + real_client = httpx.Client + monkeypatch.setattr( + httpx, "Client", lambda **kwargs: real_client(transport=httpx.MockTransport(fake), **kwargs) + ) + monkeypatch.setattr(socket, "getaddrinfo", fake.getaddrinfo) + monkeypatch.setattr(run_module, "sleep", fake.clock.sleep) + monkeypatch.setattr(run_module, "now", fake.clock.now) + pricing.clear_cache() + yield fake + pricing.clear_cache() + + +# ─── shared setup for the generation tests ─────────────────────────────────── + +TEST_USER = "user-under-test" +FAL_KEY = "fal-key-id:fal-key-secret" + +T2I = "fal-ai/flux/dev" +I2I = "fal-ai/nano-banana/edit" +I2V = "fal-ai/kling-video/v2.5-turbo/pro/image-to-video" +VEO = "fal-ai/veo3.1/fast/image-to-video" + + +def seed_models(session) -> dict: + """Four catalogue rows, shaped like the seeded ones they are named after. + + Built directly rather than through the migration: the test database comes from + `create_all`, which seeds nothing. That the migration's own rows are well-formed is + asserted in test_migrations.py. + """ + from app.models.generation import GenerationModel + + rows = { + T2I: GenerationModel( + endpoint_id=T2I, kind="text_to_image", label="FLUX.1 [dev]", sort_order=20, + options=json.dumps({ + "image_sizes": ["square_hd", "landscape_16_9"], + "supports_seed": True, + "max_outputs": 4, + }), + unit_price=0.025, price_unit="megapixel", price_currency="USD", + ), + I2I: GenerationModel( + endpoint_id=I2I, kind="image_to_image", label="Nano Banana (edit)", sort_order=10, + image_field="image_urls", image_field_is_list=True, max_images=3, + options=json.dumps({ + "aspect_ratios": ["auto", "1:1", "16:9"], + "supports_seed": True, + "max_outputs": 4, + }), + unit_price=0.039, price_unit="image", price_currency="USD", + ), + I2V: GenerationModel( + endpoint_id=I2V, kind="image_to_video", label="Kling 2.5 Turbo Pro", sort_order=10, + image_field="image_url", max_images=1, end_image_field="tail_image_url", + options=json.dumps({"durations": ["5", "10"], "supports_negative_prompt": True}), + unit_price=0.07, price_unit="second", price_currency="USD", + ), + VEO: GenerationModel( + endpoint_id=VEO, kind="image_to_video", label="Veo 3.1 Fast", sort_order=30, + image_field="image_url", max_images=1, + options=json.dumps({ + "durations": ["4s", "6s", "8s"], + "resolutions": ["720p", "1080p"], + "supports_seed": True, + "supports_audio": True, + "supports_negative_prompt": True, + }), + extra_params=json.dumps({"generate_audio": False}), + unit_price=0.10, price_unit="second", price_currency="USD", + ), + } + for row in rows.values(): + session.add(row) + session.commit() + for row in rows.values(): + session.refresh(row) + return rows + + +def set_fal_key(session, user_id: str = TEST_USER) -> None: + from app.settings_store import FAL_API_KEY, set_setting + + set_setting(session, user_id, FAL_API_KEY, FAL_KEY) + + +def upload(client, name: str, media_type: str) -> dict: + response = client.post( + "/api/assets", + files=[("files", (name, (FIXTURES / name).read_bytes(), media_type))], + ) + assert response.status_code in (200, 201), response.text + return response.json()["created"][0] + + +def run_job(session, monkeypatch, job_id: str): + """Run a queued generation on this thread, against the test database.""" + from app.jobs import enrichment as enrichment_jobs + from app.models.job import EnrichmentJob + + queue = enrichment_jobs.generation_queue() + monkeypatch.setattr(queue, "engine", session.get_bind()) + enrichment_jobs._run_generation_job(job_id) + session.expire_all() + return session.get(EnrichmentJob, job_id) diff --git a/backend/tests/test_generation_api.py b/backend/tests/test_generation_api.py new file mode 100644 index 0000000..742543b --- /dev/null +++ b/backend/tests/test_generation_api.py @@ -0,0 +1,473 @@ +"""`POST /api/generate` and `/regenerate` — what is refused before anything is queued. + +The order of the checks is part of the contract (docs/m8-ai-generation.md), so several +tests send a request wrong in two ways and assert which one is reported. +""" + +import json + +import pytest +from sqlmodel import select + +from app.jobs import enrichment as enrichment_jobs +from app.models.asset import Asset +from app.models.job import KIND_GENERATE, EnrichmentJob +from tests.fal_stub import ( # noqa: F401 - `fal` is a fixture + I2I, + I2V, + T2I, + TEST_USER, + VEO, + fal, + seed_models, + set_fal_key, + upload, +) + +PROMPT = "A lighthouse on a rocky headland under a bruised sky" + + +@pytest.fixture +def models(session): + return seed_models(session) + + +@pytest.fixture +def keyed(session, models): + set_fal_key(session) + return models + + +def _post(library, body): + return library.post("/api/generate", json=body) + + +def _refused(response, status, code): + assert response.status_code == status, response.text + detail = response.json()["detail"] + assert detail["code"] == code + assert detail["message"] + return detail["message"] + + +def _request_of(session, job_id): + return json.loads(session.get(EnrichmentJob, job_id).payload)["request"] + + +def _someone_elses_image(session): + asset = Asset( + user_id="someone-else", name="Their photo", asset_type="image", + storage_key="someone-else/photo.jpg", file_format="jpg", + ) + session.add(asset) + session.commit() + return asset + + +# ─── what a valid request queues ───────────────────────────────────────────── + + +def test_a_generation_is_queued_as_a_job(library, session, keyed): + response = _post(library, { + "model_id": keyed[T2I].id, + "prompt": f" {PROMPT} ", + "params": {"image_size": "landscape_16_9", "seed": 7, "num_outputs": 3}, + }) + + assert response.status_code == 202, response.text + job = response.json()["data"] + assert job["kind"] == "enrichment" + assert job["action"] == KIND_GENERATE + assert job["status"] == "queued" + assert job["asset_id"] is None + assert job["asset_name"] == PROMPT + assert job["model"] == T2I + assert job["result_asset_id"] is None + assert job["result_asset_ids"] == [] + + request = _request_of(session, job["id"]) + assert request["endpoint_id"] == T2I + assert request["kind"] == "text_to_image" + assert request["prompt"] == PROMPT + assert request["parameters"] == {"image_size": "landscape_16_9", "seed": 7, "num_images": 3} + assert request["sources"] == [] + + +def test_catalogue_defaults_are_part_of_the_stored_parameters(library, session, keyed): + photo = upload(library, "sample_image.jpg", "image/jpeg") + response = _post(library, { + "model_id": keyed[VEO].id, + "prompt": PROMPT, + "base_asset_ids": [photo["id"]], + "params": {"duration": "8s", "resolution": "1080p"}, + }) + assert response.status_code == 202, response.text + + # Veo's `generate_audio: false` default is recorded, so a regeneration after an + # admin changes it still makes what this one made. + parameters = _request_of(session, response.json()["data"]["id"])["parameters"] + assert parameters == {"generate_audio": False, "duration": "8s", "resolution": "1080p"} + + +def test_a_user_choice_overrides_a_catalogue_default(library, session, keyed): + photo = upload(library, "sample_image.jpg", "image/jpeg") + response = _post(library, { + "model_id": keyed[VEO].id, + "prompt": PROMPT, + "base_asset_ids": [photo["id"]], + "params": {"generate_audio": True}, + }) + assert _request_of(session, response.json()["data"]["id"])["parameters"]["generate_audio"] is True + + +def test_it_runs_on_the_generation_queue_not_the_enrichment_one(library, session, keyed): + job_id = _post(library, {"model_id": keyed[T2I].id, "prompt": PROMPT}).json()["data"]["id"] + + assert job_id in list(enrichment_jobs.generation_queue()._queue.queue) + assert job_id not in list(enrichment_jobs.queue()._queue.queue) + + +def test_cancelling_reaches_the_generation_queue(library, session, keyed): + job_id = _post(library, {"model_id": keyed[T2I].id, "prompt": PROMPT}).json()["data"]["id"] + + response = library.delete(f"/api/activity/enrichment/{job_id}") + + assert response.json()["data"]["status"] == "cancelled" + assert enrichment_jobs.generation_queue().is_cancelled(job_id) + assert not enrichment_jobs.queue().is_cancelled(job_id) + + +# ─── the checks, in order ──────────────────────────────────────────────────── + + +def test_no_key_is_reported_first(library, models): + message = _refused(_post(library, {"model_id": "no-such-model", "prompt": ""}), 400, "fal_key_missing") + assert "Settings" in message + + +def test_an_unknown_model(library, keyed): + _refused(_post(library, {"model_id": "no-such-model", "prompt": ""}), 404, "model_not_found") + + +def test_a_switched_off_model(library, session, keyed): + keyed[T2I].is_active = False + session.add(keyed[T2I]) + session.commit() + _refused(_post(library, {"model_id": keyed[T2I].id, "prompt": ""}), 409, "model_inactive") + + +@pytest.mark.parametrize("prompt", ["", " ", "x" * 4001]) +def test_the_prompt_must_be_there_and_not_too_long(library, keyed, prompt): + _refused(_post(library, {"model_id": keyed[T2I].id, "prompt": prompt}), 422, "invalid_generation") + + +def test_a_prompt_of_exactly_the_limit_is_accepted(library, keyed): + assert _post(library, {"model_id": keyed[T2I].id, "prompt": "x" * 4000}).status_code == 202 + + +def test_text_to_image_takes_no_base(library, keyed): + photo = upload(library, "sample_image.jpg", "image/jpeg") + message = _refused( + _post(library, {"model_id": keyed[T2I].id, "prompt": PROMPT, "base_asset_ids": [photo["id"]]}), + 422, "invalid_generation", + ) + assert "takes no base image" in message + + +@pytest.mark.parametrize("count", [0, 4]) +def test_image_to_image_takes_between_one_and_its_maximum(library, keyed, count): + photo = upload(library, "sample_image.jpg", "image/jpeg") + message = _refused( + _post(library, {"model_id": keyed[I2I].id, "prompt": PROMPT, "base_asset_ids": [photo["id"]] * count}), + 422, "invalid_generation", + ) + assert "between 1 and 3" in message + + +@pytest.mark.parametrize("count", [0, 2]) +def test_image_to_video_takes_exactly_one_start_image(library, keyed, count): + photo = upload(library, "sample_image.jpg", "image/jpeg") + _refused( + _post(library, {"model_id": keyed[I2V].id, "prompt": PROMPT, "base_asset_ids": [photo["id"]] * count}), + 422, "invalid_generation", + ) + + +def test_an_end_frame_needs_a_model_with_somewhere_to_put_it(library, keyed): + photo = upload(library, "sample_image.jpg", "image/jpeg") + message = _refused( + _post(library, { + "model_id": keyed[VEO].id, "prompt": PROMPT, + "base_asset_ids": [photo["id"]], "end_frame_asset_id": photo["id"], + }), + 422, "invalid_generation", + ) + assert "end frame" in message + + +@pytest.mark.parametrize( + "model, params, fragment", + [ + (T2I, {"image_size": "enormous"}, "square_hd, landscape_16_9"), + # The declared value is the string "5"; the number 5 is a different request. + (I2V, {"duration": 5}, "5, 10"), + (T2I, {"aspect_ratio": "1:1"}, "does not take an aspect ratio"), + (I2V, {"resolution": "720p"}, "does not take a resolution"), + ], +) +def test_a_choice_must_be_one_the_model_declares(library, keyed, model, params, fragment): + photo = upload(library, "sample_image.jpg", "image/jpeg") + bases = [photo["id"]] if model != T2I else [] + message = _refused( + _post(library, {"model_id": keyed[model].id, "prompt": PROMPT, "base_asset_ids": bases, "params": params}), + 422, "invalid_generation", + ) + assert fragment in message + + +@pytest.mark.parametrize( + "model, params", + [ + (I2V, {"seed": 1}), # Kling 2.5 reports and takes no seed + (T2I, {"negative_prompt": "blur"}), # FLUX has no negative prompt + (T2I, {"generate_audio": True}), # nor any audio + ], +) +def test_a_feature_the_model_lacks_is_refused(library, keyed, model, params): + photo = upload(library, "sample_image.jpg", "image/jpeg") + bases = [photo["id"]] if model != T2I else [] + _refused( + _post(library, {"model_id": keyed[model].id, "prompt": PROMPT, "base_asset_ids": bases, "params": params}), + 422, "invalid_generation", + ) + + +def test_an_empty_negative_prompt_is_not_a_request_for_one(library, keyed): + response = _post(library, {"model_id": keyed[T2I].id, "prompt": PROMPT, "params": {"negative_prompt": ""}}) + assert response.status_code == 202 + + +@pytest.mark.parametrize("model, count", [(T2I, 0), (T2I, 5), (I2V, 2)]) +def test_the_number_of_outputs_is_bounded(library, keyed, model, count): + photo = upload(library, "sample_image.jpg", "image/jpeg") + bases = [photo["id"]] if model != T2I else [] + _refused( + _post(library, {"model_id": keyed[model].id, "prompt": PROMPT, "base_asset_ids": bases, "params": {"num_outputs": count}}), + 422, "invalid_generation", + ) + + +def test_someone_elses_base_is_not_found(library, session, keyed): + theirs = _someone_elses_image(session) + _refused( + _post(library, {"model_id": keyed[I2I].id, "prompt": PROMPT, "base_asset_ids": [theirs.id]}), + 404, "asset_not_found", + ) + + +def test_a_bad_request_is_reported_before_an_unknown_base(library, keyed): + _refused( + _post(library, {"model_id": keyed[I2I].id, "prompt": "", "base_asset_ids": ["nope"]}), + 422, "invalid_generation", + ) + + +def test_ownership_is_checked_before_usability(library, session, keyed): + video = upload(library, "sample_video.mp4", "video/mp4") + theirs = _someone_elses_image(session) + _refused( + _post(library, {"model_id": keyed[I2I].id, "prompt": PROMPT, "base_asset_ids": [video["id"], theirs.id]}), + 404, "asset_not_found", + ) + + +def test_a_base_must_be_an_image(library, keyed): + video = upload(library, "sample_video.mp4", "video/mp4") + message = _refused( + _post(library, {"model_id": keyed[I2I].id, "prompt": PROMPT, "base_asset_ids": [video["id"]]}), + 422, "invalid_base", + ) + assert "not an image" in message + + +def test_a_base_must_own_a_file(library, session, keyed): + clip = Asset(user_id=TEST_USER, name="A clip", asset_type="image", source="clip") + session.add(clip) + session.commit() + _refused( + _post(library, {"model_id": keyed[I2I].id, "prompt": PROMPT, "base_asset_ids": [clip.id]}), + 422, "invalid_base", + ) + + +def test_a_base_must_be_a_format_pillow_reads(library, keyed): + response = library.post( + "/api/assets", + files=[("files", ("logo.svg", b'', "image/svg+xml"))], + ) + svg = response.json()["created"][0] + message = _refused( + _post(library, {"model_id": keyed[I2I].id, "prompt": PROMPT, "base_asset_ids": [svg["id"]]}), + 422, "invalid_base", + ) + assert "SVG" in message + + +def test_a_base_must_actually_be_readable(library, keyed): + response = library.post( + "/api/assets", files=[("files", ("broken.png", b"not a png at all", "image/png"))] + ) + broken = response.json()["created"][0] + message = _refused( + _post(library, {"model_id": keyed[I2I].id, "prompt": PROMPT, "base_asset_ids": [broken["id"]]}), + 422, "invalid_base", + ) + assert "could not be read" in message + + +def test_the_end_frame_is_checked_like_a_base(library, session, keyed): + photo = upload(library, "sample_image.jpg", "image/jpeg") + theirs = _someone_elses_image(session) + _refused( + _post(library, { + "model_id": keyed[I2V].id, "prompt": PROMPT, + "base_asset_ids": [photo["id"]], "end_frame_asset_id": theirs.id, + }), + 404, "asset_not_found", + ) + + +def test_generating_requires_authentication(client): + assert client.post("/api/generate", json={}).status_code == 401 + + +# ─── regenerate ────────────────────────────────────────────────────────────── + + +@pytest.fixture +def made(library, session, keyed): + """A generated image→image asset, as the job would have left it.""" + first = upload(library, "sample_image.jpg", "image/jpeg") + second = upload(library, "sample_image.png", "image/png") + asset = Asset( + user_id=TEST_USER, + name=PROMPT, + description=PROMPT, + asset_type="image", + source="ai_generated", + storage_key=f"{TEST_USER}/made.png", + ai_model=I2I, + ai_generation_type="image_to_image", + ai_prompt=PROMPT, + ai_parameters=json.dumps({"aspect_ratio": "1:1", "num_images": 1, "seed": 42}), + ai_source_assets=json.dumps([ + {"asset_id": first["id"], "role": "base", "name": "First"}, + {"asset_id": second["id"], "role": "base", "name": "Second"}, + ]), + ai_seed=1234, + ) + session.add(asset) + session.commit() + return {"asset": asset, "bases": [first["id"], second["id"]]} + + +def _regenerate(library, asset_id, body=None): + return library.post(f"/api/generate/{asset_id}/regenerate", json=body) + + +def test_regenerate_resends_the_stored_request(library, session, made): + response = _regenerate(library, made["asset"].id) + assert response.status_code == 202, response.text + job = response.json()["data"] + assert job["action"] == KIND_GENERATE + assert job["model"] == I2I + + request = _request_of(session, job["id"]) + assert request["endpoint_id"] == I2I + assert request["prompt"] == PROMPT + assert request["image_field"] == "image_urls" and request["image_field_is_list"] is True + assert [s["asset_id"] for s in request["sources"]] == made["bases"] + assert {s["role"] for s in request["sources"]} == {"base"} + assert request["regenerated_from"] == made["asset"].id + # Everything as stored, except the seed: a regeneration that reproduced the same + # picture would not be one. + assert request["parameters"] == {"aspect_ratio": "1:1", "num_images": 1} + + +def test_regenerate_with_a_new_prompt(library, session, made): + response = _regenerate(library, made["asset"].id, {"prompt": "The same, at night"}) + assert _request_of(session, response.json()["data"]["id"])["prompt"] == "The same, at night" + + +def test_regenerate_can_reuse_the_seed_fal_reported(library, session, made): + response = _regenerate(library, made["asset"].id, {"reuse_seed": True}) + assert _request_of(session, response.json()["data"]["id"])["parameters"]["seed"] == 1234 + + +def test_reuse_seed_does_nothing_for_a_model_without_seeds(library, session, keyed, made): + options = json.loads(keyed[I2I].options) + options["supports_seed"] = False + keyed[I2I].options = json.dumps(options) + session.add(keyed[I2I]) + session.commit() + + response = _regenerate(library, made["asset"].id, {"reuse_seed": True}) + assert "seed" not in _request_of(session, response.json()["data"]["id"])["parameters"] + + +def test_regenerate_names_the_deleted_base(library, session, made): + assert library.delete(f"/api/assets/{made['bases'][1]}").status_code == 204 + message = _refused(_regenerate(library, made["asset"].id), 409, "source_asset_missing") + assert "Second" in message + + +def test_regenerate_needs_the_model_still_active(library, session, keyed, made): + keyed[I2I].is_active = False + session.add(keyed[I2I]) + session.commit() + _refused(_regenerate(library, made["asset"].id), 409, "model_unavailable") + + +def test_regenerate_needs_the_model_still_in_the_catalogue(library, session, keyed, made): + session.delete(keyed[I2I]) + session.commit() + message = _refused(_regenerate(library, made["asset"].id), 409, "model_unavailable") + assert I2I in message + + +def test_only_a_generated_asset_can_be_regenerated(library, keyed): + photo = upload(library, "sample_image.jpg", "image/jpeg") + _refused(_regenerate(library, photo["id"]), 409, "not_generated") + + +def test_someone_elses_asset_cannot_be_regenerated(library, session, keyed): + theirs = _someone_elses_image(session) + theirs.ai_model = I2I + theirs.ai_generation_type = "image_to_image" + session.add(theirs) + session.commit() + assert _regenerate(library, theirs.id).status_code == 404 + + +def test_regenerate_without_a_key(library, session, made): + from app.settings_store import FAL_API_KEY, clear_setting + + clear_setting(session, TEST_USER, FAL_API_KEY) + _refused(_regenerate(library, made["asset"].id), 400, "fal_key_missing") + + +def test_a_regenerated_job_runs_end_to_end(library, session, monkeypatch, made, fal): + from tests.fal_stub import run_job + + job_id = _regenerate(library, made["asset"].id, {"reuse_seed": True}).json()["data"]["id"] + job = run_job(session, monkeypatch, job_id) + assert job.status == "done", job.error_message + + (submitted,) = fal.submissions() + assert submitted["seed"] == 1234 + assert len(submitted["image_urls"]) == 2 + made_again = session.exec( + select(Asset).where(Asset.source == "ai_generated", Asset.id != made["asset"].id) + ).one() + assert made_again.ai_prompt == PROMPT + assert [s["asset_id"] for s in json.loads(made_again.ai_source_assets)] == made["bases"] diff --git a/backend/tests/test_generation_catalogue.py b/backend/tests/test_generation_catalogue.py new file mode 100644 index 0000000..e7baa59 --- /dev/null +++ b/backend/tests/test_generation_catalogue.py @@ -0,0 +1,356 @@ +"""The model catalogue, who may change it, and the fal key. + +Admins are named in config (`ADMIN_USERS`) because the Notes token carries no admin +claim. The signed-in test user is `user-under-test` / `tester`. +""" + +import json + +import pytest +from sqlmodel import select + +from app.config import settings +from app.models.setting import UserSetting +from app.models.user import User +from app.settings_store import FAL_API_KEY, get_setting +from tests.fal_stub import I2I, I2V, T2I, TEST_USER, VEO, seed_models +from tests.test_auth import make_token + +NEW = { + "endpoint_id": "fal-ai/flux/schnell", + "kind": "text_to_image", + "label": "FLUX.1 [schnell]", + "options": {"image_sizes": ["square_hd"], "supports_seed": True, "max_outputs": 4}, + "unit_price": 0.003, + "price_unit": "megapixel", + "price_currency": "USD", +} + +VIDEO = { + "endpoint_id": "fal-ai/minimax/hailuo-02/standard/image-to-video", + "kind": "image_to_video", + "label": "Hailuo 02", + "image_field": "image_url", + "max_images": 1, + "end_image_field": "end_image_url", + "options": {"durations": ["6", "10"], "resolutions": ["512P", "768P"]}, +} + + +@pytest.fixture +def admin(monkeypatch): + monkeypatch.setattr(settings, "admin_users", TEST_USER) + + +@pytest.fixture +def models(session): + return seed_models(session) + + +def _refused(response, status, code): + assert response.status_code == status, response.text + assert response.json()["detail"]["code"] == code + return response.json()["detail"]["message"] + + +# ─── reading ───────────────────────────────────────────────────────────────── + + +def test_everyone_can_read_the_catalogue_in_form_order(auth_client, models): + body = auth_client.get("/api/generate/models").json() + + assert body["total"] == 4 + order = [(m["kind"], m["endpoint_id"]) for m in body["data"]] + assert order == [ + ("image_to_image", I2I), + ("image_to_video", I2V), + ("image_to_video", VEO), + ("text_to_image", T2I), + ] + + +def test_a_row_reads_with_every_option_present(auth_client, models): + body = auth_client.get("/api/generate/models").json()["data"] + flux = next(m for m in body if m["endpoint_id"] == T2I) + + assert flux["options"] == { + "aspect_ratios": [], + "image_sizes": ["square_hd", "landscape_16_9"], + "durations": [], + "resolutions": [], + "supports_seed": True, + "supports_negative_prompt": False, + "supports_audio": False, + "max_outputs": 4, + } + assert flux["extra_params"] == {} + assert flux["image_field"] is None and flux["max_images"] == 0 + assert flux["unit_price"] == 0.025 and flux["price_unit"] == "megapixel" + assert flux["created_at"] and flux["updated_at"] + + veo = next(m for m in body if m["endpoint_id"] == VEO) + assert veo["extra_params"] == {"generate_audio": False} + + +def test_inactive_rows_are_for_admins_only(auth_client, session, models, monkeypatch): + models[T2I].is_active = False + session.add(models[T2I]) + session.commit() + + plain = auth_client.get("/api/generate/models", params={"include_inactive": True}).json() + assert T2I not in [m["endpoint_id"] for m in plain["data"]] + + monkeypatch.setattr(settings, "admin_users", TEST_USER) + assert T2I not in [m["endpoint_id"] for m in auth_client.get("/api/generate/models").json()["data"]] + everything = auth_client.get("/api/generate/models", params={"include_inactive": True}).json() + assert T2I in [m["endpoint_id"] for m in everything["data"]] + + +def test_the_catalogue_requires_authentication(client): + assert client.get("/api/generate/models").status_code == 401 + + +# ─── who is an admin ───────────────────────────────────────────────────────── + + +def test_a_non_admin_cannot_change_the_catalogue(auth_client, models): + message = _refused(auth_client.post("/api/generate/models", json=NEW), 403, "admin_required") + assert message + _refused(auth_client.patch(f"/api/generate/models/{models[T2I].id}", json={"label": "x"}), 403, "admin_required") + _refused(auth_client.delete(f"/api/generate/models/{models[T2I].id}"), 403, "admin_required") + + +@pytest.mark.parametrize("configured", ["user-under-test", "TESTER", " someone , Tester "]) +def test_admin_users_matches_an_id_or_a_username(auth_client, monkeypatch, configured): + monkeypatch.setattr(settings, "admin_users", configured) + assert auth_client.post("/api/generate/models", json=NEW).status_code == 201 + + +def test_an_id_matches_exactly(auth_client, monkeypatch): + monkeypatch.setattr(settings, "admin_users", "USER-UNDER-TEST") + _refused(auth_client.post("/api/generate/models", json=NEW), 403, "admin_required") + + +def test_the_user_rows_own_flag_still_counts(auth_client, session): + session.add(User(id=TEST_USER, username="tester", is_admin=True)) + session.commit() + assert auth_client.post("/api/generate/models", json=NEW).status_code == 201 + + +def test_me_reports_admin_from_config(client, monkeypatch): + token = make_token(sub="notes-user-1", username="davior") + headers = {"Authorization": f"Bearer {token}"} + + assert client.get("/api/me", headers=headers).json()["data"]["is_admin"] is False + + monkeypatch.setattr(settings, "admin_users", "Davior") + assert client.get("/api/me", headers=headers).json()["data"]["is_admin"] is True + + monkeypatch.setattr(settings, "admin_users", "notes-user-1") + assert client.get("/api/me", headers=headers).json()["data"]["is_admin"] is True + + +# ─── creating ──────────────────────────────────────────────────────────────── + + +def test_an_admin_adds_a_model(auth_client, session, admin): + response = auth_client.post("/api/generate/models", json=NEW) + + assert response.status_code == 201, response.text + body = response.json()["data"] + assert body["endpoint_id"] == "fal-ai/flux/schnell" + assert body["is_active"] is True + assert body["note"] == "" + assert body["options"]["image_sizes"] == ["square_hd"] + assert body["options"]["max_outputs"] == 4 + assert body["options"]["durations"] == [] + assert body["unit_price"] == 0.003 + + listed = [m["endpoint_id"] for m in auth_client.get("/api/generate/models").json()["data"]] + assert "fal-ai/flux/schnell" in listed + + +def test_a_video_model_with_an_end_frame(auth_client, admin): + response = auth_client.post("/api/generate/models", json=VIDEO) + assert response.status_code == 201, response.text + assert response.json()["data"]["end_image_field"] == "end_image_url" + + +def test_a_duplicate_endpoint_is_a_conflict(auth_client, admin, models): + _refused( + auth_client.post("/api/generate/models", json={**NEW, "endpoint_id": T2I}), + 409, "endpoint_exists", + ) + + +@pytest.mark.parametrize( + "change, fragment", + [ + ({"kind": "text_to_video"}, "kind"), + ({"label": " "}, "label"), + ({"endpoint_id": "fal-ai/../admin"}, "endpoint_id"), + ({"endpoint_id": "flux"}, "endpoint_id"), + ({"endpoint_id": None}, "endpoint_id"), + ({"image_field": "image_url"}, "no image_field"), + ({"max_images": 1}, "max_images must be 0"), + ({"end_image_field": "tail_image_url"}, "end frame"), + ({"options": "square"}, "options must be an object"), + ({"options": {"image_size": ["square_hd"]}}, "no setting called"), + ({"options": {"image_sizes": "square_hd"}}, "options.image_sizes"), + ({"options": {"max_outputs": 5}}, "max_outputs"), + ({"options": {"max_outputs": 0}}, "max_outputs"), + ({"options": {"supports_seed": "yes"}}, "supports_seed"), + ({"extra_params": ["generate_audio"]}, "extra_params must be an object"), + ({"price_unit": "token"}, "price_unit"), + ({"unit_price": -1}, "unit_price"), + ], +) +def test_an_invalid_text_to_image_entry_is_refused(auth_client, admin, change, fragment): + message = _refused( + auth_client.post("/api/generate/models", json={**NEW, **change}), 422, "invalid_model_entry" + ) + assert fragment in message + + +@pytest.mark.parametrize( + "change, fragment", + [ + ({"image_field": None}, "image_field"), + ({"max_images": 0}, "at least one base image"), + ({"max_images": 2}, "image_field_is_list"), + ({"options": {"max_outputs": 2}}, "max_outputs must be 1"), + ({"image_field": "image url"}, "image_field"), + ], +) +def test_an_invalid_video_entry_is_refused(auth_client, admin, change, fragment): + message = _refused( + auth_client.post("/api/generate/models", json={**VIDEO, **change}), 422, "invalid_model_entry" + ) + assert fragment in message + + +def test_an_image_to_image_entry_cannot_take_an_end_frame(auth_client, admin): + entry = { + "endpoint_id": "fal-ai/flux-pro/kontext", "kind": "image_to_image", "label": "Kontext", + "image_field": "image_url", "max_images": 1, "end_image_field": "tail_image_url", + } + message = _refused(auth_client.post("/api/generate/models", json=entry), 422, "invalid_model_entry") + assert "end frame" in message + + +# ─── updating ──────────────────────────────────────────────────────────────── + + +def test_patch_changes_only_the_fields_sent(auth_client, admin, models): + before = auth_client.get("/api/generate/models").json()["data"] + flux_before = next(m for m in before if m["endpoint_id"] == T2I) + + response = auth_client.patch(f"/api/generate/models/{models[T2I].id}", json={"note": "Good default"}) + + assert response.status_code == 200, response.text + after = response.json()["data"] + assert after["note"] == "Good default" + for field in ("label", "options", "unit_price", "price_unit", "sort_order", "is_active"): + assert after[field] == flux_before[field] + assert after["updated_at"] >= flux_before["updated_at"] + + +def test_patch_null_clears_a_nullable_field(auth_client, session, admin, models): + response = auth_client.patch( + f"/api/generate/models/{models[T2I].id}", + json={"unit_price": None, "price_unit": None, "note": None}, + ) + + assert response.status_code == 200, response.text + body = response.json()["data"] + assert body["unit_price"] is None + assert body["price_unit"] is None + assert body["note"] == "" + session.expire_all() + assert session.get(type(models[T2I]), models[T2I].id).unit_price is None + + +def test_patch_null_on_an_end_frame_field_removes_it(auth_client, admin, models): + response = auth_client.patch(f"/api/generate/models/{models[I2V].id}", json={"end_image_field": None}) + assert response.status_code == 200 + assert response.json()["data"]["end_image_field"] is None + + +def test_patch_null_cannot_clear_a_required_field(auth_client, admin, models): + _refused( + auth_client.patch(f"/api/generate/models/{models[T2I].id}", json={"label": None}), + 422, "invalid_model_entry", + ) + _refused( + auth_client.patch(f"/api/generate/models/{models[I2I].id}", json={"image_field": None}), + 422, "invalid_model_entry", + ) + + +def test_patch_validates_the_row_it_would_leave(auth_client, admin, models): + # Neither field alone is wrong; together they describe an edit endpoint as text → + # image, which is the gecko-notes seed bug this rule exists for. + message = _refused( + auth_client.patch(f"/api/generate/models/{models[I2I].id}", json={"kind": "text_to_image"}), + 422, "invalid_model_entry", + ) + assert "text-to-image" in message + + +def test_patch_to_an_existing_endpoint_is_a_conflict(auth_client, admin, models): + _refused( + auth_client.patch(f"/api/generate/models/{models[T2I].id}", json={"endpoint_id": I2I}), + 409, "endpoint_exists", + ) + + +def test_patch_can_switch_a_model_off(auth_client, admin, models): + response = auth_client.patch(f"/api/generate/models/{models[T2I].id}", json={"is_active": False}) + assert response.json()["data"]["is_active"] is False + assert T2I not in [m["endpoint_id"] for m in auth_client.get("/api/generate/models").json()["data"]] + + +def test_patch_an_unknown_model(auth_client, admin): + _refused(auth_client.patch("/api/generate/models/nope", json={"note": "x"}), 404, "model_not_found") + + +# ─── deleting ──────────────────────────────────────────────────────────────── + + +def test_an_admin_deletes_a_model(auth_client, admin, models): + response = auth_client.delete(f"/api/generate/models/{models[T2I].id}") + assert response.status_code == 204 + assert response.content == b"" + _refused(auth_client.delete(f"/api/generate/models/{models[T2I].id}"), 404, "model_not_found") + + +# ─── the fal key ───────────────────────────────────────────────────────────── + + +def test_the_fal_key_is_stored_and_never_returned(auth_client, session): + assert auth_client.get("/api/settings/generation").json() == {"data": {"fal_key_configured": False}} + + response = auth_client.put("/api/settings/generation", json={"fal_api_key": " fal-secret-123 "}) + + assert response.status_code == 200 + assert response.json() == {"data": {"fal_key_configured": True}} + assert "fal-secret-123" not in auth_client.get("/api/settings/generation").text + # Encrypted at rest, and decrypts to what was typed, trimmed. + row = session.exec( + select(UserSetting).where(UserSetting.user_id == TEST_USER, UserSetting.key == FAL_API_KEY) + ).one() + assert "fal-secret-123" not in row.value + assert json.loads(row.value).startswith("enc:") + assert get_setting(session, TEST_USER, FAL_API_KEY) == "fal-secret-123" + + +def test_an_omitted_key_is_left_alone_and_an_empty_one_removes_it(auth_client): + auth_client.put("/api/settings/generation", json={"fal_api_key": "fal-secret-123"}) + + assert auth_client.put("/api/settings/generation", json={}).json()["data"]["fal_key_configured"] is True + removed = auth_client.put("/api/settings/generation", json={"fal_api_key": ""}) + assert removed.json()["data"]["fal_key_configured"] is False + + +def test_generation_settings_require_authentication(client): + assert client.get("/api/settings/generation").status_code == 401 diff --git a/backend/tests/test_generation_job.py b/backend/tests/test_generation_job.py new file mode 100644 index 0000000..9bcb6f8 --- /dev/null +++ b/backend/tests/test_generation_job.py @@ -0,0 +1,721 @@ +"""The generate job, end to end, with fal.ai stubbed at the httpx seam. + +Everything on GAM's side runs for real: the request body is built from the catalogue +row, the bases are normalised by Pillow, the result is streamed through the capped +download, ingested by `ingest_file` (probe, thumbnail, index), costed, and read back +through the API. What fal says is the one assumption — see `fal_stub.py`. +""" + +import base64 +import io +import json + +import httpx +import pytest +from PIL import Image +from sqlmodel import select + +from app.config import settings +from app.embeddings import PROVIDER_OLLAMA +from app.jobs import enrichment as enrichment_jobs +from app.media_tools import ffmpeg_available +from app.models.asset import Asset +from app.models.job import KIND_EMBED, KIND_TRANSCRIBE, EnrichmentJob +from app.models.usage import UsageEvent +from app.settings_store import DEEPGRAM_API_KEY, EMBEDDING_PROVIDER, set_setting +from tests.fal_stub import ( # noqa: F401 - `fal` is a fixture + CDN, + FAL_KEY, + FIXTURES, + I2I, + I2V, + T2I, + TEST_USER, + ProcessDied, + fal, + run_job, + seed_models, + set_fal_key, + upload, +) + +needs_ffmpeg = pytest.mark.skipif(not ffmpeg_available(), reason="ffmpeg not on PATH") + +PROMPT = "A red fox curled up in fresh snow at golden hour, shallow depth of field" + + +@pytest.fixture +def models(session): + return seed_models(session) + + +@pytest.fixture +def ready(session, models, fal): + """A catalogue, a key, and fal standing by.""" + set_fal_key(session) + return models + + +def _start(library, body): + response = library.post("/api/generate", json=body) + assert response.status_code == 202, response.text + return response.json()["data"]["id"] + + +def _generate(library, session, monkeypatch, body): + job_id = _start(library, body) + run_job(session, monkeypatch, job_id) + return job_id + + +def _activity(library, job_id): + return library.get(f"/api/activity/enrichment/{job_id}").json()["data"] + + +def _generated(session): + return session.exec(select(Asset).where(Asset.source == "ai_generated")).all() + + +def _usage(session): + return session.exec(select(UsageEvent).where(UsageEvent.provider == "fal.ai")).all() + + +def _decoded(uri): + header, _, payload = uri.partition(",") + return header, Image.open(io.BytesIO(base64.b64decode(payload))) + + +# ─── text → image ──────────────────────────────────────────────────────────── + + +@needs_ffmpeg +def test_text_to_image_end_to_end(library, session, monkeypatch, ready, fal): + job_id = _generate( + library, + session, + monkeypatch, + { + "model_id": ready[T2I].id, + "prompt": PROMPT, + "params": {"image_size": "square_hd", "seed": 42, "num_outputs": 1}, + }, + ) + + job = _activity(library, job_id) + assert job["status"] == "done", job["error_message"] + assert job["action"] == "generate" + assert job["model"] == T2I + assert job["detail"] == "1 image" + + (asset,) = _generated(session) + assert job["result_asset_id"] == asset.id + assert job["result_asset_ids"] == [asset.id] + + # What fal was sent: catalogue choices, the seed, the count, and the prompt — and + # authenticated with fal's own scheme. + (submitted,) = fal.submissions() + assert submitted == {"prompt": PROMPT, "image_size": "square_hd", "seed": 42, "num_images": 1} + (submit,) = fal.made("POST", host="queue.fal.run") + assert str(submit.url) == f"https://queue.fal.run/{T2I}" + assert submit.headers["authorization"] == f"Key {FAL_KEY}" + + # The CDN download carries no credentials: the key goes to fal and nowhere else. + (download,) = fal.downloads() + assert "authorization" not in download.headers + + assert asset.asset_type == "image" + assert asset.source == "ai_generated" + assert asset.ai_model == T2I + assert asset.ai_generation_type == "text_to_image" + assert asset.ai_prompt == PROMPT + assert json.loads(asset.ai_parameters) == {"image_size": "square_hd", "seed": 42, "num_images": 1} + assert json.loads(asset.ai_source_assets) == [] + assert asset.ai_seed == 1234 + assert asset.ai_generated_at is not None + assert asset.description == PROMPT + assert asset.name == "A red fox curled up in fresh snow at golden hour, shallow depth of field" + provenance = json.loads(asset.field_provenance) + assert provenance["name"] == "ai" and provenance["description"] == "ai" + + body = library.get(f"/api/assets/{asset.id}").json()["data"] + assert body["source"] == "ai_generated" + assert body["thumb_url"] + assert body["width"] == 400 and body["height"] == 400 + assert body["generation"]["model"] == T2I + assert body["generation"]["kind"] == "text_to_image" + assert body["generation"]["prompt"] == PROMPT + assert body["generation"]["parameters"] == {"image_size": "square_hd", "seed": 42, "num_images": 1} + assert body["generation"]["sources"] == [] + assert body["generation"]["seed"] == 1234 + assert body["generation"]["generated_at"] + + # One usage row: fal's billed units times fal's own price, so exact. + (usage,) = _usage(session) + assert usage.asset_id == asset.id + assert usage.user_id == TEST_USER + assert usage.kind == "image" + assert usage.model == T2I + assert usage.units == 1 + assert usage.unit_type == "megapixels" + assert usage.cost == pytest.approx(0.025) + assert usage.currency == "USD" + assert usage.cost_estimated is False + assert usage.external_ref == "fal-req-1" + + totals = library.get(f"/api/usage/assets/{asset.id}").json()["data"] + assert totals["cost"] == pytest.approx(0.025) + assert totals["estimated"] is False + + +def test_the_prompt_makes_the_generation_findable(library, session, monkeypatch, ready, fal): + _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + hits = library.get("/api/assets", params={"q": "fox"}).json() + assert hits["total"] == 1 + listing = library.get("/api/assets", params={"source": "ai_generated"}).json() + assert listing["total"] == 1 + + +def test_a_long_prompt_names_the_asset_at_a_word_boundary(library, session, monkeypatch, ready, fal): + prompt = "A lighthouse " + "on a rocky headland under a bruised sky " * 6 + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": prompt}) + + (asset,) = _generated(session) + assert len(asset.name) <= 80 + assert asset.name.endswith("…") + assert prompt.startswith(asset.name[:-1]) + assert asset.description == prompt.strip() + assert _activity(library, job_id)["asset_name"] == asset.name + + +def test_several_outputs_become_several_assets_in_order(library, session, monkeypatch, ready, fal): + fal.result_body = { + "images": [ + {"url": f"{CDN}/out-0.png", "content_type": "image/png"}, + {"url": f"{CDN}/out-1.jpg", "content_type": "image/jpeg"}, + ], + "seed": 7, + } + fal.files[f"{CDN}/out-1.jpg"] = FIXTURES / "sample_image.jpg" + fal.result_headers = {"x-fal-billable-units": "2", "x-fal-request-id": "fal-req-1"} + + job_id = _generate( + library, session, monkeypatch, + {"model_id": ready[T2I].id, "prompt": PROMPT, "params": {"num_outputs": 2}}, + ) + + job = _activity(library, job_id) + assert job["detail"] == "2 images" + assert fal.submissions()[0]["num_images"] == 2 + assets = {a.id: a for a in _generated(session)} + assert len(assets) == 2 + first, second = job["result_asset_ids"] + assert job["result_asset_id"] == first + assert assets[first].file_format == "png" + assert assets[second].file_format == "jpg" + + # One request, one bill — attached to the first output. + (usage,) = _usage(session) + assert usage.asset_id == first + assert usage.units == 2 + assert usage.cost == pytest.approx(0.05) + + +def test_an_embedding_provider_gets_the_new_asset_embedded(library, session, monkeypatch, ready, fal): + set_setting(session, TEST_USER, EMBEDDING_PROVIDER, PROVIDER_OLLAMA) + _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + (asset,) = _generated(session) + embeds = session.exec( + select(EnrichmentJob).where(EnrichmentJob.asset_id == asset.id, EnrichmentJob.kind == KIND_EMBED) + ).all() + assert len(embeds) == 1 + + +# ─── image → image ─────────────────────────────────────────────────────────── + + +def test_image_to_image_sends_every_base_as_a_data_uri_in_order(library, session, monkeypatch, ready, fal): + fal.price = (0.039, "image") + photo = upload(library, "sample_image.jpg", "image/jpeg") + cutout = upload(library, "sample_image.png", "image/png") + + job_id = _generate( + library, session, monkeypatch, + { + "model_id": ready[I2I].id, + "prompt": "Put the subject of the second picture into the first", + "base_asset_ids": [photo["id"], cutout["id"]], + "params": {"aspect_ratio": "1:1"}, + }, + ) + assert _activity(library, job_id)["status"] == "done" + + (submitted,) = fal.submissions() + assert set(submitted) == {"prompt", "aspect_ratio", "num_images", "image_urls"} + first, second = submitted["image_urls"] + # JPEG for the opaque photo; PNG for the one with real transparency, which an edit + # model needs to see. + header, image = _decoded(first) + assert header == "data:image/jpeg;base64" + assert image.size == (800, 600) + header, image = _decoded(second) + assert header == "data:image/png;base64" + assert image.mode == "RGBA" + + (asset,) = _generated(session) + assert asset.ai_generation_type == "image_to_image" + assert json.loads(asset.ai_parameters) == {"aspect_ratio": "1:1", "num_images": 1} + assert json.loads(asset.ai_source_assets) == [ + {"asset_id": photo["id"], "role": "base", "name": "sample_image"}, + {"asset_id": cutout["id"], "role": "base", "name": "sample_image"}, + ] + # A base is not a parent: no inherited attribution, no Clip tab. + assert asset.parent_asset_id is None + + # Priced per image, because fal's pricing API says this endpoint is. + (usage,) = _usage(session) + assert usage.unit_type == "images" + assert usage.cost == pytest.approx(0.039) + assert usage.cost_estimated is False + + +def _upload_bytes(library, name, data, media_type): + response = library.post("/api/assets", files=[("files", (name, data, media_type))]) + assert response.status_code in (200, 201), response.text + return response.json()["created"][0] + + +def test_a_base_is_turned_upright_and_capped_before_it_is_sent(library, session, monkeypatch, ready, fal): + # A phone photo: stored landscape, tagged "rotate 90° clockwise to view". + picture = Image.new("RGB", (3000, 1000), (200, 40, 40)) + exif = Image.Exif() + exif[0x0112] = 6 + buffer = io.BytesIO() + picture.save(buffer, format="JPEG", exif=exif) + phone = _upload_bytes(library, "phone.jpg", buffer.getvalue(), "image/jpeg") + + _generate( + library, session, monkeypatch, + {"model_id": ready[I2I].id, "prompt": "Make it night", "base_asset_ids": [phone["id"]]}, + ) + + (submitted,) = fal.submissions() + _header, sent = _decoded(submitted["image_urls"][0]) + # Upright (portrait), and the long edge brought down to the 2048 px cap. + assert sent.size == (683, 2048) + + +def test_a_base_still_too_large_after_resizing_is_refused(library, session, monkeypatch, ready, fal): + monkeypatch.setattr(settings, "generation_max_data_uri_mb", 0) + photo = upload(library, "sample_image.jpg", "image/jpeg") + job_id = _generate( + library, session, monkeypatch, + {"model_id": ready[I2I].id, "prompt": "Make it night", "base_asset_ids": [photo["id"]]}, + ) + + job = _activity(library, job_id) + assert job["status"] == "error" + assert job["error_message"].startswith("A base image is larger than the 0 MB limit") + assert fal.submissions() == [] + + +# ─── image → video ─────────────────────────────────────────────────────────── + + +@needs_ffmpeg +def test_image_to_video_puts_the_end_frame_in_the_models_field(library, session, monkeypatch, ready, fal): + # A Deepgram key is set, which would transcribe an uploaded video on arrival. A + # generated one is not transcribed: most are silent, and each would be billed. + set_setting(session, TEST_USER, DEEPGRAM_API_KEY, "dg-key") + start = upload(library, "sample_image.jpg", "image/jpeg") + end = upload(library, "attributed_image.jpg", "image/jpeg") + fal.result_body = {"video": {"url": f"{CDN}/clip.mp4", "content_type": "video/mp4"}} + fal.files[f"{CDN}/clip.mp4"] = FIXTURES / "sample_video.mp4" + fal.result_headers = {"x-fal-billable-units": "5", "x-fal-request-id": "fal-req-1"} + fal.price = (0.07, "second") + + job_id = _generate( + library, session, monkeypatch, + { + "model_id": ready[I2V].id, + "prompt": "The fox wakes and runs off", + "base_asset_ids": [start["id"]], + "end_frame_asset_id": end["id"], + "params": {"duration": "5", "negative_prompt": "blur"}, + }, + ) + job = _activity(library, job_id) + assert job["status"] == "done", job["error_message"] + assert job["detail"] == "1 video" + + (submitted,) = fal.submissions() + assert submitted["image_url"].startswith("data:image/jpeg;base64,") + assert submitted["tail_image_url"].startswith("data:image/jpeg;base64,") + assert submitted["duration"] == "5" + assert submitted["negative_prompt"] == "blur" + # A video model makes one, and is not sent a count it has no field for. + assert "num_images" not in submitted + # Polled at the app path fal's queue uses, not the full endpoint id. + assert fal.made("GET", suffix="/status")[0].url.path == "/fal-ai/kling-video/requests/req-1/status" + + (asset,) = _generated(session) + assert asset.asset_type == "video" + assert asset.duration_seconds == pytest.approx(2.0, abs=0.2) + assert asset.ai_seed is None + assert json.loads(asset.ai_source_assets) == [ + {"asset_id": start["id"], "role": "base", "name": "sample_image"}, + {"asset_id": end["id"], "role": "end_frame", "name": "attributed_image"}, + ] + + transcribes = session.exec( + select(EnrichmentJob).where(EnrichmentJob.asset_id == asset.id, EnrichmentJob.kind == KIND_TRANSCRIBE) + ).all() + assert transcribes == [] + + (usage,) = _usage(session) + assert usage.kind == "video" + # Not "seconds", which the totals already count as audio transcribed. + assert usage.unit_type == "generated_seconds" + assert usage.units == 5 + assert usage.cost == pytest.approx(0.35) + + +def test_a_base_deleted_while_queued_fails_readably(library, session, monkeypatch, ready, fal): + photo = upload(library, "sample_image.jpg", "image/jpeg") + job_id = _start( + library, + {"model_id": ready[I2I].id, "prompt": "Make it night", "base_asset_ids": [photo["id"]]}, + ) + assert library.delete(f"/api/assets/{photo['id']}").status_code == 204 + + job = run_job(session, monkeypatch, job_id) + assert job.status == "error" + assert job.error_message == "A base image was deleted before this could run" + assert fal.submissions() == [] + + +# ─── restarts ──────────────────────────────────────────────────────────────── + + +def test_a_restart_resumes_the_stored_request_instead_of_submitting_again( + library, session, monkeypatch, ready, fal +): + fal.statuses = [{"status": "IN_QUEUE", "queue_position": 2}, ProcessDied] + job_id = _start(library, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + with pytest.raises(ProcessDied): + run_job(session, monkeypatch, job_id) + + # What the killed process left: a running row whose request ids were committed + # before the first poll. + session.expire_all() + job = session.get(EnrichmentJob, job_id) + assert job.status == "processing" + stored = json.loads(job.payload) + assert stored["fal_request"]["request_id"] == "req-1" + assert stored["fal_request"]["cancel_url"].endswith("/requests/req-1/cancel") + + queue = enrichment_jobs.generation_queue() + monkeypatch.setattr(queue, "engine", session.get_bind()) + queue.recover_pending() + session.expire_all() + assert session.get(EnrichmentJob, job_id).status == "queued" + + fal.statuses = [{"status": "COMPLETED"}] + job = run_job(session, monkeypatch, job_id) + + assert job.status == "done", job.error_message + assert len(fal.submissions()) == 1 + assert len(_generated(session)) == 1 + + +def test_a_restart_after_saving_does_not_save_or_bill_twice(library, session, monkeypatch, ready, fal): + # Dies looking up the price: after the asset is saved, before the cost is recorded. + fal.pricing_raises = ProcessDied + job_id = _start(library, {"model_id": ready[T2I].id, "prompt": PROMPT}) + with pytest.raises(ProcessDied): + run_job(session, monkeypatch, job_id) + + session.expire_all() + stored = json.loads(session.get(EnrichmentJob, job_id).payload) + assert len(stored["created_asset_ids"]) == 1 + assert stored["usage_recorded"] is False + + fal.pricing_raises = None + job = run_job(session, monkeypatch, job_id) + + assert job.status == "done", job.error_message + assert len(_generated(session)) == 1 + assert len(fal.submissions()) == 1 + assert len(fal.downloads()) == 1 + # The result was kept with the job, so the rerun did not even ask for it again. + assert len(fal.made("GET", host="queue.fal.run", suffix="/req-1")) == 1 + assert len(_usage(session)) == 1 + assert json.loads(job.payload)["created_asset_ids"] == stored["created_asset_ids"] + + +def test_running_a_finished_generation_again_changes_nothing(library, session, monkeypatch, ready, fal): + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + calls = len(fal.calls) + + job = session.get(EnrichmentJob, job_id) + job.status = "queued" + session.add(job) + session.commit() + job = run_job(session, monkeypatch, job_id) + + assert job.status == "done" + assert len(_generated(session)) == 1 + assert len(_usage(session)) == 1 + assert len(fal.calls) == calls + + +# ─── stopping ──────────────────────────────────────────────────────────────── + + +def test_a_user_cancel_tells_fal_to_stop(library, session, monkeypatch, ready, fal): + fal.statuses = [{"status": "IN_PROGRESS"}] + job_id = _start(library, {"model_id": ready[T2I].id, "prompt": PROMPT}) + fal.clock.on_sleep = lambda: library.delete(f"/api/activity/enrichment/{job_id}") + + job = run_job(session, monkeypatch, job_id) + + assert job.status == "cancelled" + (cancel,) = fal.cancels() + assert str(cancel.url) == f"https://queue.fal.run/fal-ai/flux/requests/req-1/cancel" + assert cancel.headers["authorization"] == f"Key {FAL_KEY}" + assert _generated(session) == [] + assert _usage(session) == [] + + +def test_the_deadline_cancels_at_fal_and_says_so(library, session, monkeypatch, ready, fal): + monkeypatch.setattr(settings, "generation_poll_seconds", 60) + fal.statuses = [{"status": "IN_QUEUE", "queue_position": 0}, {"status": "IN_PROGRESS"}] + job_id = _start(library, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + job = run_job(session, monkeypatch, job_id) + + assert job.status == "error" + assert job.error_message == "fal.ai did not finish within 30 minutes" + assert len(fal.cancels()) == 1 + # Waited the whole thirty minutes on the fake clock, in steps of the poll interval. + assert sum(fal.clock.slept) == pytest.approx(30 * 60) + assert set(fal.clock.slept) == {60} + + +def test_a_status_that_fails_once_is_polled_again(library, session, monkeypatch, ready, fal): + fal.statuses = [httpx.Response(502, json={"detail": "Bad gateway"}), {"status": "COMPLETED"}] + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + assert _activity(library, job_id)["status"] == "done" + + +def test_completed_with_an_error_is_a_failure(library, session, monkeypatch, ready, fal): + fal.statuses = [{"status": "COMPLETED", "error": "Content policy violation", "error_type": "content_policy"}] + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + job = _activity(library, job_id) + assert job["status"] == "error" + assert job["error_message"] == "fal.ai could not generate this: Content policy violation" + # Checked before fetching a result that is not there. + assert fal.made("GET", host="queue.fal.run", suffix="/req-1") == [] + assert _generated(session) == [] + + +def test_a_rejected_key_is_named_as_one(library, session, monkeypatch, ready, fal): + fal.submit_response = httpx.Response(401, json={"detail": "Invalid key"}) + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + job = _activity(library, job_id) + assert job["status"] == "error" + assert job["error_message"] == "fal.ai rejected the API key — check it in Settings" + + +def test_a_validation_error_names_the_field(library, session, monkeypatch, ready, fal): + fal.submit_response = httpx.Response( + 422, + json={"detail": [{"loc": ["body", "image_urls"], "msg": "field required", "type": "missing"}]}, + ) + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + assert _activity(library, job_id)["error_message"] == ( + "fal.ai refused the request: image_urls: field required" + ) + + +def test_an_error_on_the_result_fetch_is_reported(library, session, monkeypatch, ready, fal): + fal.result_response = httpx.Response( + 422, json={"detail": [{"loc": ["body", "prompt"], "msg": "too long", "type": "value_error"}]} + ) + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + assert _activity(library, job_id)["error_message"] == "fal.ai refused the request: prompt: too long" + + +# ─── the download ──────────────────────────────────────────────────────────── + + +def test_a_result_url_pointing_inward_is_not_fetched(library, session, monkeypatch, ready, fal): + fal.result_body = {"images": [{"url": "https://metadata.internal/latest/x.png"}]} + fal.addresses["metadata.internal"] = "169.254.169.254" + + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + job = _activity(library, job_id) + assert job["status"] == "error" + assert job["error_message"].startswith("The generated file is at an address this app will not fetch") + assert fal.downloads() == [] + + +def test_a_redirect_inward_is_not_followed(library, session, monkeypatch, ready, fal): + fal.files[f"{CDN}/out-0.png"] = httpx.Response(302, headers={"location": "https://10.0.0.5/x.png"}) + + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + assert "will not fetch" in _activity(library, job_id)["error_message"] + assert [r.url.host for r in fal.downloads()] == ["v3.fal.media"] + + +def test_a_public_redirect_is_followed(library, session, monkeypatch, ready, fal): + fal.files[f"{CDN}/out-0.png"] = httpx.Response(302, headers={"location": "/files/moved.png"}) + fal.files[f"{CDN}/moved.png"] = FIXTURES / "sample_image.png" + + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + assert _activity(library, job_id)["status"] == "done" + + +def test_redirects_are_followed_at_most_three_times(library, session, monkeypatch, ready, fal): + for hop in range(4): + fal.files[f"{CDN}/hop-{hop}.png"] = httpx.Response( + 302, headers={"location": f"{CDN}/hop-{hop + 1}.png"} + ) + fal.files[f"{CDN}/hop-4.png"] = FIXTURES / "sample_image.png" + fal.result_body = {"images": [{"url": f"{CDN}/hop-0.png"}]} + + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + assert _activity(library, job_id)["error_message"] == ( + "Could not download the generated file (too many redirects)" + ) + assert len(fal.downloads()) == 4 + + +def test_an_inline_result_is_saved_without_a_download(library, session, monkeypatch, ready, fal): + encoded = base64.b64encode((FIXTURES / "sample_image.jpg").read_bytes()).decode() + fal.result_body = {"images": [{"url": f"data:image/jpeg;base64,{encoded}"}]} + + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + assert _activity(library, job_id)["status"] == "done" + assert fal.downloads() == [] + (asset,) = _generated(session) + assert asset.file_format == "jpg" + + +def test_an_oversized_file_is_refused_from_its_content_length(library, session, monkeypatch, ready, fal): + monkeypatch.setattr(settings, "generation_max_download_mb", 1) + fal.files[f"{CDN}/out-0.png"] = httpx.Response( + 200, content=b"\0" * (2 * 1024 * 1024), headers={"content-type": "image/png"} + ) + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + job = _activity(library, job_id) + assert job["error_message"] == "The generated file is larger than the 1 MB limit" + assert _generated(session) == [] + + +def test_an_oversized_file_without_a_length_is_stopped_while_streaming( + library, session, monkeypatch, ready, fal +): + monkeypatch.setattr(settings, "generation_max_download_mb", 1) + fal.files[f"{CDN}/out-0.png"] = httpx.Response( + 200, + stream=httpx.ByteStream(b"\0" * (2 * 1024 * 1024 + 1)), + headers={"content-type": "image/png"}, + ) + job_id = _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + assert _activity(library, job_id)["error_message"] == "The generated file is larger than the 1 MB limit" + assert _generated(session) == [] + + +def test_the_extension_comes_from_the_content_type_when_the_url_has_none( + library, session, monkeypatch, ready, fal +): + fal.result_body = {"images": [{"url": f"{CDN}/abc123"}]} + fal.files[f"{CDN}/abc123"] = httpx.Response( + 200, content=(FIXTURES / "sample_image.jpg").read_bytes(), headers={"content-type": "image/jpeg"} + ) + _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + (asset,) = _generated(session) + assert asset.file_format == "jpg" + + +# ─── the cost ──────────────────────────────────────────────────────────────── + + +def test_without_the_pricing_api_the_catalogue_price_is_an_estimate( + library, session, monkeypatch, ready, fal +): + fal.pricing_status = 500 + fal.result_headers = {"x-fal-billable-units": "2", "x-fal-request-id": "fal-req-1"} + _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + (usage,) = _usage(session) + assert usage.units == 2 + assert usage.unit_type == "megapixels" + assert usage.cost == pytest.approx(0.05) # 2 × the catalogue's 0.025 + assert usage.cost_estimated is True + + +@needs_ffmpeg +def test_without_the_header_the_units_are_worked_out_from_the_files( + library, session, monkeypatch, ready, fal +): + fal.result_headers = {"x-fal-request-id": "fal-req-9"} + fal.result_body = {"images": [{"url": f"{CDN}/big.jpg"}]} + fal.files[f"{CDN}/big.jpg"] = FIXTURES / "sample_image.jpg" # 800×600: one megapixel + _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + (usage,) = _usage(session) + assert usage.units == 1 + assert usage.unit_type == "megapixels" + assert usage.cost == pytest.approx(0.025) + assert usage.cost_estimated is True + assert usage.external_ref == "fal-req-9" + + +def test_with_no_price_anywhere_the_cost_is_left_unknown(library, session, monkeypatch, ready, fal): + fal.pricing_status = 404 + ready[T2I].unit_price = None + session.add(ready[T2I]) + session.commit() + _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + (usage,) = _usage(session) + assert usage.units == 1 + assert usage.cost is None + assert usage.currency is None + assert usage.cost_estimated is None + + +def test_a_total_mixing_a_bill_with_an_estimate_is_an_estimate(library, session, monkeypatch, ready, fal): + _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + assert library.get("/api/usage").json()["data"]["totals"]["estimated"] is False + + session.add(UsageEvent( + user_id=TEST_USER, kind="ai", provider="anthropic", model="claude", units=100, + unit_type="tokens", cost=0.01, currency="USD", cost_estimated=True, + )) + session.commit() + assert library.get("/api/usage").json()["data"]["totals"]["estimated"] is True + + +def test_the_price_is_asked_for_with_the_users_key_and_cached(library, session, monkeypatch, ready, fal): + for _ in range(2): + _generate(library, session, monkeypatch, {"model_id": ready[T2I].id, "prompt": PROMPT}) + + (lookup,) = fal.made("GET", host="api.fal.ai") + assert lookup.url.params["endpoint_id"] == T2I + assert lookup.headers["authorization"] == f"Key {FAL_KEY}" + assert len(_usage(session)) == 2 diff --git a/backend/tests/test_job_runner.py b/backend/tests/test_job_runner.py index 70a8efb..11ae899 100644 --- a/backend/tests/test_job_runner.py +++ b/backend/tests/test_job_runner.py @@ -33,6 +33,7 @@ class FakeJob(SQLModel, table=True): id: str = Field(default_factory=lambda: str(uuid.uuid4()), primary_key=True) user_id: str = Field(default="u1") + kind: str = Field(default="transcribe") status: str = Field(default="queued") stage: str = Field(default="") progress: int = Field(default=0) @@ -49,6 +50,9 @@ class FakeJob(SQLModel, table=True): # no business appearing in the application's schema. SQLModel.metadata.remove(FakeJob.__table__) +# Taken at import, before conftest's autouse fixture replaces it for every test. +from app.jobs.enrichment import start as REAL_ENRICHMENT_START # noqa: E402 + @pytest.fixture(name="job_engine") def job_engine_fixture(): @@ -282,6 +286,83 @@ def test_recovery_requeues_work_a_restart_interrupted(job_engine): assert read_job(job_engine, cancelled).status == "cancelled" +# ─── two queues over one table ─────────────────────────────────────────────── +# +# Generation (M8) has its own queue over the same table as everything else, so each +# queue must recover and sweep only its own rows — otherwise both requeue everything on +# start, and a video ends up on the transcription worker. + + +def _queued_ids(queue: JobQueue) -> set: + return set(queue._queue.queue) + + +def test_recovery_takes_only_the_kinds_a_queue_owns(job_engine): + generation = JobQueue(FakeJob, lambda _id: None, name="gen", engine=job_engine, kinds={"generate"}) + enrichment = JobQueue( + FakeJob, lambda _id: None, name="enr", engine=job_engine, exclude_kinds={"generate"} + ) + video = make_job(job_engine, kind="generate", status="processing") + transcript = make_job(job_engine, kind="transcribe", status="processing") + + generation.recover_pending() + assert _queued_ids(generation) == {video} + + enrichment.recover_pending() + assert _queued_ids(enrichment) == {transcript} + + +def test_an_unfiltered_queue_still_recovers_everything(job_engine): + queue = JobQueue(FakeJob, lambda _id: None, name="all", engine=job_engine) + ids = {make_job(job_engine, kind=kind, status="queued") for kind in ("generate", "transcribe")} + + queue.recover_pending() + assert _queued_ids(queue) == ids + + +def test_sweeping_ends_only_the_kinds_a_queue_owns(job_engine): + generation = JobQueue(FakeJob, lambda _id: None, name="gen", engine=job_engine, kinds={"generate"}) + stalled = {} + for kind in ("generate", "transcribe"): + job_id = make_job(job_engine, kind=kind, status="processing") + with Session(job_engine) as session: + row = session.get(FakeJob, job_id) + row.updated_at = utcnow() - timedelta(hours=5) + session.add(row) + session.commit() + stalled[kind] = job_id + + assert generation.sweep_stale() == 1 + assert read_job(job_engine, stalled["generate"]).status == "error" + assert generation.is_cancelled(stalled["generate"]) + # The other queue's row is that queue's to sweep. + assert read_job(job_engine, stalled["transcribe"]).status == "processing" + assert not generation.is_cancelled(stalled["transcribe"]) + + +def test_the_app_routes_generation_to_its_own_queue(): + from app.jobs import enrichment + from app.models.job import KIND_GENERATE, KIND_TRANSCRIBE + + assert enrichment.queue_for(KIND_GENERATE) is enrichment.generation_queue() + assert enrichment.queue_for(KIND_TRANSCRIBE) is enrichment.queue() + assert enrichment.generation_queue().kinds == {KIND_GENERATE} + assert enrichment.queue().exclude_kinds == {KIND_GENERATE} + + +def test_one_start_starts_both_queues(monkeypatch): + """conftest stubs `enrichment.start` to keep worker threads out of the suite. That + holds only while it is the one place every queue is started from.""" + from app.jobs import enrichment + + started = [] + monkeypatch.setattr(enrichment.queue(), "start", lambda: started.append("enrichment")) + monkeypatch.setattr(enrichment.generation_queue(), "start", lambda: started.append("generation")) + + REAL_ENRICHMENT_START() + assert started == ["enrichment", "generation"] + + # ─── error rendering ───────────────────────────────────────────────────────── diff --git a/backend/tests/test_migrations.py b/backend/tests/test_migrations.py index 27b9c84..93b9478 100644 --- a/backend/tests/test_migrations.py +++ b/backend/tests/test_migrations.py @@ -215,6 +215,92 @@ def test_add_clip_columns_survives_real_foreign_key_references(alembic_config): assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] +def test_the_generation_catalogue_is_seeded(alembic_config): + """M8 ships with three models per kind, so a fresh install can generate before + anybody has been made an admin. Each seeded row must also pass the catalogue's own + validation: a seed the admin editor would refuse to save is a seed that sends fal + something it refuses.""" + import json + + from app.generation import catalogue + + config, db_path = alembic_config + command.upgrade(config, "head") + + with sqlite3.connect(db_path) as conn: + conn.row_factory = sqlite3.Row + rows = [dict(r) for r in conn.execute("SELECT * FROM generationmodel")] + + by_endpoint = {row["endpoint_id"]: row for row in rows} + assert set(by_endpoint) == { + "fal-ai/flux/schnell", + "fal-ai/flux/dev", + "fal-ai/nano-banana-2", + "fal-ai/nano-banana/edit", + "fal-ai/bytedance/seedream/v4.5/edit", + "fal-ai/flux-pro/kontext", + "fal-ai/kling-video/v2.5-turbo/pro/image-to-video", + "fal-ai/minimax/hailuo-02/standard/image-to-video", + "fal-ai/veo3.1/fast/image-to-video", + } + kinds = [row["kind"] for row in rows] + assert {kind: kinds.count(kind) for kind in set(kinds)} == { + "text_to_image": 3, + "image_to_image": 3, + "image_to_video": 3, + } + + for row in rows: + entry = { + **row, + "image_field_is_list": bool(row["image_field_is_list"]), + "is_active": bool(row["is_active"]), + "options": json.loads(row["options"]), + "extra_params": json.loads(row["extra_params"]), + } + catalogue.validate(entry) + assert row["is_active"] == 1 + assert row["price_currency"] == "USD" + assert row["unit_price"] > 0 + + seedream = by_endpoint["fal-ai/bytedance/seedream/v4.5/edit"] + assert seedream["max_images"] == 10 and seedream["image_field"] == "image_urls" + kling = by_endpoint["fal-ai/kling-video/v2.5-turbo/pro/image-to-video"] + assert kling["end_image_field"] == "tail_image_url" + assert json.loads(kling["options"])["durations"] == ["5", "10"] + veo = by_endpoint["fal-ai/veo3.1/fast/image-to-video"] + assert json.loads(veo["extra_params"]) == {"generate_audio": False} + assert json.loads(veo["options"])["durations"] == ["4s", "6s", "8s"] + + +def test_the_generation_downgrade_survives_real_foreign_key_references(alembic_config): + """Dropping the `ai_*` columns recreates `asset`, the operation `7d4b9c1a6f28` found + SQLite refuses while `assettag` rows reference it. Proven against a populated table + for the reason that test gives: an empty one cannot fail this way.""" + config, db_path = alembic_config + command.upgrade(config, "head") + + with sqlite3.connect(db_path) as conn: + conn.execute( + "INSERT INTO asset (id, user_id, name, asset_type, source, storage_key, size_bytes," + " field_provenance, upload_date, modified_date, metadata_modified_date, ai_model)" + " VALUES ('a1', 'u', 'Fox', 'image', 'ai_generated', 'k1', 100, '{}'," + " '2026-01-01', '2026-01-01', '2026-01-01', 'fal-ai/flux/dev')" + ) + conn.execute("INSERT INTO tag (id, user_id, name, created_at) VALUES ('t1', 'u', 'Fox', '2026-01-01')") + conn.execute("INSERT INTO assettag (asset_id, tag_id, created_at) VALUES ('a1', 't1', '2026-01-01')") + conn.commit() + + command.downgrade(config, "9c2e08b4a1f7") + + with sqlite3.connect(db_path) as conn: + columns = {row[1] for row in conn.execute("PRAGMA table_info(asset)")} + assert not any(name.startswith("ai_") for name in columns) + assert conn.execute("SELECT id FROM asset WHERE id = 'a1'").fetchone() is not None + assert conn.execute("SELECT * FROM assettag WHERE asset_id = 'a1'").fetchone() is not None + assert "generationmodel" not in _tables(db_path) + + def test_asset_fts_rebuild_preserves_the_existing_index(alembic_config): """`9c2e08b4a1f7` drops and recreates asset_fts, which throws away its contents. diff --git a/docker-compose.yml b/docker-compose.yml index 685458c..50f6e9f 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -20,6 +20,15 @@ services: - URL_IMPORT_MAX_HEIGHT=${URL_IMPORT_MAX_HEIGHT:-1080} - URL_IMPORT_MAX_PLAYLIST_ITEMS=${URL_IMPORT_MAX_PLAYLIST_ITEMS:-200} - URL_IMPORT_COOKIES_FILE=${URL_IMPORT_COOKIES_FILE:-} + - ADMIN_USERS=${ADMIN_USERS:-} + - GENERATION_CONCURRENCY=${GENERATION_CONCURRENCY:-2} + - GENERATION_TIMEOUT_MINUTES=${GENERATION_TIMEOUT_MINUTES:-30} + - GENERATION_POLL_SECONDS=${GENERATION_POLL_SECONDS:-3} + - GENERATION_MAX_DOWNLOAD_MB=${GENERATION_MAX_DOWNLOAD_MB:-1024} + - GENERATION_BASE_MAX_EDGE=${GENERATION_BASE_MAX_EDGE:-2048} + - GENERATION_MAX_DATA_URI_MB=${GENERATION_MAX_DATA_URI_MB:-8} + - FAL_QUEUE_BASE_URL=${FAL_QUEUE_BASE_URL:-https://queue.fal.run} + - FAL_API_BASE_URL=${FAL_API_BASE_URL:-https://api.fal.ai} # Never published to the host: the frontend proxies it, and nothing else should # reach it directly — ProxyHeadersMiddleware trusts X-Forwarded-* from any peer. expose: diff --git a/docs/deployment.md b/docs/deployment.md index f37b4f4..6212fbe 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -135,6 +135,27 @@ Changing the model later does not invalidate what is stored — vectors record t that produced them and only matching ones are searched — so a switch quietly shrinks the searchable set until the affected assets are embedded again. +## Administrators + +Gecko Notes' token says who someone is, not whether they are an admin, so GAM is told +in `.env`: + +```env +ADMIN_USERS=davior # Notes usernames or user ids, comma-separated +``` + +Usernames match case-insensitively; ids exactly. An admin is the only one who can edit +the AI generation model catalogue (Settings → Generation); everyone else uses it as it +stands. Left empty, nobody can edit it and the catalogue the migration seeded is what +everyone gets — which works, until fal retires one of those endpoints. + +## AI generation + +Per user, like transcription: each person adds their own fal.ai key at **Settings → +Generation**, and fal bills that account. Nothing to configure at deploy time beyond +`ADMIN_USERS` above. The `GENERATION_*` settings in `.env.example` tune concurrency, +the timeout and the size limits; the defaults suit one host. + ## Importing from YouTube Nothing to configure unless YouTube refuses the server. Many VPS and datacenter IP ranges From abbc74069e6512fafe840ccd72e8cb52c5309490 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 13:38:33 +0000 Subject: [PATCH 5/7] =?UTF-8?q?feat(frontend):=20M8=20=E2=80=94=20generate?= =?UTF-8?q?=20images=20and=20video=20with=20fal.ai?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One GenerateForm behind three entry points, built to the API section of docs/m8-ai-generation.md: - api/generate.ts: the catalogue and generation types beside generateApi (models, createModel, updateModel, deleteModel, start, regenerate). Option values stay string | number end to end, because fal's endpoints disagree on the type of a duration and the server checks verbatim. - api/settings.ts gains generationSettingsApi for the fal key; Asset gains `generation`, User `is_admin`, ActivityJob `result_asset_ids`. - stores/generation.ts: the active catalogue, loaded once, staleness-guarded, reset on sign-out. - GenerateForm: kind follows the bases (none → text → image; one or two → image → image or image → video with the second as the end frame), models filtered to what can take these bases, every option control shown only when the row declares it, fal_key_missing as a Settings signpost. - Entry points: a third section in the add panel; "Generate from these" in the selection bar when every pick is an image with a file (bases in pick order); a Generate tab leading with provenance, Regenerate, Regenerate with the same seed, and Edit and generate (the form, pre-filled). - LibraryView adds every output of a finished generation to the grid. - Settings: the fal key, and the catalogue — read-only, or full CRUD for admins with lists that round-trip their values' types. - "AI cost (est.)" and the usage panel's "not a bill" now follow `estimated`, since fal's billing header makes an exact cost possible. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01TPmZjuM4q96DFRfU9cHL33 --- frontend/src/api/assets.ts | 6 + frontend/src/api/auth.ts | 4 + frontend/src/api/generate.ts | 222 +++++ frontend/src/api/settings.ts | 22 + frontend/src/api/transcripts.ts | 9 +- .../src/components/ActivityIndicator.test.tsx | 12 + frontend/src/components/ActivityIndicator.tsx | 1 + frontend/src/components/AddPanel.tsx | 13 +- frontend/src/components/AppShell.test.tsx | 2 +- frontend/src/components/AssetDetail.tsx | 19 +- frontend/src/components/EmbedButton.test.tsx | 1 + .../src/components/EnrichmentButton.test.tsx | 1 + frontend/src/components/GenerateForm.test.tsx | 490 +++++++++++ frontend/src/components/GenerateForm.tsx | 503 ++++++++++++ frontend/src/components/GenerateTab.tsx | 323 ++++++++ .../src/components/GenerationSettings.tsx | 768 ++++++++++++++++++ frontend/src/components/RequireAuth.test.tsx | 5 +- frontend/src/components/SelectionBar.test.tsx | 81 +- frontend/src/components/SelectionBar.tsx | 53 +- .../src/components/SuggestionPanel.test.tsx | 1 + .../src/components/TranscriptPanel.test.tsx | 4 + frontend/src/components/UrlImport.test.tsx | 1 + frontend/src/components/UsagePanel.test.tsx | 11 + frontend/src/components/UsagePanel.tsx | 26 +- frontend/src/stores/activity.test.ts | 1 + frontend/src/stores/auth.test.ts | 15 +- frontend/src/stores/auth.ts | 4 + frontend/src/stores/generation.test.ts | 75 ++ frontend/src/stores/generation.ts | 63 ++ frontend/src/stores/library.ts | 4 +- frontend/src/test-fixtures.ts | 64 +- frontend/src/views/AssetView.test.tsx | 234 +++++- frontend/src/views/LibraryView.test.tsx | 117 +++ frontend/src/views/LibraryView.tsx | 34 +- frontend/src/views/SettingsView.test.tsx | 245 ++++++ frontend/src/views/SettingsView.tsx | 2 + 36 files changed, 3397 insertions(+), 39 deletions(-) create mode 100644 frontend/src/api/generate.ts create mode 100644 frontend/src/components/GenerateForm.test.tsx create mode 100644 frontend/src/components/GenerateForm.tsx create mode 100644 frontend/src/components/GenerateTab.tsx create mode 100644 frontend/src/components/GenerationSettings.tsx create mode 100644 frontend/src/stores/generation.test.ts create mode 100644 frontend/src/stores/generation.ts diff --git a/frontend/src/api/assets.ts b/frontend/src/api/assets.ts index 27c2a5a..6321c34 100644 --- a/frontend/src/api/assets.ts +++ b/frontend/src/api/assets.ts @@ -1,4 +1,5 @@ import client from '@/api/client' +import type { AssetGeneration } from '@/api/generate' import type { Tag } from '@/api/tags' import type { ActivityJob } from '@/api/transcripts' @@ -75,6 +76,11 @@ export interface Asset { credit: string attribution_inherited: string[] + /** M8. How a generated asset was made — enough to make it again. Null for anything + * fal did not produce. The bases are named inside it, not as `parent_asset_id`: a + * parent would lend this its attribution and put it in the base's Clip tab. */ + generation: AssetGeneration | null + upload_date: string modified_date: string metadata_modified_date: string diff --git a/frontend/src/api/auth.ts b/frontend/src/api/auth.ts index 9fcb514..c71a868 100644 --- a/frontend/src/api/auth.ts +++ b/frontend/src/api/auth.ts @@ -12,6 +12,10 @@ import { loadConfig } from '@/api/config' export interface User { id: string username: string + /** Set from GAM's own `ADMIN_USERS`, not from the Notes token, which carries no such + * claim. Gates editing the generation catalogue — and only the UI for it: the server + * checks again on every write. */ + is_admin: boolean } interface DataResponse { diff --git a/frontend/src/api/generate.ts b/frontend/src/api/generate.ts new file mode 100644 index 0000000..d3dfe0d --- /dev/null +++ b/frontend/src/api/generate.ts @@ -0,0 +1,222 @@ +import client from '@/api/client' +import type { Asset } from '@/api/assets' +import type { ActivityJob } from '@/api/transcripts' + +/** + * M8: making new media with fal.ai, over `routers/generate.py`. + * + * Two halves. The catalogue is a global, admin-managed table of fal endpoints — data + * rather than code, because fal renames and retires endpoint ids every few months — and + * each row describes its endpoint's dialect: which image field it takes, which choices + * it offers, and the exact values it accepts for each. Generating returns the queued job + * rather than an asset, the way a URL import does: an image finishes on the first or + * second poll of fal's queue, a video takes minutes, and either can make several assets. + */ + +export type GenerationKind = 'text_to_image' | 'image_to_image' | 'image_to_video' + +export const GENERATION_KINDS: GenerationKind[] = [ + 'text_to_image', + 'image_to_image', + 'image_to_video', +] + +export const KIND_LABELS: Record = { + text_to_image: 'Text → image', + image_to_image: 'Image → image', + image_to_video: 'Image → video', +} + +/** + * A choice exactly as the endpoint takes it. Not normalised to a string: fal's endpoints + * disagree on the type of the same idea — Kling wants `"5"`, Veo `"8s"`, Hailuo `6` — and + * the server checks a submitted value against the row's list verbatim, so a value that + * changed type on its way through the form would be refused as "not offered". + */ +export type OptionValue = string | number + +export interface GenerationOptions { + aspect_ratios: OptionValue[] + image_sizes: OptionValue[] + durations: OptionValue[] + resolutions: OptionValue[] + supports_seed: boolean + supports_negative_prompt: boolean + supports_audio: boolean + /** 1–4 for images; always 1 for video. */ + max_outputs: number +} + +export interface GenerationModel { + id: string + /** fal's own id, e.g. `fal-ai/flux/dev`. Unique across the catalogue. */ + endpoint_id: string + kind: GenerationKind + label: string + note: string + sort_order: number + is_active: boolean + /** The request field the base image goes in. Null for text → image. */ + image_field: string | null + image_field_is_list: boolean + /** 0 for text → image, at least 1 otherwise. */ + max_images: number + /** Image → video only, and only for endpoints that take a last frame. */ + end_image_field: string | null + options: GenerationOptions + /** Merged into every request body first — a catalogue-level default. */ + extra_params: Record + /** The list price, the fallback when fal's pricing API cannot be asked. */ + unit_price: number | null + price_unit: string | null + price_currency: string | null + created_at: string + updated_at: string +} + +/** What an admin writes. Everything but the server's own bookkeeping. */ +export type GenerationModelInput = Omit< + GenerationModel, + 'id' | 'created_at' | 'updated_at' +> + +/** + * Every user-facing choice, always sent in full. Null means "not chosen", which leaves the + * endpoint's (or the catalogue's) default in charge; the server refuses a non-null value + * the model does not declare, so the form only ever sets the ones it offered. + */ +export interface GenerationParams { + aspect_ratio: OptionValue | null + image_size: OptionValue | null + duration: OptionValue | null + resolution: OptionValue | null + seed: number | null + negative_prompt: string | null + generate_audio: boolean | null + num_outputs: number +} + +export interface GenerationCreate { + model_id: string + prompt: string + /** In order. For image → video, exactly one: the start frame. */ + base_asset_ids: string[] + end_frame_asset_id: string | null + params: GenerationParams +} + +export interface RegenerateRequest { + /** Null re-sends the stored prompt. */ + prompt: string | null + /** Send the seed fal reported last time, when the model takes one. */ + reuse_seed: boolean +} + +export interface GenerationSource { + asset_id: string + role: 'base' | 'end_frame' + /** A snapshot taken at generation time, so a deleted base still reads as something. */ + name: string +} + +/** What a generated asset remembers about how it was made. */ +export interface AssetGeneration { + /** The endpoint id, not a catalogue row id — rows come and go, endpoints are the record. */ + model: string + kind: GenerationKind + prompt: string + /** The request body as sent, minus the prompt and the images — catalogue defaults + * included, so it still describes the run after the catalogue changes. */ + parameters: Record + sources: GenerationSource[] + /** What fal reported back, when it reported one. */ + seed: number | null + generated_at: string +} + +interface DataResponse { + data: T +} + +interface ListResponse { + data: T[] + total: number +} + +export const generateApi = { + /** Active rows only, unless an admin asks for the rest — the server ignores the flag + * for everybody else. Ordered by kind, sort order, label. */ + models(includeInactive = false): Promise { + return client + .get>('/generate/models', { + params: { include_inactive: includeInactive }, + }) + .then((r) => r.data.data) + }, + + createModel(input: GenerationModelInput): Promise { + return client + .post>('/generate/models', input) + .then((r) => r.data.data) + }, + + /** A PATCH: only the fields present change, and an explicit null clears one. */ + updateModel( + id: string, + changes: Partial + ): Promise { + return client + .patch>(`/generate/models/${id}`, changes) + .then((r) => r.data.data) + }, + + deleteModel(id: string): Promise { + return client.delete(`/generate/models/${id}`).then(() => undefined) + }, + + start(request: GenerationCreate): Promise { + return client + .post>('/generate', request) + .then((r) => r.data.data) + }, + + regenerate(assetId: string, request: RegenerateRequest): Promise { + return client + .post>(`/generate/${assetId}/regenerate`, request) + .then((r) => r.data.data) + }, +} + +/** + * Whether an asset can be handed to fal as a base image. + * + * The server's rule is "an image that owns a file Pillow can read"; the last part it + * checks itself and explains when it refuses. A clip never qualifies — it has no bytes + * of its own, only a range of its parent's — and a missing file has nothing to send. + */ +export function canBeBase(asset: Asset): boolean { + return ( + asset.asset_type === 'image' && + asset.source !== 'clip' && + !asset.missing && + asset.file_url !== null + ) +} + +/** + * "≈ $0.025 per megapixel, list price". + * + * Not `formatCost`: that rounds anything over a cent to two places, and a per-unit list + * price of $0.025 shown as "$0.03" is a different price. + */ +export function formatListPrice(model: GenerationModel): string | null { + if (model.unit_price === null) return null + const currency = model.price_currency || 'USD' + const symbol = currency === 'USD' ? '$' : `${currency} ` + const amount = model.unit_price.toLocaleString('en-US', { + minimumFractionDigits: 2, + maximumFractionDigits: 4, + }) + const unit = model.price_unit ? ` per ${model.price_unit}` : '' + return `≈ ${symbol}${amount}${unit}, list price` +} diff --git a/frontend/src/api/settings.ts b/frontend/src/api/settings.ts index 4a00102..2b14c17 100644 --- a/frontend/src/api/settings.ts +++ b/frontend/src/api/settings.ts @@ -35,6 +35,28 @@ export const speechSettingsApi = { }, } +/** M8. Only the fal.ai key is per user; the model catalogue is global, in `api/generate`. */ +export interface GenerationSettings { + fal_key_configured: boolean +} + +export const generationSettingsApi = { + get(): Promise { + return client + .get>('/settings/generation') + .then((r) => r.data.data) + }, + + update(changes: { + /** '' clears the stored key; omit the field to leave it alone. */ + fal_api_key?: string + }): Promise { + return client + .put>('/settings/generation', changes) + .then((r) => r.data.data) + }, +} + export interface EmbeddingSettings { provider: string model: string diff --git a/frontend/src/api/transcripts.ts b/frontend/src/api/transcripts.ts index c22955f..b7b1dfc 100644 --- a/frontend/src/api/transcripts.ts +++ b/frontend/src/api/transcripts.ts @@ -40,10 +40,13 @@ export interface ActivityJob { asset_id: string | null asset_name: string model: string - /** M7 only: set once a `extract_subvideo` "extract" job finishes, to the asset it - * created. `asset_id` stays pointed at the source throughout, so this is the only - * place to learn the result's id. */ + /** Set once a job that creates an asset finishes — M7's `extract_subvideo`, a URL + * import, a generation — to the asset it created. `asset_id` stays pointed at the + * source throughout, so this is the only place to learn the result's id. */ result_asset_id: string | null + /** Every asset the job created, in order — a generation can make up to four. + * `result_asset_id` stays the first of them. */ + result_asset_ids: string[] error_message: string | null created_at: string updated_at: string diff --git a/frontend/src/components/ActivityIndicator.test.tsx b/frontend/src/components/ActivityIndicator.test.tsx index 36b0e6c..a541f22 100644 --- a/frontend/src/components/ActivityIndicator.test.tsx +++ b/frontend/src/components/ActivityIndicator.test.tsx @@ -20,6 +20,7 @@ function job(overrides: Partial = {}): ActivityJob { asset_name: 'Interview', model: 'nova-3', result_asset_id: null, + result_asset_ids: [], error_message: null, created_at: '2026-09-14T10:00:00Z', updated_at: '2026-09-14T10:00:00Z', @@ -77,6 +78,17 @@ describe('ActivityIndicator', () => { expect(screen.getByText('Giordano lecture')).toBeInTheDocument() }) + it('names a generation by its prompt', async () => { + renderWith([ + job({ action: 'generate', asset_id: null, asset_name: 'a red fox in snow' }), + ]) + + await userEvent.click(screen.getByRole('button', { name: /1 job running/i })) + + expect(screen.getByText('Generating')).toBeInTheDocument() + expect(screen.getByText('a red fox in snow')).toBeInTheDocument() + }) + it('keeps a failed job visible with its reason', async () => { renderWith([job({ status: 'error', error_message: 'provider says no' })]) diff --git a/frontend/src/components/ActivityIndicator.tsx b/frontend/src/components/ActivityIndicator.tsx index afeb29e..7324432 100644 --- a/frontend/src/components/ActivityIndicator.tsx +++ b/frontend/src/components/ActivityIndicator.tsx @@ -20,6 +20,7 @@ const LABELS: Record = { autotag: 'Suggesting tags', extract_text: 'Reading text', import_url: 'Importing', + generate: 'Generating', } function subject(job: ActivityJob): string { diff --git a/frontend/src/components/AddPanel.tsx b/frontend/src/components/AddPanel.tsx index 0539ae7..908dec5 100644 --- a/frontend/src/components/AddPanel.tsx +++ b/frontend/src/components/AddPanel.tsx @@ -1,4 +1,5 @@ import { X } from 'lucide-react' +import GenerateForm from '@/components/GenerateForm' import UploadZone from '@/components/UploadZone' import UrlImport from '@/components/UrlImport' @@ -10,12 +11,15 @@ interface Props { } /** - * The two ways into the library — files and links — folded behind the library's + button. + * The three ways into the library — files, links, and M8's text → image generation — + * folded behind the library's + button. Generating from existing images starts from + * those images instead (the selection bar, an asset's Generate tab), since that is where + * they are picked. * * Hidden rather than unmounted when closed. Unmounting would throw away a half-pasted - * link and the import options with it, and would take an upload's progress bar off the - * page while the upload itself carried on in the store; closing the panel is putting - * it out of the way, not cancelling anything. + * link or a half-written prompt and the options with them, and would take an upload's + * progress bar off the page while the upload itself carried on in the store; closing the + * panel is putting it out of the way, not cancelling anything. * * The `hidden` attribute goes on an element with no display utility of its own: * Tailwind's `[hidden]` rule sits in the base layer, so a `flex` or `grid` class on the @@ -51,6 +55,7 @@ export default function AddPanel({ open, onClose }: Props) {
+
) diff --git a/frontend/src/components/AppShell.test.tsx b/frontend/src/components/AppShell.test.tsx index 93912c5..7ceb465 100644 --- a/frontend/src/components/AppShell.test.tsx +++ b/frontend/src/components/AppShell.test.tsx @@ -36,7 +36,7 @@ describe('AppShell', () => { // `signOut` has existed since M2 and was called by nothing — there was no way out // of a session short of clearing localStorage by hand. useAuthStore.setState({ - user: { id: 'u1', username: 'tester' }, + user: { id: 'u1', username: 'tester', is_admin: false }, status: 'authenticated', }) const signOut = vi.fn() diff --git a/frontend/src/components/AssetDetail.tsx b/frontend/src/components/AssetDetail.tsx index b868627..5722d9c 100644 --- a/frontend/src/components/AssetDetail.tsx +++ b/frontend/src/components/AssetDetail.tsx @@ -12,11 +12,13 @@ import { Tags as TagsIcon, Trash2, Wand2, + WandSparkles, X, } from 'lucide-react' import type { Asset, AssetUpdate } from '@/api/assets' import { tagsApi } from '@/api/tags' import { enrichmentApi } from '@/api/enrichment' +import { canBeBase } from '@/api/generate' import { clipsApi } from '@/api/clips' import { activityApi } from '@/api/transcripts' import { formatCost, usageApi, type UsageTotals } from '@/api/usage' @@ -31,6 +33,7 @@ import ClipEditor from '@/components/ClipEditor' import DocumentTextPanel from '@/components/DocumentTextPanel' import EmbedButton from '@/components/EmbedButton' import EnrichmentButton from '@/components/EnrichmentButton' +import GenerateTab from '@/components/GenerateTab' import SuggestionPanel from '@/components/SuggestionPanel' import TagInput from '@/components/TagInput' import Tabs, { type TabSpec } from '@/components/Tabs' @@ -540,7 +543,9 @@ export default function AssetDetail({ {usage && usage.total_events > 0 && ( 0 ? formatCost(usage.cost, usage.currency) @@ -678,6 +683,18 @@ export default function AssetDetail({ }, ] : []), + // M8. An image that can be a base, or anything fal made — a generated video has no + // form here but still has its provenance and its Regenerate. + ...(canBeBase(asset) || asset.generation + ? [ + { + id: 'generate', + label: 'Generate', + icon: WandSparkles, + content: , + }, + ] + : []), { id: 'attribution', label: 'Source', diff --git a/frontend/src/components/EmbedButton.test.tsx b/frontend/src/components/EmbedButton.test.tsx index 88c125d..6468f45 100644 --- a/frontend/src/components/EmbedButton.test.tsx +++ b/frontend/src/components/EmbedButton.test.tsx @@ -22,6 +22,7 @@ function job(overrides: Partial = {}): ActivityJob { asset_name: 'Interview', model: 'text-embedding-3-small', result_asset_id: null, + result_asset_ids: [], error_message: null, created_at: '2026-09-14T10:00:00Z', updated_at: '2026-09-14T10:00:00Z', diff --git a/frontend/src/components/EnrichmentButton.test.tsx b/frontend/src/components/EnrichmentButton.test.tsx index cfe8a94..357c093 100644 --- a/frontend/src/components/EnrichmentButton.test.tsx +++ b/frontend/src/components/EnrichmentButton.test.tsx @@ -25,6 +25,7 @@ function job(overrides: Partial = {}): ActivityJob { asset_name: 'clip.mp4', model: 'claude-sonnet-4-20250514', result_asset_id: null, + result_asset_ids: [], error_message: null, created_at: '2026-09-15T06:00:00Z', updated_at: '2026-09-15T06:00:00Z', diff --git a/frontend/src/components/GenerateForm.test.tsx b/frontend/src/components/GenerateForm.test.tsx new file mode 100644 index 0000000..3c92fb2 --- /dev/null +++ b/frontend/src/components/GenerateForm.test.tsx @@ -0,0 +1,490 @@ +import { render, screen, waitFor, within } from '@testing-library/react' +import userEvent from '@testing-library/user-event' +import { MemoryRouter } from 'react-router-dom' +import { AxiosError } from 'axios' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import GenerateForm, { type GeneratePreset } from '@/components/GenerateForm' +import type { Asset } from '@/api/assets' +import { generateApi, type GenerationModel } from '@/api/generate' +import { activityApi, type ActivityJob } from '@/api/transcripts' +import { useActivityStore } from '@/stores/activity' +import { useGenerationStore } from '@/stores/generation' +import { makeGenerationModel, noAttribution } from '@/test-fixtures' + +function image(id: string, name = id): Asset { + return { + id, + name, + description: null, + summary: null, + ...noAttribution, + asset_type: 'image', + source: 'local_upload', + parent_asset_id: null, + in_point: null, + out_point: null, + original_name: `${id}.jpg`, + mime_type: 'image/jpeg', + file_format: 'jpg', + size_bytes: 100, + duration_seconds: null, + width: 800, + height: 600, + codec: null, + file_url: `/media/u/${id}.jpg`, + thumb_url: `/media/u/${id}.thumb.jpg`, + missing: false, + tags: [], + upload_date: '2026-10-01T00:00:00Z', + modified_date: '2026-10-01T00:00:00Z', + metadata_modified_date: '2026-10-01T00:00:00Z', + } +} + +function queuedJob(overrides: Partial = {}): ActivityJob { + return { + id: 'g1', + kind: 'enrichment', + action: 'generate', + status: 'queued', + stalled: false, + stage: 'Queued', + progress: 0, + detail: '', + asset_id: null, + asset_name: 'a red fox in snow', + model: 'fal-ai/flux/dev', + result_asset_id: null, + result_asset_ids: [], + error_message: null, + created_at: '2026-10-07T10:00:00Z', + updated_at: '2026-10-07T10:00:00Z', + ...overrides, + } +} + +function codedError(code: string, message: string) { + const error = new AxiosError('failed') + // @ts-expect-error - a minimal response is all apiErrorCode and apiErrorMessage read + error.response = { data: { detail: { code, message } } } + return error +} + +const SCHNELL = makeGenerationModel({ + id: 'schnell', + endpoint_id: 'fal-ai/flux/schnell', + label: 'FLUX schnell', + sort_order: 20, +}) +const DEV = makeGenerationModel({ + id: 'dev', + endpoint_id: 'fal-ai/flux/dev', + label: 'FLUX dev', + sort_order: 10, + note: 'Good with text', + unit_price: 0.025, + price_unit: 'megapixel', + price_currency: 'USD', + options: { image_sizes: ['square_hd', 'landscape_16_9'], supports_seed: true }, +}) +const EDIT_ONE = makeGenerationModel({ + id: 'kontext', + endpoint_id: 'fal-ai/flux-pro/kontext', + kind: 'image_to_image', + label: 'Kontext', + image_field: 'image_url', + max_images: 1, +}) +const EDIT_MANY = makeGenerationModel({ + id: 'seedream', + endpoint_id: 'fal-ai/bytedance/seedream/v4.5/edit', + kind: 'image_to_image', + label: 'Seedream', + sort_order: 5, + image_field: 'image_urls', + image_field_is_list: true, + max_images: 10, +}) +const KLING = makeGenerationModel({ + id: 'kling', + endpoint_id: 'fal-ai/kling-video/v2.5-turbo/pro/image-to-video', + kind: 'image_to_video', + label: 'Kling', + image_field: 'image_url', + max_images: 1, + end_image_field: 'tail_image_url', + options: { durations: ['5', '10'], supports_negative_prompt: true }, +}) +const VEO = makeGenerationModel({ + id: 'veo', + endpoint_id: 'fal-ai/veo3.1/fast/image-to-video', + kind: 'image_to_video', + label: 'Veo', + image_field: 'image_url', + max_images: 1, + extra_params: { generate_audio: false }, + options: { durations: ['8s'], supports_audio: true }, +}) + +const CATALOGUE = [SCHNELL, DEV, EDIT_ONE, EDIT_MANY, KLING, VEO] + +function renderForm( + props: { bases?: Asset[]; preset?: GeneratePreset } = {}, + catalogue: GenerationModel[] = CATALOGUE +) { + useGenerationStore.setState({ models: catalogue, loaded: true }) + return render( + + + + ) +} + +const form = () => screen.getByRole('form', { name: 'Generate' }) +const modelNames = () => + within(screen.getByLabelText('Model')) + .getAllByRole('option') + .map((option) => option.textContent) + +beforeEach(() => { + // Here rather than in afterEach: the form subscribes to both, and resetting them while + // the last test's form is still mounted is a render outside act(). + useActivityStore.getState().reset() + useGenerationStore.getState().reset() + // Mounting asks for the catalogue; `renderForm` seeds it, so this only guards against + // a test that forgot to reaching for a server that is not there. + vi.spyOn(generateApi, 'models').mockResolvedValue([]) +}) + +afterEach(() => { + vi.restoreAllMocks() +}) + +describe('GenerateForm: what it offers', () => { + it('makes text → image from nothing, and offers only those models', () => { + renderForm() + + expect(screen.queryByRole('radio')).toBeNull() + expect(modelNames()).toEqual(['FLUX dev', 'FLUX schnell']) + }) + + it('offers image → image or image → video for one base', async () => { + const user = userEvent.setup() + renderForm({ bases: [image('b1')] }) + + expect(screen.getByRole('radio', { name: 'Image → image' })).toBeChecked() + expect(screen.getByRole('radio', { name: 'Image → video' })).toBeInTheDocument() + expect(screen.queryByRole('radio', { name: 'Text → image' })).toBeNull() + expect(modelNames()).toEqual(['Kontext', 'Seedream']) + + await user.click(screen.getByRole('radio', { name: 'Image → video' })) + expect(modelNames()).toEqual(['Kling', 'Veo']) + }) + + it('stops offering video past a start and an end frame', () => { + renderForm({ bases: [image('b1'), image('b2'), image('b3')] }) + + expect(screen.queryByRole('radio', { name: 'Image → video' })).toBeNull() + }) + + it('leaves out an edit model that cannot take this many bases', () => { + // Kontext takes one image. Offering it for two is offering a 422. + renderForm({ bases: [image('b1'), image('b2')] }) + + expect(modelNames()).toEqual(['Seedream']) + }) + + it('says so when no model can take that many', () => { + renderForm({ bases: Array.from({ length: 11 }, (_, i) => image(`b${i}`)) }) + + expect(screen.getByText('No image → image model takes 11 base images.')).toBeVisible() + expect(screen.getByRole('button', { name: 'Generate' })).toBeDisabled() + }) + + it('offers only video models that take an end frame for two bases', async () => { + const user = userEvent.setup() + renderForm({ bases: [image('b1'), image('b2')] }) + + await user.click(screen.getByRole('radio', { name: 'Image → video' })) + + expect(modelNames()).toEqual(['Kling']) + }) + + it('shows the chosen model’s note and list price', () => { + renderForm() + + expect( + screen.getByText('Good with text · ≈ $0.025 per megapixel, list price') + ).toBeVisible() + }) +}) + +describe('GenerateForm: option controls', () => { + const ALL = makeGenerationModel({ + id: 'all', + label: 'Everything', + sort_order: 1, + options: { + aspect_ratios: ['16:9', '1:1'], + image_sizes: ['square_hd'], + durations: [6, 10], + resolutions: ['720p'], + supports_seed: true, + supports_negative_prompt: true, + supports_audio: true, + max_outputs: 4, + }, + }) + const NONE = makeGenerationModel({ id: 'none', label: 'Nothing', sort_order: 0 }) + + const CONTROLS = [ + 'Aspect ratio', + 'Image size', + 'Duration', + 'Resolution', + 'Seed', + 'Negative prompt', + 'Generate audio', + 'Number of outputs', + ] + + it('renders none that the model does not declare', () => { + renderForm({}, [NONE, ALL]) + + for (const label of CONTROLS) expect(screen.queryByLabelText(label)).toBeNull() + }) + + it('renders each one the model declares', async () => { + const user = userEvent.setup() + renderForm({}, [NONE, ALL]) + + await user.selectOptions(screen.getByLabelText('Model'), 'all') + + for (const label of CONTROLS) expect(screen.getByLabelText(label)).toBeVisible() + expect( + within(screen.getByLabelText('Number of outputs')) + .getAllByRole('option') + .map((o) => o.textContent) + ).toEqual(['1', '2', '3', '4']) + }) + + it('shows the catalogue’s audio default without sending it as a choice', async () => { + // Veo's row turns audio off because it raises the price by half. The box reflects + // that, and an untouched box sends null so the row's default stays in charge. + const user = userEvent.setup() + const start = vi.spyOn(generateApi, 'start').mockResolvedValue(queuedJob()) + vi.spyOn(activityApi, 'list').mockResolvedValue([]) + renderForm({ bases: [image('b1')] }) + + await user.click(screen.getByRole('radio', { name: 'Image → video' })) + await user.selectOptions(screen.getByLabelText('Model'), 'veo') + expect(screen.getByLabelText('Generate audio')).not.toBeChecked() + + await user.type(screen.getByLabelText('Prompt'), 'waves') + await user.click(screen.getByRole('button', { name: 'Generate' })) + + await waitFor(() => expect(start).toHaveBeenCalledTimes(1)) + expect(start.mock.calls[0][0].params.generate_audio).toBeNull() + }) +}) + +describe('GenerateForm: submitting', () => { + it('sends exactly what was chosen, and null for the rest', async () => { + const user = userEvent.setup() + const start = vi.spyOn(generateApi, 'start').mockResolvedValue(queuedJob()) + vi.spyOn(activityApi, 'list').mockResolvedValue([]) + renderForm() + + await user.type(screen.getByLabelText('Prompt'), ' a red fox in snow ') + await user.selectOptions(screen.getByLabelText('Image size'), 'landscape_16_9') + await user.type(screen.getByLabelText('Seed'), '42') + await user.click(screen.getByRole('button', { name: 'Generate' })) + + await waitFor(() => expect(start).toHaveBeenCalledTimes(1)) + expect(start.mock.calls[0][0]).toEqual({ + model_id: 'dev', + prompt: 'a red fox in snow', + base_asset_ids: [], + end_frame_asset_id: null, + params: { + aspect_ratio: null, + image_size: 'landscape_16_9', + duration: null, + resolution: null, + seed: 42, + negative_prompt: null, + generate_audio: null, + num_outputs: 1, + }, + }) + }) + + it('keeps a value the type the endpoint declared it as', async () => { + // Hailuo wants `6`, Kling wants `"5"`: the server compares against the row's list + // verbatim, so a number that came back as a string would be refused. + const user = userEvent.setup() + const hailuo = makeGenerationModel({ + id: 'hailuo', + kind: 'image_to_video', + label: 'Hailuo', + image_field: 'image_url', + max_images: 1, + options: { durations: [6, 10] }, + }) + const start = vi.spyOn(generateApi, 'start').mockResolvedValue(queuedJob()) + vi.spyOn(activityApi, 'list').mockResolvedValue([]) + renderForm({ bases: [image('b1')] }, [hailuo]) + + await user.click(screen.getByRole('radio', { name: 'Image → video' })) + await user.selectOptions(screen.getByLabelText('Duration'), '6') + await user.type(screen.getByLabelText('Prompt'), 'pan left') + await user.click(screen.getByRole('button', { name: 'Generate' })) + + await waitFor(() => expect(start).toHaveBeenCalledTimes(1)) + expect(start.mock.calls[0][0].params.duration).toBe(6) + }) + + it('sends every base, in order, for image → image', async () => { + const user = userEvent.setup() + const start = vi.spyOn(generateApi, 'start').mockResolvedValue(queuedJob()) + vi.spyOn(activityApi, 'list').mockResolvedValue([]) + renderForm({ bases: [image('b2'), image('b1')] }) + + await user.type(screen.getByLabelText('Prompt'), 'merge them') + await user.click(screen.getByRole('button', { name: 'Generate' })) + + await waitFor(() => expect(start).toHaveBeenCalledTimes(1)) + expect(start.mock.calls[0][0]).toMatchObject({ + model_id: 'seedream', + base_asset_ids: ['b2', 'b1'], + end_frame_asset_id: null, + }) + }) + + it('labels the bases as frames for video, and sends the second as the end frame', async () => { + const user = userEvent.setup() + const start = vi.spyOn(generateApi, 'start').mockResolvedValue(queuedJob()) + vi.spyOn(activityApi, 'list').mockResolvedValue([]) + renderForm({ bases: [image('first', 'Dawn'), image('last', 'Dusk')] }) + + const bases = within(screen.getByRole('list', { name: 'Base images' })) + expect(bases.getByText('Base 1')).toBeVisible() + + await user.click(screen.getByRole('radio', { name: 'Image → video' })) + + const items = bases.getAllByRole('listitem') + expect(within(items[0]).getByText('Start frame')).toBeVisible() + expect(within(items[0]).getByText('Dawn')).toBeVisible() + expect(within(items[1]).getByText('End frame')).toBeVisible() + expect(within(items[1]).getByText('Dusk')).toBeVisible() + + await user.type(screen.getByLabelText('Prompt'), 'day turns to night') + await user.click(screen.getByRole('button', { name: 'Generate' })) + + await waitFor(() => expect(start).toHaveBeenCalledTimes(1)) + expect(start.mock.calls[0][0]).toMatchObject({ + model_id: 'kling', + base_asset_ids: ['first'], + end_frame_asset_id: 'last', + }) + }) + + it('confirms, keeps the prompt, and tells the activity feed', async () => { + const user = userEvent.setup() + vi.spyOn(generateApi, 'start').mockResolvedValue(queuedJob()) + const poll = vi.spyOn(activityApi, 'list').mockResolvedValue([queuedJob()]) + renderForm() + + await user.type(screen.getByLabelText('Prompt'), 'a red fox in snow') + await user.click(screen.getByRole('button', { name: 'Generate' })) + + expect(await within(form()).findByRole('status')).toHaveTextContent( + 'Queued “a red fox in snow” — follow it under background activity.' + ) + // Kept: the next thing anyone does is adjust the wording and go again. + expect(screen.getByLabelText('Prompt')).toHaveValue('a red fox in snow') + await waitFor(() => expect(poll).toHaveBeenCalled()) + }) + + it('points at Settings when there is no fal.ai key', async () => { + const user = userEvent.setup() + vi.spyOn(generateApi, 'start').mockRejectedValue( + codedError('fal_key_missing', 'Add a fal.ai key in Settings first') + ) + renderForm() + + await user.type(screen.getByLabelText('Prompt'), 'a fox') + await user.click(screen.getByRole('button', { name: 'Generate' })) + + expect( + await screen.findByRole('link', { name: /add one in settings/i }) + ).toHaveAttribute('href', '/settings') + expect(screen.getByRole('alert')).toHaveTextContent(/needs a fal\.ai key/i) + }) + + it('shows any other refusal in the server’s own words', async () => { + const user = userEvent.setup() + vi.spyOn(generateApi, 'start').mockRejectedValue( + codedError('invalid_generation', 'The prompt is longer than 4,000 characters') + ) + renderForm() + + await user.type(screen.getByLabelText('Prompt'), 'a fox') + await user.click(screen.getByRole('button', { name: 'Generate' })) + + expect(await screen.findByRole('alert')).toHaveTextContent( + 'The prompt is longer than 4,000 characters' + ) + expect(screen.queryByRole('link')).toBeNull() + }) + + it('cannot be submitted without a prompt', () => { + renderForm() + + expect(screen.getByRole('button', { name: 'Generate' })).toBeDisabled() + }) +}) + +describe('GenerateForm: pre-filled', () => { + it('starts from a stored generation’s model, prompt and choices', async () => { + const user = userEvent.setup() + const start = vi.spyOn(generateApi, 'start').mockResolvedValue(queuedJob()) + vi.spyOn(activityApi, 'list').mockResolvedValue([]) + renderForm({ + bases: [image('b1')], + preset: { + model: KLING.endpoint_id, + kind: 'image_to_video', + prompt: 'the tide comes in', + parameters: { duration: '10', negative_prompt: 'blur', cfg_scale: 0.5 }, + }, + }) + + expect(screen.getByRole('radio', { name: 'Image → video' })).toBeChecked() + expect(screen.getByLabelText('Model')).toHaveValue('kling') + expect(screen.getByLabelText('Prompt')).toHaveValue('the tide comes in') + expect(screen.getByLabelText('Duration')).toHaveValue('10') + expect(screen.getByLabelText('Negative prompt')).toHaveValue('blur') + + await user.click(screen.getByRole('button', { name: 'Generate' })) + + await waitFor(() => expect(start).toHaveBeenCalledTimes(1)) + expect(start.mock.calls[0][0].params).toMatchObject({ + duration: '10', + negative_prompt: 'blur', + }) + }) + + it('says so when the model it was made with is no longer offered', () => { + renderForm({ + preset: { + model: 'fal-ai/imagen4/preview', + kind: 'text_to_image', + prompt: 'a fox', + parameters: {}, + }, + }) + + expect(screen.getByText(/fal-ai\/imagen4\/preview, which made this/)).toBeVisible() + }) +}) diff --git a/frontend/src/components/GenerateForm.tsx b/frontend/src/components/GenerateForm.tsx new file mode 100644 index 0000000..b97dcec --- /dev/null +++ b/frontend/src/components/GenerateForm.tsx @@ -0,0 +1,503 @@ +import { useEffect, useId, useMemo, useState, type FormEvent } from 'react' +import { Link } from 'react-router-dom' +import { Loader2, Sparkles } from 'lucide-react' +import type { Asset } from '@/api/assets' +import { apiErrorCode, apiErrorMessage } from '@/api/client' +import { + formatListPrice, + generateApi, + KIND_LABELS, + type GenerationCreate, + type GenerationKind, + type GenerationModel, + type GenerationOptions, + type OptionValue, +} from '@/api/generate' +import AssetThumb from '@/components/AssetThumb' +import { useActivityStore } from '@/stores/activity' +import { useGenerationStore } from '@/stores/generation' + +/** + * One form for every way into fal.ai — the add panel, a selection, an asset's own tab. + * + * What it can make follows from what it was handed. No bases is text → image. One or two + * is image → image or image → video, where a second base can only be the video's end + * frame; three or more is image → image alone. The model list is then the catalogue's + * rows of that kind that can actually take these bases, and every option control below + * it renders only when the chosen row declares it — the server refuses anything else, so + * offering it would only be offering a 422. + * + * Like `UrlImport`, it owns no progress of its own: submitting queues a job, says so, and + * leaves the rest to the activity indicator and to `LibraryView`, which puts the results + * in the grid when the job finishes. + */ + +/** "Edit and generate": a stored generation, to start the form from. */ +export interface GeneratePreset { + /** The endpoint id, which is what a generated asset records — catalogue rows come + * and go, so it is matched by endpoint rather than by row id. */ + model: string + kind: GenerationKind + prompt: string + /** The request body as it was sent. Only the keys this form has controls for are + * read back; the rest are catalogue defaults that will be merged in again anyway. */ + parameters: Record +} + +interface Props { + /** Says what this form is for, on screen and as its accessible name — the library + * page can have three of these mounted at once. */ + title: string + /** The base images, in order. The order matters: for a video the first is the start + * frame and the second the end frame. */ + bases?: Asset[] + preset?: GeneratePreset + onCancel?: () => void +} + +type ChoiceKey = 'aspect_ratio' | 'image_size' | 'duration' | 'resolution' + +/** The four "pick one of the values the endpoint accepts" controls. */ +const CHOICES: Array<{ + key: ChoiceKey + list: keyof Pick< + GenerationOptions, + 'aspect_ratios' | 'image_sizes' | 'durations' | 'resolutions' + > + label: string +}> = [ + { key: 'aspect_ratio', list: 'aspect_ratios', label: 'Aspect ratio' }, + { key: 'image_size', list: 'image_sizes', label: 'Image size' }, + { key: 'duration', list: 'durations', label: 'Duration' }, + { key: 'resolution', list: 'resolutions', label: 'Resolution' }, +] + +const NO_CHOICES: Record = { + aspect_ratio: '', + image_size: '', + duration: '', + resolution: '', +} + +/** What the server accepts for this many bases — see the component's own comment. */ +function kindsFor(baseCount: number): GenerationKind[] { + if (baseCount === 0) return ['text_to_image'] + if (baseCount <= 2) return ['image_to_image', 'image_to_video'] + return ['image_to_image'] +} + +/** Whether a row of this kind can take these bases, so the server will not refuse it. */ +function fits(model: GenerationModel, kind: GenerationKind, baseCount: number): boolean { + if (model.kind !== kind) return false + if (kind === 'image_to_image') return model.max_images >= baseCount + if (kind === 'image_to_video') return baseCount === 1 || model.end_image_field !== null + return true +} + +function roleOf(kind: GenerationKind, index: number, count: number): string { + if (kind === 'image_to_video') return index === 0 ? 'Start frame' : 'End frame' + return count === 1 ? 'Base image' : `Base ${index + 1}` +} + +/** The option controls hold strings; the endpoint's own value is looked up on submit, + * so a `6` goes out as a number and a `"5"` as a string, exactly as declared. */ +function declared(values: OptionValue[], chosen: string): OptionValue | null { + return values.find((value) => String(value) === chosen) ?? null +} + +function presetChoices(preset?: GeneratePreset): Record { + if (!preset) return NO_CHOICES + const read = (key: ChoiceKey) => { + const value = preset.parameters[key] + return typeof value === 'string' || typeof value === 'number' ? String(value) : '' + } + return { + aspect_ratio: read('aspect_ratio'), + image_size: read('image_size'), + duration: read('duration'), + resolution: read('resolution'), + } +} + +function presetOutputs(preset?: GeneratePreset): number { + // `num_images` is fal's name for it, and so what the stored request body carries. + const value = preset?.parameters.num_images + return typeof value === 'number' && value >= 1 ? value : 1 +} + +export default function GenerateForm({ title, bases = [], preset, onCancel }: Props) { + const id = useId() + const catalogue = useGenerationStore((s) => s.models) + const catalogueLoaded = useGenerationStore((s) => s.loaded) + const catalogueError = useGenerationStore((s) => s.error) + const ensureLoaded = useGenerationStore((s) => s.ensureLoaded) + const refreshActivity = useActivityStore((s) => s.refresh) + + const [kindChoice, setKindChoice] = useState( + preset?.kind ?? null + ) + const [modelId, setModelId] = useState(null) + const [prompt, setPrompt] = useState(preset?.prompt ?? '') + const [choices, setChoices] = useState(() => presetChoices(preset)) + const [seed, setSeed] = useState(() => { + const value = preset?.parameters.seed + return typeof value === 'number' ? String(value) : '' + }) + const [negative, setNegative] = useState(() => { + const value = preset?.parameters.negative_prompt + return typeof value === 'string' ? value : '' + }) + // Null until touched, which sends null and leaves the catalogue's default standing — + // Veo's row sets `generate_audio: false` because audio raises its price by half, and + // an untouched box must not quietly override that. + const [audio, setAudio] = useState(() => { + const value = preset?.parameters.generate_audio + return typeof value === 'boolean' ? value : null + }) + const [outputs, setOutputs] = useState(() => presetOutputs(preset)) + + const [submitting, setSubmitting] = useState(false) + const [queued, setQueued] = useState(null) + const [error, setError] = useState(null) + const [keyMissing, setKeyMissing] = useState(false) + + useEffect(() => { + ensureLoaded() + }, [ensureLoaded]) + + const kinds = kindsFor(bases.length) + const kind = kindChoice && kinds.includes(kindChoice) ? kindChoice : kinds[0] + + const ofKind = useMemo( + () => catalogue.filter((m) => m.kind === kind), + [catalogue, kind] + ) + const candidates = useMemo( + () => + ofKind + .filter((m) => fits(m, kind, bases.length)) + .sort((a, b) => a.sort_order - b.sort_order || a.label.localeCompare(b.label)), + [ofKind, kind, bases.length] + ) + + // Derived rather than stored, so a catalogue that arrives after the form mounts, or a + // kind switch that empties the old choice, still lands on something offered. + const model = + candidates.find((m) => m.id === modelId) ?? + candidates.find((m) => m.endpoint_id === preset?.model) ?? + candidates[0] ?? + null + const presetModelGone = + preset !== undefined && + catalogueLoaded && + !catalogue.some((m) => m.endpoint_id === preset.model) + + const options = model?.options + const maxOutputs = Math.max(1, options?.max_outputs ?? 1) + const price = model ? formatListPrice(model) : null + + const unavailable = (): string => { + if (catalogueError) return catalogueError + if (!catalogueLoaded) return 'Loading models…' + if (kind === 'image_to_video' && bases.length > 1 && ofKind.length > 0) { + return 'None of the video models takes an end frame. Select one image to use as the start frame.' + } + if (kind === 'image_to_image' && ofKind.length > 0) { + return `No image → image model takes ${bases.length} base images.` + } + return `No ${KIND_LABELS[kind].toLowerCase()} model is available yet.` + } + + const submit = async (event: FormEvent) => { + event.preventDefault() + const text = prompt.trim() + if (!text || !model || submitting) return + if (options?.supports_seed && seed.trim() && !/^\d+$/.test(seed.trim())) { + setError('The seed has to be a whole number') + return + } + + const video = kind === 'image_to_video' + const request: GenerationCreate = { + model_id: model.id, + prompt: text, + base_asset_ids: + kind === 'text_to_image' + ? [] + : video + ? bases.slice(0, 1).map((b) => b.id) + : bases.map((b) => b.id), + end_frame_asset_id: video && bases[1] ? bases[1].id : null, + params: { + aspect_ratio: declared(model.options.aspect_ratios, choices.aspect_ratio), + image_size: declared(model.options.image_sizes, choices.image_size), + duration: declared(model.options.durations, choices.duration), + resolution: declared(model.options.resolutions, choices.resolution), + seed: model.options.supports_seed && seed.trim() ? Number(seed.trim()) : null, + negative_prompt: + model.options.supports_negative_prompt && negative.trim() + ? negative.trim() + : null, + generate_audio: model.options.supports_audio ? audio : null, + num_outputs: Math.min(Math.max(1, outputs), maxOutputs), + }, + } + + setSubmitting(true) + setError(null) + setKeyMissing(false) + setQueued(null) + try { + const job = await generateApi.start(request) + // The prompt stays: the next thing anyone does after a generation is adjust the + // wording and go again, and retyping it would be the whole cost of that. + setQueued(job.asset_name || text) + void refreshActivity() + } catch (err) { + // A signpost rather than a failure, as a missing AI provider is elsewhere. + if (apiErrorCode(err) === 'fal_key_missing') setKeyMissing(true) + else setError(apiErrorMessage(err, 'Could not start generating')) + } finally { + setSubmitting(false) + } + } + + return ( +
void submit(e)} + className="space-y-3 rounded-lg border border-gray-200 px-3 py-2.5 dark:border-gray-700" + aria-label={title} + > +

+

+ + {bases.length > 0 && ( +
    + {bases.map((base, index) => ( +
  • + +

    + {roleOf(kind, index, bases.length)} +

    +

    + {base.name} +

    +
  • + ))} +
+ )} + + {kinds.length > 1 && ( +
+ What to make + {kinds.map((each) => ( + + ))} +
+ )} + +
+ + {model ? ( + + ) : ( +

{unavailable()}

+ )} + {model && (model.note || price) && ( +

+ {[model.note, price].filter(Boolean).join(' · ')} +

+ )} + {presetModelGone && ( +

+ {preset?.model}, which made this, is no longer offered — pick another. +

+ )} +
+ +
+ +