fix(api): report partial failures and bound batch bodies on image routes - #9394
Conversation
Three related gaps on the image endpoints, all of which the video endpoints already handle (they were fixed there during the invoke-ai#9163 review): 1. `star_images_in_list` / `unstar_images_in_list` re-raised the first HTTPException mid-batch, so one foreign name discarded the response payload for images that HAD been starred — the client never invalidated their caches and the UI showed them unstarred until the next full refresh. They now skip foreign/missing names like `delete_images_from_list` does, and dedup repeated names so one name can't land in two result buckets. 2. Those same handlers swallowed genuine storage failures with `except Exception: pass`, reporting a success-shaped response for images that were never updated. `StarredImagesResult` / `UnstarredImagesResult` gain `failed_images` (mirroring `DeleteImagesResult` and the video models), and the frontend toasts a partial-failure warning like the video star/unstar mutations do. 3. The `image_names` batch bodies (delete/star/unstar/images_by_names) were unbounded, and `list_image_dtos` had no pagination bounds — a negative LIMIT means *unlimited* in SQLite. Adds MAX_IMAGE_BATCH_SIZE (mirroring MAX_VIDEO_BATCH_SIZE), a 255-char per-name cap, and ge=0/le=MAX_PAGE_SIZE on the list route. The lower bound on `limit` is 0, not 1: the frontend issues count-only queries with limit=0. Deferred non-blocker from PR invoke-ai#9163. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…atch bounds Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JPPhoto
left a comment
There was a problem hiding this comment.
I came across the following issue:
invokeai/app/api/routers/images.py:695-727:/downloadstill accepts unboundedimage_names, performs per-name authorization, then schedules bulk work. An authenticated client can submit an oversized body and consume request/DB resources despite the new batch limits elsewhere. Test: POST 1001 names and assert 422 before authorization or background-task execution.
Suggestions:
- Consider a shared
ImageNamesBatchmodel for every explicit-name batch endpoint; this would apply limits consistently and prevent/downloaddrift.
`/download` was the one explicit-name batch route the bounds pass missed. It accepts `image_names`, authorizes every name individually, then schedules the bulk-download background task, so an authenticated client could still submit an oversized body and buy a per-name DB lookup each. Applies the same `ImageName` / `MAX_IMAGE_BATCH_SIZE` constraints the other four routes already use. Rejection is FastAPI request validation, so it lands before any authorization lookup or background task. Adds `/download` and `/images_by_names` to the existing bounds test, and a drift guard that walks the published OpenAPI schema and fails if any `/v1/images` request body takes an `image_names` array without both a list bound and a per-name length bound — the limits were applied route-by-route, which is how `/download` was missed in the first place. Scoped to the images router deliberately: `/v1/board_images/batch` and `/batch/delete` are unbounded too, but bounding them would reject a change-board request the UI can produce today, so they need to be paired with client-side chunking in a follow-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… client-side The bounds pass applied its limit route-by-route, which left two gaps. Backend: `/images/download` accepted `image_names` unbounded, authorized every name individually, then scheduled the bulk-download task. `/board_images/batch` and `/batch/delete` were unbounded too, and loop per name for permission checks. All three now carry the same `MAX_IMAGE_BATCH_SIZE` bound as the rest. That constant's comment understated the cost it guards. The authorization helpers short-circuit on the first hit, so an admin or a direct owner costs 0-1 queries per name -- but a user reading someone else's Shared/Public board falls through to `boards.get_dto()`, which is six queries including three COUNT aggregates over the board's contents. Since the routes are `async def` and the loop is synchronous, that work blocks the event loop. Hence one uniform bound, with no route granted a laxer one. Frontend: nothing capped a gallery *selection*. Select-all reads the whole board's name list, so one keystroke on a large board produced a selection an order of magnitude past the bound, and no batch call chunked. Delete was the worst case -- `handleDeletions` swallows the rejection, so an oversized delete silently did nothing. All seven batch calls now split oversized bodies into conforming requests. The five mutating ones merge the per-chunk results so callers and `invalidatesTags` still see one aggregate result; `images_by_names` concatenates (a plain ordered list, and its caller only upserts by name); `/download` cannot merge, so an oversized selection becomes several zips -- the socket handler already fetches per `bulk_download_complete` event, keyed on the event's item name. Chunks run sequentially: each is already up to 1000 names of DB work, and firing them concurrently would hand back exactly what the bound took away. A mid-run failure resolves to a partial success, not an error. The earlier chunks are already committed, and a bare error would discard their payload -- the very bug the partial-failure reporting on these routes exists to fix. It is not only the RTK cache at stake: `handleDeletions` drives the gallery selection and strips deleted images out of nodes, canvas layers and reference images off `deleted_images`, and none of that runs on a rejection. So the merged result is returned, the unreached names are toasted as failures, and only a run where nothing landed is an error. Tests: the bounds test covers all seven batch routes; a drift guard walks the published OpenAPI schema and fails if any `image_names` body ships without both a list bound and a per-name length bound, pinning the exact route set so a route the walk *skips* cannot pass unnoticed. Frontend unit tests cover chunk splitting, result merging, and both failure paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extends §1 of this PR to the two routes it missed. `add_images_to_board` and `remove_images_from_board` did `except HTTPException: raise` inside the per-name loop, so one name the caller doesn't own — or one deleted by a concurrent session — discarded the response payload for every image that had *already* been moved in the same request. Those moves are committed; only the report is lost, so the client never invalidated their caches and the UI kept showing them on their old board until a full refresh. Both now skip such names and dedup repeated ones, matching star/unstar/delete. `AddImagesToBoardResult` / `RemoveImagesFromBoardResult` gain the `failed_images` list the other batch results already carry, populated for genuine storage failures only — an auth skip is not a failure and must not be toasted as one. The field is required rather than defaulted because the client toasts off it, and an optional one reaches TypeScript as `undefined`; the single-image routes pass an empty list, where a failure is a 500 and never a partial success. `DeleteImagesResult` is tightened the same way, and delete finally toasts its partial failures — it never did, and `handleDeletions` swallows every outcome, so a delete that only partly landed said nothing at all. Skipping removes the early abort that used to cap an unauthorized batch at one check, so `remove_images_from_board` memoizes board write-access per board id. `_assert_board_write_access` goes through `boards.get_dto()` — six queries, three of them COUNT aggregates over the board's contents — and both routes are `async def` with synchronous DB calls, so unmemoized a 1000-name batch on one board is ~6000 blocking queries on the event loop: exactly what the bound in the previous commit exists to prevent. The check sits outside the per-name try so there is precisely one way to skip a name for authorization; folding it in left two paths to the same outcome and neither was individually load-bearing. `remove_images_from_board` resolves the DTO in its own block, narrowed to `ImageRecordNotFoundException`. It is the one route that reads the DTO *before* any authorization check, so an image deleted between the client building its selection and this request would otherwise be indistinguishable from a storage failure and toasted as one — while a real storage error must still reach `failed_images`. The maintenance pre-check loop skips the same exception: raised from inside an `except HTTPException:` handler it would replace the 409 with a 500. The authorization guarantees are unchanged — only the reporting is. `test_non_owner_cannot_batch_add_other_users_images_to_own_board` is updated for the new shape and asserts the move was never *attempted*, since `board_images` is a MagicMock in that fixture and asserting on `board_image_records` would pass no matter what the route did. Three new tests cover the remove side, which had no authorization test at all — and could not have had one: the fixture left `urls` as None, so `ImageService.get_dto` raised `AttributeError` for every image and every name was skipped before the ownership check ran, passing regardless of what the route did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Confirmed, and fixed — thanks, this was a good catch and pulling on it turned up more than the one route.
On the shared Three things fell out of chasing it, all now in the PR: Two more unbounded routes. The comment on The client could exceed the bound in one keystroke. A mid-run chunk failure resolves as a partial success rather than an error, for the same reason §1 of this PR exists: the earlier chunks are committed, and a bare error discards their payload. That one isn't only about the RTK cache — Two things I looked at and left alone deliberately, both pre-existing and PR-wide rather than specific to these routes — written up at the bottom of the PR description: the maintenance pre-check loop still 403s a batch containing a foreign name, and on Full suites green: |
JPPhoto
left a comment
There was a problem hiding this comment.
A few things to fix:
-
invokeai/frontend/web/src/services/api/endpoints/images.ts:616-631can schedule earlier download chunks, then reject after a later chunk fails. The UI shows “request failed” while earlier ZIPs still arrive. Test: mock chunk 1 as 202 and chunk 2 as 403; verify partial status handling. -
invokeai/app/api/routers/images.py:534-547, 624-638, 670-681reportsImageRecordNotFoundExceptionasfailed_imagesafter authorization passes. Concurrent deletion is an intentional skip, not a storage failure. Test: pass ownership, raise the exception during the subsequent service call, and assert the name is absent from both result lists.
Suggestions:
-
Consider one server-side download job for the full authorized selection; this preserves atomic authorization and one coherent result.
-
Instead of generic-catching not-found races, classify
ImageRecordNotFoundExceptionas a skip; this aligns image routes with video and board-removal behavior. -
Consider deriving the client chunk size from the generated schema; this prevents frontend/backend limit drift.
`/images/delete`, `/star` and `/unstar` reported an `ImageRecordNotFoundException` raised after the ownership check in `failed_images`, so an image deleted by a concurrent session between the client building its selection and the request landing toasted "1 image could not be updated" for an outcome the user actually got. `remove_images_from_board` already resolves the same race as a skip; these three now match it, and the name is absent from both result lists. Star/unstar reach the race through the `get_dto` read-back inside `ImageService.update`: the UPDATE matches no row and raises nothing, so a name that vanished mid-batch surfaces only on the read that follows. The skip is only sound if the exception means what its name says, and it did not: `image_records.get()` re-raised every `sqlite3.Error` as `ImageRecordNotFoundException`, so a locked, corrupt or unreadable database was indistinguishable from a concurrent delete. Under the new skip that would have turned a wholly failed batch into 200 with two empty lists and no toast at all — strictly worse than the spurious warning the skip removes. The translation is dropped in `get()` and `get_metadata()`; storage errors now propagate as themselves, which also un-swallows them for the two existing skips in `board_images.py`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chunked `/images/download` loop returned a bare error on any chunk failure. The route answers 202 the moment it has scheduled the background task, so every chunk before the failing one is already producing a zip: the user saw "Problem preparing download" from the `matchRejected` listener while those zips landed in their downloads anyway. It now follows `buildChunkedImageBatchQueryFn` — only a run where nothing was scheduled surfaces as an error; a partial run resolves with the first chunk's payload and warns with the count of names that made it into no zip. The warning has its own toast id, since the toast system updates in place and sharing `IMAGES_FAILED_TO_UPDATE` would let one count replace the other. "Something was scheduled" is tracked in its own flag rather than inferred from the payload: `fetchBaseQuery` resolves an empty response entity as `data: null`, so a 202 whose body did not survive the trip back leaves nothing to return even though the task was scheduled. The `queryFn` is extracted as `bulkDownloadQueryFn` so it can be tested. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ial-failures-and-bounds # Conflicts: # invokeai/frontend/web/public/locales/en.json
|
Both confirmed and fixed at 93420df. Also merged 1.
|
JPPhoto
left a comment
There was a problem hiding this comment.
Merge blockers:
-
invokeai/frontend/web/src/services/api/endpoints/images.ts:128-156,251-292rereads auth token per chunk viainvokeai/frontend/web/src/services/api/index.ts:111-125. If user A starts a large mutation, then logout/login switches to B, later chunks run as B. RTK 2.8.2resetApiStateonly resets reducer state, not the running request source. Public-board permissions can let B mutate A's images. Effect: cross-user partial mutations/downloads; earlier changes cannot roll back. Likelihood: Low-medium, but reachable. Recovery: manually restore board/star state; deletes may be irreversible. Test: pause after chunk one, change token, resume, and assert later chunks abort or retain A's credentials. -
invokeai/app/api/routers/board_images.py:239-248,267-278caches board write authorization for the entire remove batch. If a public board becomes private after the first image, later removals reuseTruewithout rechecking. Effect: a previously authorized contributor can remove additional images after permission revocation. Likelihood: Low, but valid under concurrent multiuser updates. Recovery: restore removed board assignments. Test: change board visibility after the first removal and assert subsequent removals are skipped.
Other findings/issues:
-
invokeai/app/api/routers/board_images.py:165-185does not catchImageRecordNotFoundExceptionor the SQLite FK failure frominvokeai/app/services/shared/sqlite_migrator/migrations/migration_1.py:22-34. Deleting an image after ownership validation makes the insert fail and adds it tofailed_images, contradicting the PR metadata claim that deleted names are skipped. Effect: false failure warning for an already-deleted image. Likelihood: Medium under concurrent deletion. Test: coordinate deletion between ownership validation and insert; assert the name is omitted fromfailed_images. -
invokeai/frontend/web/src/services/api/endpoints/images.ts:704-747discards all successful DTO chunks when a later chunk fails.useRangeBasedImageFetchingignores the rejected promise atinvokeai/frontend/web/src/features/gallery/hooks/useRangeBasedImageFetching.ts:63-68. Effect: earlier thumbnails never enter cache after a transient later failure. Likelihood: Medium for large gallery ranges. Recovery: scroll/reload and hope the range retries. Test: return DTOs for chunk one, an error for chunk two, and assert chunk-one DTOs are still upserted. -
invokeai/frontend/web/src/features/changeBoardModal/components/ChangeBoardModal.tsx:88-118fires the new partial-result image mutation without awaiting it, then resets selection after only the video promises settle. Effect:failed_imagesare left on the old board while the modal clears them, making retry difficult. Likelihood: Medium during partial batch failures. Recovery: manually reselect failed images. Test: resolve the mutation withfailed_imagesand assert those names remain selected. -
invokeai/frontend/web/src/services/api/endpoints/images.ts:257-274chunksimage_nameseven whenboard_idis also supplied. The backend prioritizesboard_idatinvokeai/app/services/bulk_download/bulk_download_default.py:47-55; thus 1001 names plus a board ID schedules duplicate full-board ZIP jobs. Effect: avoidable CPU, disk, and download amplification. Likelihood: Low; no current UI caller, but the API accepts both fields. Recovery: cancel/delete duplicate downloads. Test: submit both fields with over 1000 names and assert rejection or one normalized request.
Corner/impossible cases:
invokeai/frontend/web/src/services/api/endpoints/images.ts:262-292permits a null 202 payload, butinvokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/bulkDownload.tsx:11-28dereferences it immediately. The current route always returns a model atinvokeai/app/api/routers/images.py:733-772, so this needs a body-stripping proxy. Effect: fulfilled download action throws and the preparing toast is lost. Likelihood: Very low with the current server, reachable with the proxy behavior modeled by the PR's own test. Test: dispatch a fulfilled action withnullpayload and assert the listener does not throw.
Suggestions:
-
Instead of reading ambient auth state for every chunk, capture the auth generation at operation start and abort when it changes; this prevents cross-account continuation.
-
Instead of caching board permission across a mutable batch, authorize and mutate within one transaction or snapshot; this removes the permission TOCTOU window.
-
Instead of waiting for every DTO chunk before upserting, upsert each successful chunk immediately; this preserves useful results after later failures.
-
Instead of accepting both
board_idandimage_names, reject the combination or normalize it before chunking; this eliminates duplicate full-board jobs. -
Instead of clearing board-change selection immediately, await the image mutation and retain
failed_images; this makes partial moves retryable.
|
Thanks — all three confirmed and addressed in 1defaf5, with two residual gaps an adversarial pass over that fix then caught, closed in a18445b.
On the zips that are scheduled and then orphaned by expiry: agreed that per-account persistence + replay after re-auth is the actual fix — filed as #9531 so it doesn't live only in a code comment. |
JPPhoto
left a comment
There was a problem hiding this comment.
(Note: This review excludes the issue mentioned in #9531.)
Merge blockers:
- Cross-user takeover can consume old-session results.
invokeai/frontend/web/src/services/api/endpoints/images.ts:170-176,257-295handles errors without rechecking auth. Effect: old results/tags may affect the new user's state. Likelihood: Low but plausible with multi-tab use or flaky requests. Recovery: classify auth changes before processing any error. Test: switch users before a chunk returns an error and assert no old result or invalidation is applied.
Other findings/issues:
-
Lost mutation responses leave stale local references.
invokeai/frontend/web/src/services/api/endpoints/images.ts:263-295only invalidates caches; it does not report which deletes actually committed. Effect: canvas, node, and reference-image references can remain stale. Likelihood: Low-medium during timeout, disconnect, parsing, or 5xx failures. Recovery: add mutation ids and server-side outcome reconciliation. Test: commit a delete, returnFETCH_ERROR, and assert local deletion cleanup. -
Storage errors during image authorization are silently skipped.
invokeai/frontend/web/src/services/api/endpoints/images.py:670-688,722-741catches allHTTPException, while_access.py:29-40converts board lookup failures into 403s. Effect: star/unstar can report success without updating or warning. Likelihood: Very low. Recovery: distinguish authorization denial from storage failure. Test: make board lookup raise a database error and assertfailed_images.
Suggestions:
-
Instead of treating indeterminate mutations as failed without an outcome, use per-operation ids and a reconciliation endpoint, because cache invalidation cannot repair persisted workspace references.
-
Consider typed authorization and storage exceptions, because this prevents database faults from being mistaken for ordinary permission skips.
… storage errors into auth skips Two round-10 review findings: - fetchChunk triaged only *successful* responses for takeover, on the theory that an error carries nothing the next session could consume. Since the indeterminate-error reconciliation, the error path consumes plenty: it dispatches as-if-committed invalidations and returns partial aggregates the UI applies. An error returning into a taken-over session is now rewritten to the auth-changed hard abort, so the loops consume nothing. Mere expiry still passes through untriaged - the everyday 401 must keep reaching the partial path, where committed work is reported and pruned. - assert_image_owner wrapped the board-ownership fallback in a bare except-pass, so a database error during the board lookup became a 403 - and the batch star/unstar loops treat a 403 as a silent auth skip: not applied, not reported, nothing toasted. The lookup now reads the board record (owner and visibility are all the decision needs), catches only BoardRecordNotFoundException, and lets storage errors propagate into each loop's failed_images arm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UE38kzGWoj4w6dYiPqa3CY
An implementation that let BoardRecordNotFoundException propagate alongside the storage errors would toast a failure for a name whose only problem is that its board vanished mid-request. The gone-board arm stays a silent auth skip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UE38kzGWoj4w6dYiPqa3CY
|
All three confirmed. The blocker and the star/unstar finding are fixed in 8a508f7 (+ a coverage pin in 4c56cd0, the new head); the lost-response finding is the one thing I'm deliberately not solving inside this PR — filed as #9533 with the design constraint that makes it a server-side change, reasoning below. Takeover can consume old-session results on the error path — Real, and it's a regression my own round-9 change created: the "error carries nothing the next session could consume" rationale in Storage errors laundered into auth skips on star/unstar — Real, and took both halves of your typed-exceptions suggestion as far as this PR's routes go: Lost responses leave stale workspace references — Real, and I attempted the client-side version before concluding it must be yours truly's follow-up instead: the natural probe ( |
JPPhoto
left a comment
There was a problem hiding this comment.
Merge blockers:
- In-flight session A 401s can log out session B before the new guard runs.
invokeai/frontend/web/src/services/api/index.ts:141-146dispatches logout without checking the request token is still current;invokeai/frontend/web/src/services/api/endpoints/images.ts:170-183checks only afterward. Effect: cross-user session destruction. Likelihood: Low-medium. Recovery: reauthenticate. Test: return A's 401 after B takes over and assert B remains authenticated.
Other findings/issues:
-
Lost delete responses leave stale workspace references.
invokeai/frontend/web/src/services/api/endpoints/images.ts:270-302reports uncertain names only as failed, whileinvokeai/frontend/web/src/features/deleteImageModal/store/state.ts:96-126cleans onlydeleted_images. Effect: deleted images remain in canvas, nodes, or reference-image state. Likelihood: Low-medium. Recovery: manual cleanup or reconciliation. Test: commit a delete, returnFETCH_ERROR, and assert local references are removed. -
Star/unstar can commit, then report failure without invalidating the image cache.
invokeai/app/services/images/images_default.py:128-143updates the record before its DTO read;invokeai/app/api/routers/images.py:670-688,723-734reports later errors asfailed_images, butinvokeai/frontend/web/src/services/api/endpoints/images.ts:323-337invalidates only successful names. Effect: stale starred state. Likelihood: Low. Recovery: invalidate failed-image tags and refetch. Test: commit the update, fail DTO retrieval, and assert the image cache is invalidated.
Suggestions:
-
Instead of dispatching session expiry for every token-bearing 401, verify that the request token is still current, because an old request must not terminate a newly adopted session.
-
Instead of treating uncertain deletes as permanently failed, use operation IDs and outcome reconciliation, because retrying cannot repair local workspace references.
-
Instead of separating board authorization from mutation, use a transaction or conditional write, because per-name checks still leave a permission-revocation race.
`dynamicBaseQuery` ended the session on any 401 that carried a token, using the token captured when the request went out. A request that is still in flight when someone else takes over the tab — a login here, or one in another tab, since localStorage is shared — then logs out the user who never issued it. The 401 is only evidence about the credential that was sent, so the session ends only while that credential is still the live one. Byte equality, deliberately the opposite of `isSameAuthContext`: a sliding-window refresh must not qualify either, because a 401 for the token it replaced says nothing about the replacement. Nothing is lost by waiting — the next request carries the live token and its 401 ends the session here. Also invalidate `failed_images` on star/unstar. `ImageService.update` writes the record and then reads the DTO back; a failure in that read reports the name as failed with the row already starred, and invalidating only the successes leaves the client showing the pre-star value until a full reload. The remove-from-board helper already does this for the same reason.
Three things the adversarial pass turned up. The blocker was only half fixed. `ProtectedRoute` ends the session on a 401 from `getCurrentUser` with no freshness check at all, and `sessionExpiredLogout` removes `auth_token` from localStorage — which is shared across tabs. So the same stale 401 still deleted the replacement session's credential: the query goes out during page load carrying an expired token, another tab logs in, and the 401 lands before the adoption poll runs. Both sites now ask the same predicate. Invalidating `failed_images` on star/unstar hands three components a new way to discard the user's input. A node image field and both reference-image components clear their value on ANY query error, so the refetch that the new invalidation triggers can silently drop a workflow input — and the refetch is likelier than usual to fail, because a name is in `failed_images` precisely when a storage failure interrupted its write. Only a 404 proves the image is gone; the video field was narrowed this way already, and the image side now has the same predicate. The new guard also let an empty-string token qualify, which sets no Authorization header and so proves nothing about any session.
Narrowing the components to a 404 was wrong on its own: a deleted image only answers 404 for an admin. `assert_image_read_access` decides on `images.user_id`, which is gone with the row, so in a multiuser deployment a deleted image is indistinguishable from someone else's and both are refused 403. Requiring 404 would have stranded every deleted image in the workflows referencing it, which is worse than the over-broad clear it replaced. So the clients act on 403 as well — and that puts an obligation on the answer. The read helper laundered every storage error from its board lookup into that same 403, so an unreadable database presented as a permission decision and would now take the user's references down with it. It reads the board record and catches only a positive not-found, matching what `assert_image_owner` already does on the mutation side; anything undecidable propagates.
… same answer The `ProtectedRoute` guard only postponed the logout it was meant to prevent. The store's token catches up when the poll adopts the new one, the effect re-runs, and the superseded 401 is still sitting in the cache: `getCurrentUser` is shared across logins, its argument never changes, it carries no tags, and the API-state reset a login normally brings is deliberately skipped when the new token belongs to the same user. On the ordering where the storage event beats the response — the common one, since it is delivered without a network round trip — the guard never even delayed it. Keyed by the token instead, so the adopted session reads its own entry and the superseded 401 is no longer in hand to act on. The argument is never sent — the token goes in a header — but it is what the answer is about, and the call site now cannot forget to pass it: the endpoint's argument type is what enforces it. Videos get the read-side pair the images just got. Trusting a 403 as "gone" is what lets a deleted item clear itself, and `isVideoMissingError` accepted only 404 — which no non-admin ever sees for a deleted video — so a deleted video would have stayed pinned in a workflow field forever. The video read helper stops laundering storage errors into that same 403.
|
All three confirmed and fixed, at Merge blocker: an in-flight 401 logging out the session that replaced itConfirmed, and worse than one site. The session now ends only while the token that was sent is still the live one (
Gating that effect on the same predicate was not enough, which my own adversarial pass caught: the store's token catches up when the poll adopts the new one, the effect re-runs, and the superseded 401 is still in the cache. So the query is now keyed by the token it will carry. It takes an argument it never sends, because the token travels in a header, but it is what the answer is about: the adopted session reads its own cache entry, and the superseded 401 is no longer in hand to act on. The call site cannot forget to pass it — the argument type is what enforces the wiring, and reverting it fails Test as you asked: A's 401 returns after B has taken over, and B stays authenticated. Also the refresh case and the unauthenticated case, plus a store-level test that a replacement token cannot read its predecessor's 401. Star/unstar committing, then reporting failure without invalidatingConfirmed. That fix needed a second one to be safe, which is the part I would flag for your attention. A node's image field and both reference-image components clear the user's input on any query error, so the refetch the new invalidation triggers could silently drop a workflow input — and it is likelier than usual to fail, since a name is in Narrowing it to 404 was wrong on its own, though: a deleted image only answers 404 to an admin. So the clients act on 403 as well, and that puts an obligation on the answer: the read helper laundered every storage error from its board lookup into that same 403, so an unreadable database presented as a permission decision and would now take the user's references down with it. It reads the board record and catches only a positive not-found, matching what Videos got the same pair. Lost delete responses leaving stale workspace referencesStill deferred, for the reason I gave last round: the only client-side repair is a probe that cannot distinguish "gone" from "couldn't look". On the suggestionsConditional writes: the remove path already got that treatment in this PR — the DELETE is scoped to the board the caller was authorized against, so a name that moved in between matches zero rows and is classified rather than reported as a success that did not happen. Extending it to star, unstar and add is worth doing, but not as a WHERE clause: the policy is a four-way disjunction across three tables, and duplicating it into SQL at each call site guarantees drift. It wants the decision pushed into the service transaction, which is #9534. What is not coveredThe three components and The remaining laundering I know of is |
JPPhoto
left a comment
There was a problem hiding this comment.
Merge blockers:
invokeai/frontend/web/src/features/nodes/components/flow/nodes/Invocation/fields/inputs/videoFieldErrors.ts:2-23: Treats every 403 as deleted/unavailable. Revoking access to a shared video's board returns 403 and silently clears workflow fields. Effect: reversible permission changes cause local workflow data loss. Likelihood: Plausible. Recovery: Manual reattachment. Test: Reference a shared video, make its board private, refetch, and assert the field remains.
Other findings/issues:
invokeai/frontend/web/src/services/api/endpoints/images.ts:271-303: An indeterminate delete chunk may commit server-side, then return an error;handleDeletionsonly cleans references for fulfilleddeleted_imagesatinvokeai/frontend/web/src/features/deleteImageModal/store/state.ts:96-129. Effect: Deleted images remain referenced by nodes, canvas, and control layers. Likelihood: Plausible during timeout, disconnect, or 5xx. Recovery: Manual cleanup; deleted files are unrecoverable. Test: Commit deletion, returnFETCH_ERROR, and assert persisted references are removed.
Suggestions:
-
Consider distinguishing confirmed deletion from denied read access in video field handling; this preserves references across reversible permission changes.
-
Instead of treating an indeterminate delete chunk as an ordinary rejected mutation, reconcile it through an operation ID and outcome endpoint; this gives cleanup code definitive deletion results.
-
Instead of hardcoding frontend batch limits, derive them from a shared contract or generated schema check; this prevents backend/frontend limit drift.
…at the client Clearing a reference on a 403 was wrong: revoking access to a shared board refuses every image on it and every one of them still exists, so a board flipped to Private would clear the workflow fields pointing at its images, and flipping it back would not bring them back. Requiring a 404 was also wrong, for the reason that produced the 403 rule — the ownership decision rests on a `user_id` that is gone with the row, so a deleted image reached the same refusal as a foreign one and no non-admin ever saw a 404 for one. Neither answer was the client's to guess, so the server draws the line. On the refusal path only, both read helpers ask whether the record is actually there: absent answers 404, present answers 403. The clients go back to treating 404 alone as gone. The cost is that a caller can now tell an absent image from one they may not read, which admins could always do, and image names are generated UUIDs. That makes `video_records.get` load-bearing, so it stops translating storage errors into not-found — otherwise a locked database would present as a deleted video and clear the user's fields. It also un-breaks the staged-delete recovery, which read that same exception as "the delete committed" and purged the staged files on a database it merely could not read. An uncertain delete now refetches the names it could not confirm. Their references are left in place, since pruning on a guess would discard work over a request that merely failed, and asking is what settles it: gone answers 404 and the components holding it let go, survived answers with its DTO. Partial by construction — canvas layers hold names with no DTO query behind them, and only handleDeletions prunes those (invoke-ai#9533).
Both DTO routes ended `except Exception: raise HTTPException(404)`, so a board lookup against an unreadable database, or a URL service failure, answered the same 404 as a missing row. That was survivable while a 404 only meant a stale cache entry. It is not survivable now that the clients drop the user's reference on one: a locked database would clear live images out of the workflows using them. Only a genuinely missing record answers 404. Narrowed on these two routes alone, because these are the 404s that are acted on destructively — the media, metadata and workflow routes keep theirs.
…split Four things. The existence probe read the whole record, so a row this version cannot deserialize — an enum value written by a newer one — failed exactly as absence does, and would have reported a live image gone. It is a bare row probe now, which also drops it to one point SELECT per refused name. `video_records.get` not translating storage errors had no test at all, and after the DTO routes were narrowed it is the only thing standing between an unreadable database and a 404 that clears the user's fields. Pinned from both ends: the store propagates rather than reporting the row missing, and the staged-delete recovery keeps the staged files instead of purging them when it cannot read the record. Reconciling an uncertain delete was inert for the deletes people actually perform. Anything up to the batch cap is a single chunk, and a single chunk that fails reports nothing back, so the endpoint's invalidation never runs on a result — the only invalidation is the one the queryFn dispatches for the lost chunk, and it described those names as committed, which is exactly the case the delete tag set skips DTOs for. It now describes them as unconfirmed too, which is what they are, so the refetch that settles them actually happens. Both arms of the reference-image reset now wait for the connection; the original's used to clear regardless of it.
|
Both confirmed and fixed. You caught the half of the tradeoff I got wrong, and chasing it showed the choice was never the client's to make. Blocker: clearing a reference on a 403Confirmed, and the harm is exactly as you describe — the refusal is reversible and the loss is not. Flip a shared board to Private and every field pointing at its images clears; flip it back and the images are all still there, but the workflows are not. The reason I widened it to 403 last round was the other half of the same problem: a deleted image also answered 403 to every non-admin, because the ownership decision rests on So the server now draws the line it is the only party able to draw. On the refusal path — the happy path pays nothing — both read helpers ask whether the record is actually there: positively absent answers 404, present answers 403. The clients are back to treating 404 alone as gone, which is your suggestion, and the case you asked for is pinned in both directions: a shared board flipped to Private still answers 403 for an image that exists, and a deleted image now answers 404 to a non-admin instead of 403. Videos got the same pair. The cost is that an authenticated caller can now tell an absent image from one they may not read. That is the answer admins have always received, and image names are generated UUIDs, so it is not something an attacker can enumerate — but it is a real change and worth naming. Two consequences fell out of making 404 load-bearing, and both were pre-existing laundering of the same shape this PR has been unwinding:
Lost delete responsesThe uncertain names are now refetched, which is as much of your reconciliation as can be done without the server-side outcome record. Their references stay in place, deliberately: pruning on a guess would discard work over a request that merely failed, and It is partial by construction, and I would rather say so than imply otherwise: it repairs what is mounted and subscribed. Canvas raster and control layers hold image names with no DTO query behind them, and only What the self-review then found in thatFour things, all fixed in the same push:
On the third suggestionAlready there: |
Summary
Follow-on to #9163 (deferred non-merge-blocker). The video endpoints got these fixes during the review; the image endpoints they were modelled on never did, so images and videos in the same selection behave differently today.
1. Batch mutations aborted on the first foreign name
star_images_in_list/unstar_images_in_list— and, as of the latest round,add_images_to_board/remove_images_from_board— didexcept HTTPException: raiseinside the per-name loop. One name the caller doesn't own, or one deleted by a concurrent session, discarded the response payload for every image that had already been changed. Those changes are committed; only the report is lost, so the client never invalidated their caches and the UI showed them stale until a full refresh.All four now skip such names, matching
delete_images_from_listand the video routes, and dedup repeated names so a name repeated in one request can't land in both result buckets.The authorization guarantees are unchanged — only the reporting is.
test_non_owner_cannot_star_imageandtest_non_owner_cannot_batch_add_other_users_images_to_own_boardare updated for the new response shape and now also assert the underlying mutation was never attempted.2. Storage failures were silently reported as success
The same loops swallowed real failures with
except Exception: pass, so a star that never reached the DB came back looking applied and vanished on reload.StarredImagesResult/UnstarredImagesResult/AddImagesToBoardResult/RemoveImagesFromBoardResultgain afailed_imageslist (mirroringDeleteImagesResult), populated for genuine failures only — an auth skip is not a failure and must not be toasted as one. The frontend toasts the partial-failure warning the video mutations already show;deleteImagesgains it too, having never surfaced its ownfailed_imagesat all.3. Unbounded request bodies and pagination
image_nameswas unbounded on every explicit-name batch route:delete,star,unstar,images_by_names,download, and bothboard_imagesbatch routes. AddsMAX_IMAGE_BATCH_SIZE = 1000(mirroringMAX_VIDEO_BATCH_SIZE) plus a 255-char per-name cap, uniformly.The per-name cost is worse than "one DB lookup". The helpers in
_access.pyshort-circuit on the first hit, so an admin or a direct owner costs 0–1 queries — but a user reading someone else's Shared/Public board falls all the way through toboards.get_dto(), which is six queries including three COUNT aggregates over the board's contents. The routes areasync defand the loops are synchronous, so that work blocks the event loop. Hence one bound everywhere, with no route granted a laxer one — andremove_images_from_boardmemoizes its per-board check, since skipping removed the early abort that used to cap an unauthorized batch at one lookup.list_image_dtoshad no pagination bounds. A negativeLIMITmeans unlimited in SQLite, solimit=-1materialized every image row into a DTO. Addsge=0/le=MAX_PAGE_SIZE, matching the video list route. Lower bound is 0, not 1 — the frontend issueslimit=0count-only queries (useHasImages).4. Nothing capped the selection those bodies come from
The bound above is only safe because the client stopped exceeding it. It doesn't:
selectAllOnPage(mod+a) reads the whole board's name list from an unpaginated endpoint, so one keystroke on a large board produces a selection an order of magnitude past the cap, and no batch call chunked. Delete was the worst case —handleDeletionsswallows the rejection, so an oversized delete silently did nothing at all.All seven batch calls now split oversized bodies into conforming requests. The five mutating ones merge the per-chunk results, so callers and
invalidatesTagsstill see one aggregate result with the shape a single request would have returned;images_by_namesconcatenates (a plain ordered list, and its caller only upserts by name);/downloadcannot merge — each request produces its own bulk-download item, so an oversized selection becomes several zips. That is fine in practice: the socket handler already fetches and saves perbulk_download_completeevent, keyed on the event's item name rather than on the POST payload.Chunks run sequentially. Each is already up to 1000 names of DB work, and firing them concurrently would hand straight back what the bound took away.
A mid-run failure resolves to a partial success, not an error — the same principle as §1. The earlier chunks are already committed, and returning a bare error would discard their payload. It isn't only the RTK cache at stake:
handleDeletionsdrives the gallery selection and strips deleted images out of nodes, canvas layers and reference images offresult.deleted_images, and none of that runs on a rejection. So the merged result is returned, the unreached names are folded into itsfailed_imagesso one toast reports one total, and only a run where nothing landed surfaces as an error.Testing
pytest tests/app— 2213 passed, 8 skipped, 6 xfailed.test_every_image_names_body_is_boundedwalks the published OpenAPI schema and fails if anyimage_namesrequest body ships without both a list bound and a per-name length bound. It pins the exact route set rather than a count floor, so a route the walk skips — a body nestingimage_namesinside a model instead of declaring it flat — can't pass unnoticed.lint:tsc,lint:eslint,lint:prettierclean;test:no-watch1797 passed (8 new, covering chunk splitting, result merging, and both failure paths).schema.ts/openapi.jsonregenerated with the locked toolchain./board_images/batch/deleteturned out to have no authorization test anywhere, and could not have had one — the fixture lefturlsas None, soImageService.get_dtoraisedAttributeErrorfor every image and every name was skipped before the ownership check ran. Any test written there would have passed regardless of what the route did. Fixed, and the remove side now has three.Reviewer notes
Two things I looked at and deliberately left alone, both pre-existing and both PR-wide rather than specific to these routes:
assert_image_move_maintenance_inactive()pre-check loop (present on star/unstar and both board routes) still uses the raising helpers, so during an image-move maintenance window a batch containing one foreign name 403s where it would otherwise return 200/201. Consistent across every route that has the pre-check; changing it is a separate decision about whether maintenance state should stay behind an ownership check.remove_images_from_boardthe authorization is board-level, not image-level, so a skip does not strictly mean "not yours": an image you own that sits on a Shared board you don't own is skipped silently. Reachable only via an admin move or a visibility change after the fact, and the previous behaviour (403, aborting the whole batch) didn't move it either — but worth knowing.🤖 Generated with Claude Code