Skip to content

fix(restore): restore documents before uploading media, fail fast on a read-only token - #3

Merged
amenophis1er merged 1 commit into
mainfrom
foreman/prevent-silent-restore-failures-and-186b
Sep 10, 2026
Merged

amenophis1er merged 1 commit into
mainfrom
foreman/prevent-silent-restore-failures-and-186b

Conversation

@amenophis1er

Copy link
Copy Markdown
Contributor

Closes #2.

Root cause

restore uploaded media before it wrote any documents. Media upload is the destructive half — it assigns new media IDs — so when the document writes then failed with 403, the target was left with a fully populated media store that nothing referenced, while the pre-existing documents still pointed at the old IDs. Every image and file 404s, site-wide.

Two things turned a permission error into corruption:

  1. Ordering. The irreversible step ran first, before anything had established that the token could write at all.
  2. Masked denials. The singleton PUT→POST and --overwrite POST→PUT fallbacks swallowed the first error, so a 403 looked like an ordinary retry rather than a hard stop.

The summary then reported Media restored: 141, References updated: 93, Restore completed and exit 0 — three separate untruths, the last of which meant no script or CI step could detect the failure.

The ordering decision

Restore now runs in three ordered phases: documents → media → reference rewrite.

Documents are written first with the backup's original media references intact. Only once at least one document has landed does media upload run; phase 3 then rewrites the references to the newly assigned IDs and PUTs back only the documents whose bodies actually changed.

This was chosen over "preflight the permissions, then keep the old order". A preflight is a check that can be wrong — permissions can be scoped per collection, and a token that passes a probe can still be refused on a specific write. Making the first document write be the gate means the destructive step is unreachable unless a real write has already succeeded. Preflighting is a second line of defence, not the fix.

Phase 2 is additionally skipped when zero documents were restored, so uploads cannot orphan themselves.

Changes

  • Fail-fast on a read-only token. The first 401/403 during document writes aborts with token lacks content:write (required to restore documents) and a non-zero exit, before any upload. The two fallback paths no longer mask a denial.
  • Honest exits. Any failure exits non-zero. Restore completed prints only on a fully clean run.
  • Honest counters. References updated counts references in documents that were successfully rewritten. Added Documents failed. Collections restored counts collections that actually received a document. Non-permission write failures are counted instead of silently skipped.
  • Orphaned-media warning. If media uploaded but some documents could not be repointed, a loud warning names the counts and the remediation. No automatic rollback — see below.
  • --url papercut. NormalizeBaseURL now appends /api only when the URL's path lacks an api segment, so /api, /api/v1 and /backend/api are left alone and the function is idempotent. Applied to both the --url flag and TROKKY_URL.
  • Output moved to cmd.OutOrStdout() / cmd.ErrOrStderr(); SilenceUsage: true so a runtime failure isn't buried under a usage dump.

Deliberately not done

No automatic rollback of uploaded media. Deleting media after a partial failure is itself destructive and cannot distinguish files this run uploaded from pre-existing ones that happen to share a checksum. Given the new ordering makes the orphaning case much harder to reach, a loud warning is the safer trade. Revisit if it shows up in practice.

isPermissionDenied matches on error-message substrings (" (HTTP 403)", "HTTP 403") because internal/client does not expose typed status codes. It is pragmatic and documented, but it is string-matching on error text and will drift if that formatting changes. A typed error in internal/client is the durable fix and belongs in its own PR.

Tests

Six new tests in cmd/restore_test.go, driven against httptest servers that record every request:

  • TestRestoreAbortsBeforeUploadingMediaWhenDocumentWritesAreForbidden — asserts zero requests reach /media/upload
  • TestRestoreFailedDocumentWritesExitNonZero
  • TestRestoreCountersReportOnlyRealWork
  • TestRestoreHappyPathReportsTrueCounts
  • TestRestoreDryRunMakesNoWrites — dry-run behaviour is unchanged
  • TestIsPermissionDenied

Plus TestNormalizeBaseURL and two ResolveCredentials tests in internal/config. Three existing config tests were updated because they asserted the old raw passthrough, which is the bug being fixed; cmd/documents_test.go was updated for the same reason. Nothing was deleted or weakened.

go build ./..., go vet ./... and go test ./... all pass.

…a read-only token

A token with content:read/media:read but no content:write made 'trokky
restore' upload every media file first — minting fresh media IDs — then
fail every document write with HTTP 403, then print "Restore completed"
and exit 0. The target instance was left with a full media store under
new IDs while its documents still referenced the old ones, so every
image 404s.

Restore is now three ordered phases: documents are written first with
their references untouched, media is uploaded second, and a third pass
rewrites media references only into documents that carry them. Media
upload is the destructive half, so nothing is uploaded until the token
has proven it can write documents; the first 401/403 on a document
write aborts with "token lacks content:write (required to restore
documents)" and a non-zero exit, before any upload happens. The
PUT->POST singleton fallback and the --overwrite POST->PUT fallback no
longer mask a permission denial.

Summary counters now report work that actually landed: documents
written, media uploaded, references contained in documents that were
successfully rewritten, and collections that actually received
documents. "Restore completed" prints only on a fully clean run; any
failure exits non-zero. If media was uploaded and some documents could
not be pointed at it, a loud orphaned-media warning names the counts.

Also normalize --url and TROKKY_URL to the API root, so
'--url http://host:3210' no longer fails with "HTTP 404: Cannot GET
/collections".

Closes #2
@amenophis1er
amenophis1er merged commit 6dd3626 into main Sep 10, 2026
1 check passed
@amenophis1er
amenophis1er deleted the foreman/prevent-silent-restore-failures-and-186b branch September 10, 2026 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

restore: read-only token uploads all media, then fails every document write, and reports success

1 participant