CV upload to document source fails on staging

[rngadam]

POST https://api.cv.staging.coderbunker.ca/api/document-source 500 (Internal Server Error)

[2026-01-27 18:19:49.753 +0000] DEBUG: API Router Middleware
    method: "POST"
    path: "/api/document-source"
[2026-01-27 18:19:50.848 +0000] DEBUG (Error): Failed to queue document processing job. Please try again.
    type: "HTTPException"
    stack: "Error: Failed to queue document processing job. Please try again.\n    at <anonymous> (/usr/src/app/dist/src/api/documentSource.js:482:19)\n    at async dispatch (/usr/src/app/node_modules/hono/dist/compose.js:22:23)\n    at async <anonymous> (/usr/src/app/dist/src/api/middleware/requireAuth.js:52:11)\n    at async dispatch (/usr/src/app/node_modules/hono/dist/compose.js:22:23)\n    at async <anonymous> (/usr/src/app/dist/src/api/index.js:26:11)\n    at async dispatch (/usr/src/app/node_modules/hono/dist/compose.js:22:23)\n    at async cors2 (/usr/src/app/node_modules/hono/dist/middleware/cors/index.js:79:11)\n    at async dispatch (/usr/src/app/node_modules/hono/dist/compose.js:22:23)\n    at async trimTrailingSlash2 (/usr/src/app/node_modules/hono/dist/middleware/trailing-slash/index.js:4:11)\n    at async dispatch (/usr/src/app/node_modules/hono/dist/compose.js:22:23)\n    at async <anonymous> (/usr/src/app/node_modules/hono/dist/hono-base.js:301:31)\n    at processTicksAndRejections (native:7:39)"

frontend: https://cv.staging.coderbunker.ca/version.json

{
  "version": "0.0.0-pr-3-deploy-staging",
  "commit": "f877339a5598c3a48595fb8840ddb77a3da72c73",
  "buildDate": "2026-01-27T16:06:56Z",
  "environment": "3/merge"
}

backend: https://api.cv.staging.coderbunker.ca/api/version

{"version":"1.2.0","commit":"4b7411acf0a2437273523746ab893a177d9b599f","buildDate":"2026-01-27T14:25:11Z","environment":"4/merge"}

[rngadam]

s3 credentials ok:

bash-5.3$ nix develop -c bun run scripts/test-s3.ts .env.cv.staging.coderbunker.ca ENV_FILE=.env.cv.staging.coderbunker.ca bun run dev
KUBECONFIG set to /Users/rngadam/coderbunker/src/modern-resume-backend/kubeconfig.yml
Loading environment from: /Users/rngadam/coderbunker/src/modern-resume-backend/.env.cv.staging.coderbunker.ca
--- S3 Configuration ---
Endpoint:   https://s3.bhs.io.cloud.ovh.net/
Region:     bhs
Bucket:     cv.staging.coderbunker.ca
Path Style: true
------------------------

1. Checking connection (listing objects)...
✅ Success! Found 0 objects in bucket.

2. Testing write permission (uploading test-write-1769538566150.txt)...
✅ Success! File uploaded.

3. Testing read permission (verifying content)...
✅ Success! Content matches.

4. Testing delete permission (cleaning up test-write-1769538566150.txt)...
✅ Success! File deleted.

🎉 S3 integration is working correctly!

[rngadam]

deploy failed because I changed the database user from cv-staging-user to cv.staging.coderbunker.ca

bash-5.3$ staging logs cv-api-deployment-5554d4d8b5-m5pz9
Starting entrypoint script...
Running database migrations...
$ bun --env-file=${ENV_FILE:-.env.local} prisma migrate deploy --schema=./src/prisma/schema.prisma
Prisma schema loaded from src/prisma/schema.prisma
Datasource "db": PostgreSQL database "cvstagingdb" at "postgresql-14835a82-o119cbf6d.database.cloud.ovh.net:20184"

25 migrations found in prisma/migrations

Error: P3009

migrate found failed migrations in the target database, new migrations will not be applied. Read more about how to resolve migration issues in a production database: https://pris.ly/d/migrate-resolve
The `20260120232716_document_source_hash_not_unique` migration started at 2026-01-28 15:42:51.478264 UTC failed


error: "prisma" exited with code 1
error: script "db:deploy" exited with code 1
❌ Error: Database migrations failed.

the migration tries to drop the index:

cvstagingdb=> DROP INDEX "DocumentSource_hash_key";
ERROR:  must be owner of index DocumentSource_hash_key

but:

cvstagingdb=> DROP INDEX "DocumentSource_hash_key";
ERROR:  must be owner of index DocumentSource_hash_key
cvstagingdb=> \di+ "DocumentSource_hash_key"
                                                        List of relations
 Schema |          Name           | Type  |      Owner      |     Table      | Persistence | Access method | Size  | Description 
--------+-------------------------+-------+-----------------+----------------+-------------+---------------+-------+-------------
 public | DocumentSource_hash_key | index | cv-staging-user | DocumentSource | permanent   | btree         | 16 kB | 

and:

cvstagingdb=> SELECT current_user;
       current_user        
---------------------------
 cv.staging.coderbunker.ca
```

fixed with:

bash-5.3$ ./scripts/psql.sh .env.postgresql-14835a82-o119cbf6d.database.cloud.ovh.net
Connecting to database using configuration from '.env.postgresql-14835a82-o119cbf6d.database.cloud.ovh.net'...
psql (17.7)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off, ALPN: postgresql)
Type "help" for help.

defaultdb=> \connect stagingdb
connection to server at "postgresql-14835a82-o119cbf6d.database.cloud.ovh.net" (51.161.81.95), port 20184 failed: FATAL: database "stagingdb" does not exist
Previous connection kept
defaultdb=> \connect cvstagingdb
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off, ALPN: postgresql)
You are now connected to database "cvstagingdb" as user "avnadmin".
cvstagingdb=> REASSIGN OWNED BY "cv-staging-user" TO "cv.staging.coderbunker.ca";
REASSIGN OWNED

but then database restore failed with:

ERROR: cannot drop constraint User_pkey on table public."User" because other objects depend on it
DETAIL: constraint UserRole_userId_fkey on table public."UserRole" depends on index public."User_pkey"
HINT: Use DROP ... CASCADE to drop the dependent objects too.
❌ Error: Database restoration failed. Single transaction rolled back.

updated the script to wipe the DB before a restore.

[rngadam]

The restore file fails:

I'm able to run restore-db.sh locally successfully, but in the deploy.yml workflow I'm getting this error:

Step 0: Checking S3 backup bucket write access...
⚠️ Warning: No write access to S3 backup bucket (backups.cv.coderbunker.ca). Skipping pre-deployment backup.
👉 Note: This is expected if you are using a Read-Only staging key for production backups.
Step 0.2: Restoring database from backup-cv.coderbunker.ca-20260122103417.sql.enc...
Loading configuration from .env.cv.staging.coderbunker.ca
Verifying connectivity for restore...
Restoring database backup into cv.staging.coderbunker.ca...
Using specified backup file: backup-cv.coderbunker.ca-20260122103417.sql.enc
Downloading backup-cv.coderbunker.ca-20260122103417.sql.enc...
fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

but with the permission fix, deployment works so I'll look at this issue later.