feat(photos): AI-generated alt text via Workers AI

.env.example

@@ -49,6 +49,16 @@ PREVIEW_SECRET=YOUR_SECRET_HERE
 # (e.g. a local MinIO mock). Real Cloudflare R2 does not need it.
 # R2_FORCE_PATH_STYLE=
 
+# Cloudflare Workers AI: writes alt text for photos (model @cf/google/gemma-4-26b-a4b-it, called over REST).
+# With both set, a photo uploaded with a blank alt gets one generated, the photo edit view shows a
+# "Regenerate alt text" button, and `npm run generate:alt-text` can backfill existing photos.
+# Leave either unset and the feature is off entirely: uploads keep a blank alt, nothing is sent to Cloudflare.
+# CLOUDFLARE_ACCOUNT_ID is the account ID shown in the Cloudflare dashboard.
+# WORKERS_AI_API_TOKEN is an API token for that account; a custom token needs the "Workers AI - Read"
+# and "Workers AI - Edit" permissions. Each generated alt text is a billed model call.
+# CLOUDFLARE_ACCOUNT_ID=
+# WORKERS_AI_API_TOKEN=
+
 # --- Local R2 mock (MinIO) --------------------------------------------------------------
 # Start it with:  npm run dev:r2     (console at http://localhost:9001, minioadmin/minioadmin)
 # Then run the app with the storage vars pointed at MinIO. Easiest via the bundled launch

CMS.md

@@ -18,6 +18,8 @@ This document describes the environment variables and Payload CMS data structure
 | `R2_REGION` | No | Typically `auto` for R2. |
 | `STORAGE_URL` | No | Public URL for R2 (e.g. `https://media.example.com`). No trailing slash. |
 | `NEXT_PUBLIC_STORAGE_URL` | No | Same as `STORAGE_URL`. Exposed to the browser so images load directly from the CDN instead of proxying through Next.js. |
+| `CLOUDFLARE_ACCOUNT_ID` | No | Cloudflare account ID for Workers AI. With `WORKERS_AI_API_TOKEN`, enables AI-generated photo alt text; if either is unset the feature is off entirely. |
+| `WORKERS_AI_API_TOKEN` | No | Cloudflare API token used to call Workers AI over REST. A custom token needs the "Workers AI - Read" and "Workers AI - Edit" permissions. |
 
 ### Storage Behavior
 
@@ -75,7 +77,8 @@ Slug: `media` · Folders: Yes
 |-------|------|-------------|
 | tags | relationship → tags | Tags for photo gallery filtering |
 | displayOrder | number | Order in photos gallery (lower = earlier) |
-| alt | text | Alt text for accessibility |
+| alt | text | Alt text for accessibility. Left blank on upload, it is written by a vision model when Workers AI is configured (see **Alt text** below) |
+| altSource | select | `ai`, `manual` or unset: who wrote `alt`. Read-only, maintained by a hook |
 | caption | richText | Image caption |
 | (upload) | — | File upload with focal point, image sizes: thumbnail, square, small, medium, large, xlarge, og |
 
@@ -92,6 +95,15 @@ Slug: `media` · Folders: Yes
 - **Delivery**: `ImageMedia` builds a real `srcSet` from `sizes.*` even when images are served straight from the CDN (`NEXT_PUBLIC_STORAGE_URL`), via a Next.js image `loader` that maps each requested width to the closest rendition. Callers that render into a known box (e.g. `PhotoGrid` tiles) pass an explicit `size`. The rendition ladder is capped at `xlarge` (1920px WebP); only `priority` (LCP) images may fall back to the raw original, so a missing/inaccurate `size` can never silently serve a multi-MB file.
 - **Backfill**: existing photos are reprocessed with `npm run reprocess:photos` (`-- --dry-run` / `-- --limit N` / `-- --force` / `-- --continue-on-error`). It re-downloads each original, has Payload rebuild `sizes.*`, then verifies every object resolves from storage before moving on and stops on the first failure. AVIF/HEIC sources decode slowly, so large batches take time. Run it against the local MinIO mock (`npm run dev:r2`) first and confirm bucket object counts before pointing it at R2.
 
+**Alt text** (`src/collections/Photos/altText/`): optional, enabled by setting both `CLOUDFLARE_ACCOUNT_ID` and `WORKERS_AI_API_TOKEN`. Descriptions come from the Workers AI vision model `@cf/google/gemma-4-26b-a4b-it`, called over REST; the photo's tags and location are passed along as hints.
+
+- **On upload**: a photo created with a blank `alt` is described during the upload itself, so the edit form already holds the text. Text typed at upload is kept as is. Existing photos are never changed by an ordinary save.
+- **`altSource`**: `ai` when the model wrote `alt`, `manual` once it is typed or edited by hand, unset while `alt` is blank and on photos whose text predates the feature (those count as hand-written). It is what lets a regeneration pass replace machine-written text without touching hand-written text.
+- **Regenerate**: the edit view has a "Regenerate alt text" button under the alt field (hidden when Workers AI is not configured). It POSTs to `/api/photos/:id/alt-text` (authenticated), which replaces whatever is there and saves it immediately; the button asks first when the current text is hand-written.
+- **Retry queue**: if the model call fails during an upload, the upload still succeeds with a blank `alt` and a `generateAltText` job is queued on the `alt-text` queue. `jobs.autoRun` in `payload.config.ts` drains that queue every 2 minutes inside the server process; the job only ever fills a blank `alt` and retries with exponential backoff.
+- **Backfill**: `npm run generate:alt-text` (`-- --dry-run` / `-- --limit N` / `-- --regenerate` / `-- --continue-on-error`) fills photos whose `alt` is blank; `--regenerate` also redoes `ai` text. Hand-written text is never touched. It makes one billed model call per photo and stops on the first failure.
+- **Caches**: writes made outside a Next.js request (the backfill script, the retry job) cannot revalidate the photos sitemap or the new-tab manifest. Those pick the text up on the next photo change in the admin, or the manifest's 24h backstop.
+
 ### Categories
 
 Slug: `categories`

package.json

@@ -11,6 +11,7 @@
     "dev:db": "docker compose -f docker-compose.yml -f docker-compose.dev.yml up mongo -d",
     "dev:r2": "docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile r2 up minio minio-setup -d",
     "dev:prod": "cross-env NODE_OPTIONS=--no-deprecation rm -rf .next && npm run build && npm run start",
+    "generate:alt-text": "cross-env NODE_OPTIONS=--no-deprecation tsx scripts/generate-alt-text.ts",
     "generate:importmap": "cross-env NODE_OPTIONS=--no-deprecation payload generate:importmap",
     "generate:types": "cross-env NODE_OPTIONS=--no-deprecation payload generate:types",
     "ii": "cross-env NODE_OPTIONS=--no-deprecation npm install",

scripts/generate-alt-text.ts

@@ -0,0 +1,201 @@
+/**
+ * Backfill script: write AI-generated alt text for existing Photos that have none (see
+ * src/collections/Photos/altText). New uploads get theirs during the upload itself; this
+ * covers everything uploaded before that existed, or while Workers AI was not configured.
+ *
+ * Each photo is one request to Cloudflare Workers AI (`@cf/google/gemma-4-26b-a4b-it`),
+ * so it needs CLOUDFLARE_ACCOUNT_ID and WORKERS_AI_API_TOKEN (see .env.example).
+ *
+ * Runs standalone (outside Next.js), same pattern as scripts/reprocess-photos.ts.
+ *
+ * ─────────────────────────────────────────────────────────────────────────────────────────
+ * WHAT IT WILL AND WON'T TOUCH
+ *
+ * - Default: only photos whose `alt` is blank (missing, null or '').
+ * - `--regenerate`: also photos whose `altSource` is 'ai', replacing text an earlier run (or
+ *   the upload hook) generated.
+ * - Never hand-written text. `generatePhotoAltText` re-reads each photo right before the
+ *   model call and skips it if it has since been given alt text it may not replace, so text
+ *   typed in the admin while this runs is safe.
+ *
+ * Only `alt` and `altSource` are written; no file is uploaded and `sizes.*` is not rebuilt.
+ *
+ * The run STOPS on the first failure (pass `--continue-on-error` to override): a bad token
+ * or an exhausted quota fails every photo the same way, and there is no point finding that
+ * out several hundred requests later.
+ *
+ * COST: every generated alt text is a billed model call. Cloudflare's model page lists
+ * $0.10 per million input tokens and $0.30 per million output tokens (about $0.00003 a photo); `--dry-run` shows how
+ * many photos a run would send, and `--limit` lets you try a handful and read the real usage
+ * off the Workers AI dashboard before committing to the lot.
+ * ─────────────────────────────────────────────────────────────────────────────────────────
+ *
+ * Usage:
+ *   npm run generate:alt-text -- --dry-run             # list what would be processed, call nothing
+ *   npm run generate:alt-text -- --limit 25            # process at most 25 photos, then stop
+ *   npm run generate:alt-text -- --regenerate          # also redo AI-written alt text
+ *   npm run generate:alt-text -- --continue-on-error   # don't stop on the first failed photo
+ *   npm run generate:alt-text                          # process every photo with blank alt text
+ */
+import 'dotenv/config'
+import { createLocalReq, getPayload, type Where } from 'payload'
+import config from '../src/payload.config'
+import { generatePhotoAltText } from '../src/collections/Photos/altText/generate'
+import { ALT_TEXT_MODEL, isAltTextConfigured } from '../src/collections/Photos/altText/workersAI'
+
+const FLAGS = ['--dry-run', '--regenerate', '--continue-on-error', '--limit']
+
+const args = process.argv.slice(2)
+const DRY_RUN = args.includes('--dry-run')
+const REGENERATE = args.includes('--regenerate')
+const CONTINUE_ON_ERROR = args.includes('--continue-on-error')
+const limitArg = args.indexOf('--limit')
+const LIMIT = limitArg >= 0 ? Number(args[limitArg + 1]) : Infinity
+const DELAY_MS = 500 // be polite: one model call per photo, strictly one at a time
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
+
+// A mistyped `--dry-run` must not turn into a real, billed run.
+const unknown = args.filter(
+  (arg, i) => !FLAGS.includes(arg) && !(limitArg >= 0 && i === limitArg + 1),
+)
+if (unknown.length) {
+  console.error(`Unknown argument(s): ${unknown.join(' ')}\nKnown flags: ${FLAGS.join(' ')}`)
+  process.exit(1)
+}
+if (limitArg >= 0 && !(Number.isInteger(LIMIT) && LIMIT > 0)) {
+  console.error('--limit needs a positive whole number, e.g. --limit 25')
+  process.exit(1)
+}
+
+/**
+ * Photos this run may write to.
+ *
+ * `exists: false` is the clause doing the work: for a text field the Mongo adapter expands
+ * it to "missing OR null OR ''", which are the three ways a blank `alt` is stored (never set,
+ * cleared through the API, cleared in the admin form). The explicit `equals: ''` is redundant
+ * with it today and only there so an adapter change can't silently drop the empty-string case.
+ *
+ * Not matched: an `alt` of nothing but whitespace. No query operator expresses that, and it
+ * takes someone typing only spaces into the field to create one.
+ */
+const candidatesWhere: Where = {
+  or: [
+    { alt: { exists: false } },
+    { alt: { equals: '' } },
+    ...(REGENERATE ? [{ altSource: { equals: 'ai' } }] : []),
+  ],
+}
+
+async function main() {
+  if (!DRY_RUN && !isAltTextConfigured()) {
+    console.error(
+      'Workers AI is not configured: set CLOUDFLARE_ACCOUNT_ID and WORKERS_AI_API_TOKEN ' +
+        '(see .env.example).\nNothing was changed. Use --dry-run to list the photos without them.',
+    )
+    process.exit(1)
+  }
+
+  const payload = await getPayload({ config })
+
+  // Collect every candidate before the first write. Paging through this query while filling
+  // in `alt` would shrink the result set under the cursor and skip photos.
+  const { docs: candidates } = await payload.find({
+    collection: 'photos',
+    depth: 0,
+    overrideAccess: true,
+    pagination: false,
+    select: { alt: true, altSource: true, filename: true },
+    sort: 'createdAt',
+    where: candidatesWhere,
+  })
+  const batch = candidates.slice(0, LIMIT)
+
+  console.log(
+    `${candidates.length} photo(s) ${REGENERATE ? 'with blank or AI-written' : 'with blank'} alt text` +
+      (batch.length < candidates.length ? `, taking the first ${batch.length} (--limit)` : '') +
+      (DRY_RUN ? '' : `. Model: ${ALT_TEXT_MODEL}`),
+  )
+
+  const label = (photo: (typeof batch)[number]) =>
+    `${photo.id} ${photo.filename ?? '(no filename)'}`
+
+  if (DRY_RUN) {
+    for (const photo of batch) {
+      const current = photo.alt?.trim()
+      console.log(
+        current
+          ? `would regenerate: ${label(photo)} (currently ${JSON.stringify(current)})`
+          : `would generate: ${label(photo)}`,
+      )
+    }
+    if (!isAltTextConfigured()) {
+      console.log(
+        '\nNote: CLOUDFLARE_ACCOUNT_ID / WORKERS_AI_API_TOKEN are not set, so a real run would refuse to start.',
+      )
+    }
+    console.log(`\nDone. (dry run) would process=${batch.length}`)
+    process.exit(0)
+  }
+
+  const { docs: users } = await payload.find({
+    collection: 'users',
+    limit: 1,
+    overrideAccess: true,
+  })
+  if (!users[0]) throw new Error('no admin user found to attribute the alt text writes to')
+  const req = await createLocalReq({ user: users[0] }, payload)
+
+  let generated = 0
+  let skipped = 0
+  let failed = 0
+
+  for (const photo of batch) {
+    try {
+      const result = await generatePhotoAltText({
+        id: photo.id,
+        // 'ai' still fills blanks, and still refuses hand-written text.
+        overwrite: REGENERATE ? 'ai' : 'empty',
+        // Skip the `revalidatePhoto` afterChange hook: `revalidateTag()` throws outside a Next.js
+        // request. The hook catches that now, so without this flag each photo would just log two
+        // warnings, but a throw there is what once skipped the R2 upload hook for a whole
+        // backfill (see scripts/reprocess-photos.ts), so scripts opt out explicitly.
+        // Consequence: this script does NOT refresh the photos sitemap or the new-tab manifest
+        // caches. They pick the new text up on the next photo change made in the admin, or, for
+        // the manifest, its 24h backstop.
+        context: { disableRevalidate: true },
+        req,
+      })
+
+      if (result.status === 'generated') {
+        generated++
+        console.log(`${label(photo)} -> ${JSON.stringify(result.alt)}`)
+      } else {
+        // Deleted, or given alt text by someone else, since the candidates were collected.
+        skipped++
+        console.log(`skipped (${result.reason}): ${label(photo)}`)
+      }
+    } catch (err) {
+      failed++
+      console.error(`FAILED ${label(photo)}:`, err instanceof Error ? err.message : err)
+      if (!CONTINUE_ON_ERROR) {
+        console.error('\nStopping on first failure (pass --continue-on-error to override).')
+        break
+      }
+    }
+
+    await sleep(DELAY_MS)
+  }
+
+  const remaining = candidates.length - generated - skipped - failed
+  console.log(
+    `\nDone. generated=${generated} skipped=${skipped} failed=${failed}` +
+      (remaining > 0 ? ` remaining=${remaining}` : ''),
+  )
+  process.exit(failed > 0 ? 1 : 0)
+}
+
+main().catch((err) => {
+  console.error('generate-alt-text failed:', err)
+  process.exit(1)
+})

src/collections/Photos.ts

@@ -12,6 +12,9 @@ import { anyone } from '../access/anyone'
 import { authenticated } from '../access/authenticated'
 import { listGallery } from '../components/ListGallery/config'
 import { appendPrefixToCollectionBeforeChangeHook } from '../hooks/uploadPrefixed'
+import { generateAltTextEndpoint } from './Photos/altText/endpoint'
+import { generateAltTextOnUpload, queueAltTextRetry } from './Photos/altText/hooks'
+import { isAltTextConfigured } from './Photos/altText/workersAI'
 import { revalidateDelete, revalidatePhoto } from './Photos/hooks/revalidatePhoto'
 import { transcodeHeicUpload } from './Photos/hooks/transcodeHeicUpload'
 
@@ -26,10 +29,11 @@ export const Photos: CollectionConfig = {
     components: listGallery(),
   },
   folders: true,
+  endpoints: [generateAltTextEndpoint],
   hooks: {
     beforeOperation: [transcodeHeicUpload],
-    beforeChange: [appendPrefixToCollectionBeforeChangeHook('photos')],
-    afterChange: [revalidatePhoto],
+    beforeChange: [appendPrefixToCollectionBeforeChangeHook('photos'), generateAltTextOnUpload],
+    afterChange: [revalidatePhoto, queueAltTextRetry],
     afterDelete: [revalidateDelete],
   },
   access: {
@@ -60,6 +64,44 @@ export const Photos: CollectionConfig = {
       name: 'alt',
       type: 'text',
       //required: true,
+      admin: {
+        description:
+          'Left blank on upload, this is written by a vision model (when Workers AI is configured).',
+      },
+    },
+    {
+      name: 'generateAltText',
+      type: 'ui',
+      admin: {
+        components: {
+          Field: {
+            // Env vars never reach the browser, so the server decides whether to show the button.
+            clientProps: { enabled: isAltTextConfigured() },
+            path: '@/collections/Photos/altText/GenerateAltTextButton#GenerateAltTextButton',
+          },
+        },
+        // A `ui` field would otherwise be offered as an (empty) list column.
+        disableListColumn: true,
+      },
+    },
+    {
+      // Maintained by the `generateAltTextOnUpload` hook, never by the client: it is what lets
+      // a regeneration pass replace machine-written text without touching hand-written text.
+      name: 'altSource',
+      type: 'select',
+      options: [
+        { label: 'AI-generated', value: 'ai' },
+        { label: 'Hand-written', value: 'manual' },
+      ],
+      access: {
+        create: () => false,
+        update: () => false,
+      },
+      admin: {
+        position: 'sidebar',
+        readOnly: true,
+        description: 'Who wrote the alt text. Editing the alt text marks it hand-written.',
+      },
     },
     {
       name: 'caption',