feat(documents): DocumentDigest built at upload and persisted on FileMetadata (offload PR-2)

[philmerrell]

Stacked on #1137 (PR-1). Merge in order; this PR's diff is the one commit on top. CI runs only for PRs targeting develop/main, so this one shows no checks until PR-1 merges and it is re-targeted — the suite result below is the local run.

Why

The offload design (spec §4) hands every turn after the attach turn a digest plus a document_read handle instead of the document's bytes. PR-1 built the handle. This builds the digest: once, at upload, off the model path, so PR-3 can rehydrate a restored document as a digest instead of today's contentless placeholder and PR-4 can swap an inline document for one without paying for extraction on a chat turn.

What

• DocumentDigest (apis/shared/files/document_digest.py), two parts on purpose:
  • Outline, deterministic. Page / paragraph / line count, headings (numbered, ALL CAPS, or title-cased short lines; markdown #) with the unit they start on, table and figure mentions, character count, and a text sample spread across the document. PDF text via the already-pinned pypdfium2, DOCX via PR-1's stdlib extractor, text family from the bytes (HTML tags stripped). A document with no detectable headings gets an outline sampled from page first-lines; outlines past 40 entries are sampled evenly.
  • Abstract, cheap model, fail-open. 3–5 sentences from Nova Micro (DOCUMENT_DIGEST_MODEL_ID) over the sample plus outline — the same text model the tool-batch summaries and compaction summary use, so no new model access. A model error or a truncated generation leaves the digest with an outline and no abstract, never without an outline.
• render_digest produces the <document-digest name upload_id format pages tables figures> block PR-3 will put in context, under a hard budget (DOCUMENT_DIGEST_MAX_TOKENS, 1,500): sections dropped from the end first, then the abstract truncated, the opening tag (the document_read handle) always kept. The estimate is stored as digest.tokens, so the ceiling is a stored fact per file.
• Persisted on FileMetadata.digest as a plain map (FileUploadRepository.update_file_digest, SET digest, tolerant of a row deleted mid-build). complete_upload schedules the build as a strong-referenced fire-and-forget task for document uploads only (not spreadsheets, decks or images); the upload response does not wait. DOCUMENT_DIGEST_ENABLED=false skips it.
• Content policy. digest.abstract and digest.sections are denylisted (model prose and heading text); digest.status / format / count / tokens are projected, and the admin attachment profile reports digested and digestTokens so coverage and the per-file cost are visible without reading a document.
• EMF DocumentDigestGenerated / DocumentDigestTokens / DocumentDigestMs in AgentCoreStack/Compaction with format and outcome (ready / no_abstract / failed) — the p95 extractor latency the spec's gate asks for is read from here in dev.
• Spec: PR-2 row marked built; §8 decisions 12–15.

What this does not change

• Nothing in the chat path reads the digest yet. Restore still writes the placeholder; PR-3 changes that and must generate lazily when digest is absent — uploads that predate this PR and agent-written files (workspace_write, the Word / Excel / PowerPoint tools register rows directly) have none.
• No CDK change: the app API task role already has bedrock:InvokeModel (title generation, api-converse).
• No RBAC, catalog or picker entry; nothing on the model path.

Tests

tests/shared/test_document_digest.py (heading heuristic incl. the one-word-paragraph case, PDF/DOCX/markdown/HTML outlines with anchors and table/figure counts, first-line fallback, even sampling past the cap, cross-document text sample, the rendered block's shape and escaping, budget trimming order, a 200-page PDF digest under 1,500 tokens, the abstract call's model id and prompt, truncated/failed generations, build_digest fail-open on extraction and abstract, class-name-only errors, content-free EMF, kill switch); tests/apis/app_api/test_file_digest_scheduling.py (documents only, task lifecycle, build/S3/repository failures never escape, no-loop no-op); update_file_digest round trip on the moto repository.

Local backend suite (with a stub openpyxl on PYTHONPATH for verification only, since this venv lacks the pin; nothing committed): 8866 passed, 3 skipped; the only 33 failures are the two spreadsheet-preview test files, which need the real openpyxl the stub cannot stand in for. CI's install settles those.

🤖 Generated with Claude Code