feat(documents): document_read tool + document-context analytics (offload PR-1)

[philmerrell]

Off develop @ c28ccdbd (the compaction stack #1125 → #1128 → #1129 → #1131 → #1132 is merged, so nothing here is stacked). Spec: docs/specs/document-context-offload.md — PR sequence re-cut in §5, field design in §6.1, decisions in §8.

Why

Attachment conversations are 11% of prod sessions and 31% of model spend. Of the 95 September sessions that peaked over 100k tokens, 25 had an attachment in their last three turns, several of them 1–3-turn sessions at 280k–530k that are one huge upload; a compaction cut cannot touch those (thresholds spec §3.6 keeps attachments out of scope on purpose). And a correctness bug: on restore, _strip_document_bytes replaces every inline document with a contentless placeholder, so a returning user's document is silently gone (14% same-file re-upload rate, 87% byte-identical). Four of five attachment sessions rebuild their agent every turn, so the loss fires on turn 2, not after an idle gap.

The spec's PRs 1–3 are the fix and ship as a unit; this is PR-1, the retrieval primitive the digest (PR-2) and the rehydrated restore (PR-3) point at. The explicit change from the spec's original plan: the analytics that were PR-7 ship here, so the cost work (PR-4) is measured from the first day and its ship/widen-pinning/abandon decision is decidable from stored rows.

What

• document_read (agents/builtin_tools/document_read_tool.py, service in apis/shared/files/document_read.py). Modelled on Strands' retrieve_offloaded_content: no args lists the session's readable documents; upload_id + page_range returns a PDF page range re-assembled server-side into a native document block (text layer plus page images, not flattened text) with a unique Bedrock-safe name and a page-numbering note; upload_id + pattern greps the text layer and returns matching lines with page (PDF) or line/paragraph (text, DOCX) numbers; upload_id alone returns a page index for PDFs or bounded text for DOCX/text with offset continuation. max_pages default 8, hard cap 20; matches capped; DOCX via a stdlib zipfile + ElementTree extractor, PDF via the already-pinned pypdfium2. No new dependency.
• Gated on session state, not the picker. Built when this turn attaches an upload or the session has a READY PDF/DOCX/TXT/MD/HTML row (one SessionIndex query, positive answers memoized). Never on enabled_tools: the workspace_files catalog key is granted to no prod role, so an RBAC gate would ship the recovery path dark. DOCUMENT_TOOL_IDS is recorded and deliberately not in INJECTED_TOOL_IDS. Kill switch DOCUMENT_READ_ENABLED=false.
• Cache key, not cache veto. The tool's presence is a document_tools element of _create_cache_key, so an agent cached before the first upload is never served without it, while an attachment session that could keep a warm agent still does (a veto would have pushed the ~19% warm attachment sessions onto the turn-2 strip). The gate is monotonic, so the key flips once, on the attach turn, when restored history has no document to lose; the resume path recomputes the same gate.
• Offloader exemption. document_read results are exempt from the tool-result offloader (OFFLOAD_EXEMPT_TOOLS in the mixin plus the plugin's should_offload); offloading the slice the model just asked for would undo the read.
• Analytics, content-free by construction (ids, counts, sizes, token estimates, format-enum keys only; the content-policy test walks the new projections and models):
  • Per-call fields on the C# row, next to prefixTokens / compactionEvents: hasDocuments, documentCount, documentTokens, documentDigests, documentsAttached, documentSlices, documentSliceTokens, documentMime, and documentReads {calls, pages, bytes} from a new AfterToolCallEvent tally in ContextLedgerHook.
  • document_stripped compaction-ledger event (documents, documentTokens) recorded by _strip_document_bytes, so the defect's reach is measured before PR-3 removes it; document_rehydrated / document_offload (with cacheGapSeconds) kinds registered for PR-3/PR-4.
  • Session rollups fullDocumentCalls, digestOnlyCalls, documentReadCalls, documentReadPages.
  • DocumentRead / DocumentReadPages / DocumentReadBytes EMF in AgentCoreStack/Compaction with mode / format properties.
  • Anatomy rows and the profile carry the fields (dataCoverage.documents); the SPA anatomy page shows a doc / digest / +Np badge per call, a Documents line in the expanded row, and a consumption summary under the Attachments card.

What this does not change

• Nothing about how documents enter context or what restore does with them: the placeholder strip is unchanged (it now also records an event). PR-2 adds the digest, PR-3 rehydrates, PR-4 offloads.
• No RBAC, catalog or picker entry; no CDK change (the flag is default-on, unset means on).
• No citations config (§6 probe note: a standalone product question).
• The tool census and every existing ledger field are untouched; new fields are additive extras on the same writes.

Tests

New: tests/shared/test_document_read.py (hand-built multi-page PDFs and DOCX archives through real pypdfium2: page identity under re-assembly, max_pages and hard cap, past-the-end clipping, pattern with page numbers, literal fallback for a bad regex, bounded index, DOCX offset continuation and paragraph numbers, text-family delegation, non-document refusal, listing and gate); tests/agents/builtin_tools/test_document_read_tool.py (factory, error surfacing, content-free EMF, routes gate incl. memo bound and fail-closed lookup, kill switch, cache-key element, offloader exemption); tests/agents/main_agent/session/test_document_context.py (footprint summary, strip event, ledger documentReads); tests/apis/app_api/admin/costs/test_document_fields.py (row decode incl. Decimal, anatomy, profile rollups from rows and from the session row, write-side deltas). SPA spec extended for the new page helpers.

Backend suite in this worktree: 8399 passed, 3 skipped, 15 failed + 8 files not collected — every one of the 23 is ModuleNotFoundError: openpyxl in the local venv (pinned in pyproject.toml, not installed here; I did not install packages). None touch this change; CI's install will settle it. The SPA spec could not be executed locally for the same reason (docx-preview / pptx-preview missing from the available node_modules), so the page changes are typechecked by CI only — please watch that job.

What remains

PR-2 (digest at upload, persisted on FileMetadata), PR-3 (restore rehydrates digest + handle; document_stripped → document_rehydrated), PR-4 (offload trigger, pinning, percentage rollout, document_offload with cache gap; age old document_read slices), PR-5 (truncation cache-live guard), PR-6 (aggregate inline-bytes guard), PR-7 (a content-free thumbs signal keyed on (sessionId, messageId) — no feedback surface exists today, so the outcome join the kaizen reviews keep asking for is designed but not buildable yet).

🤖 Generated with Claude Code