Release/1.21.0

.claude/launch.json

@@ -19,7 +19,8 @@
         "main.py"
       ],
       "port": 8000,
-      "cwd": "backend/src/apis/app_api"
+      "cwd": "backend/src/apis/app_api",
+      "autoPort": false
     },
     {
       "name": "inference-api",
@@ -28,7 +29,8 @@
         "main.py"
       ],
       "port": 8001,
-      "cwd": "backend/src/apis/inference_api"
+      "cwd": "backend/src/apis/inference_api",
+      "autoPort": false
     },
     {
       "name": "docs-site",

.claude/skills/cutting-a-release/SKILL.md

@@ -181,6 +181,67 @@ flag, or skip path**. So a failed two-index deploy costs two patch releases
 
 ---
 
+## 1b. Pre-merge prerequisite — data backfills
+
+A backfill script is not code that runs itself. Someone has to run it, against
+each environment, in a window that matters. **If this release adds one, the
+release notes must name it** — that is the only place an operator looks.
+
+### Why this is its own gate
+
+The failure is silent, and in the same shape as §1. Backfills populate a sparse
+index or a new attribute, and the read path does not error when the data is
+missing — a sparse index returns **fewer rows**, not an exception. An unrun
+backfill therefore looks like a short list, not an outage: the tool catalog
+missing half its tools, the artifact library missing the older entries. Nothing
+goes red, and no alarm fires.
+
+`develop` cannot surface it either. Whoever wrote the script ran it against dev
+by hand the day they wrote it, so dev is always fine. **Prod is the environment
+with nobody assigned**, and the release is the last moment the instruction can
+still reach someone.
+
+### How to check
+
+CI enforces this on every PR into `main`
+(`.github/workflows/pending-backfills.yml`). Locally:
+
+```bash
+node scripts/release/check-pending-backfills.mjs
+```
+
+It diffs `backend/scripts/backfill_*.py` between `origin/main` and your branch
+and fails when a script added in the range is not named in `RELEASE_NOTES.md`.
+
+It cannot verify the backfill was actually **run** — no CI job can know that.
+It guarantees the instruction reaches the person who can.
+
+### What to write
+
+Name the script, the command, and the environments. Not "a backfill is
+required" — that is the note that sends someone digging through `git log`:
+
+> **Manual step — run before/with this deploy.** Populates `EntityTypeIndex`
+> for tool rows written before it existed. Idempotent; dry-run by default.
+>
+> ```bash
+> AWS_PROFILE=<env> python backend/scripts/backfill_tool_catalog_index.py \
+>     --table <prefix>-app-roles --region us-west-2 --apply
+> ```
+>
+> Verify `skipped=0 failed=0` and that the index item count matches the tool
+> count before considering the deploy complete.
+
+### Ordering, when the release also switches a read onto the backfilled data
+
+Deploy → backfill → *then* the read. If the release carries both the backfill
+and the code that depends on it, the backfill runs **inside the release
+window**, not afterwards. A fresh deployment is exempt only if its seed path
+already writes the new keys — check, do not assume: `seed_bootstrap_data.py`
+hand-builds its items rather than going through the model.
+
+---
+
 ## 2. Branch workflow
 
 ```bash

.claude/skills/kaizen-research/SKILL.md

@@ -41,10 +41,23 @@ Friday early morning (~6am MT). `kaizen-review-prep` runs ~2 hours later (~8am M
 
    We carry custom code that exists only because Strands' caching support is
    Bedrock/Anthropic-shaped. Upstream is converging on a provider-agnostic
-   `CacheConfig`, so this is a **subtraction** item: each upstream landing should
-   delete code here, not add it. Check every run, and check it for *all* cacheable
-   families — Anthropic, OpenAI/GPT, and any newly cacheable model — not just the
-   one that prompted this.
+   `CacheConfig`, so this is *mostly* a **subtraction** item: each upstream landing
+   should delete code here, not add it. Check every run, and check it for *all*
+   cacheable families — Anthropic, OpenAI/GPT, and any newly cacheable model — not
+   just the one that prompted this.
+
+   ⚠️ **Amended 2026-09-11: it is not subtraction-only, and the second half is now
+   the more valuable one.** Two kinds of drift matter, and only the first deletes
+   code. **(i) Convergence** — upstream absorbs something we hand-roll; propose the
+   swap. **(ii) Coverage** — a Bedrock family gains prompt caching that *neither*
+   we nor upstream will use, because both gate on the same `_cache_strategy`
+   string test for `"claude"`/`"anthropic"`. Measured that day: **Nova Micro
+   accepts an explicit system cachePoint and honors it** (7,203 input tokens → 2,
+   7,201 cache-written) while `bedrock_cache_points_supported()` refuses it. That
+   is caching left on the floor, it grows every time AWS ships a cacheable model,
+   and no test can catch it — a string predicate keeps answering the same thing
+   while the platform moves underneath it. Coverage findings are **additions**;
+   file them anyway.
 
    Our carried code, and what would retire it:
 
@@ -55,6 +68,7 @@ Friday early morning (~6am MT). `kaizen-review-prep` runs ~2 hours later (~8am M
    | `build_prompt_cache_key()` in `bedrock_responses.py` | `strands/models/_openai_cache.py::apply_cache_config` — **already on upstream main**, maps `CacheConfig.cache_key` → `prompt_cache_key`. Adopt on the next pin bump. |
    | `apply_explicit_prompt_cache()` (breakpoints) | No upstream equivalent yet — `apply_cache_config` emits no `prompt_cache_breakpoint`. Ours is OFF by default (measured 57% worse); don't re-enable without re-running the probe. |
    | `cache_ttl_seconds_for()` in `observability/prompt_cache.py` | A model-derived TTL upstream. Note `apply_cache_config` maps ttl to `prompt_cache_retention` (`in_memory`/`24h`), *not* GPT-5.6's `prompt_cache_options.ttl: "30m"` — so these are not yet the same concept. |
+   | `bedrock_cache_points_supported()` + the hand-placed system cachePoint | **NOTHING upstream can retire this — settled by measurement 2026-09-11, do not re-propose.** `format_request` copies `system_prompt_content` verbatim and `_apply_system_cache_ttl` never removes a point, so the block reaches Bedrock, which answers `AccessDeniedException` on a model that can't cache. The rejection is Bedrock's, not the SDK's. Only its `tools_ttl` half is redundant (upstream applies the same test at `bedrock.py:579`). |
 
    Each run, answer:
    - Does the pinned Strands version now ship `_openai_cache.py` / a `CacheConfig`
@@ -64,7 +78,17 @@ Friday early morning (~6am MT). `kaizen-review-prep` runs ~2 hours later (~8am M
      that maps onto something we do manually?
    - Any new Bedrock model family with prompt caching? Confirm which API surface
      serves it — GPT-5.6 caches **only** over the Responses API, and the same model
-     over Converse caches not at all.
+     over Converse caches not at all. Then **measure it, don't read it**:
+
+         cd backend
+         AWS_PROFILE=dev-ai uv run python scripts/probe_bedrock_cache_point_support.py \
+             --model-id <the new model id>
+
+     One command, ~$0.01, and it answers both layers at once — what the pinned
+     Strands emits for that id, and whether Bedrock accepts and honors an explicit
+     cachePoint. A row where Bedrock caches but `_cache_strategy` is `None` is a
+     coverage finding (ii) — file it. Use `--offline-only` for a free check of the
+     SDK half alone after a pin bump.
    - Movement on #3546 / #4193, or a new `Usage` convention. Both change what our
      cost math may assume.
 
@@ -73,6 +97,15 @@ Friday early morning (~6am MT). `kaizen-review-prep` runs ~2 hours later (~8am M
    in a live measurement. `backend/scripts/probe_gpt56_cache_rates.py` is the gate:
    beat the current arm, measured, before switching.
 
+   ⚠️ **A caching A/B measures nothing if its prefix is under the model's minimum.**
+   Below that floor Bedrock **silently ignores** the cache point — no error, no
+   cache buckets, a result indistinguishable from "unsupported". Measured on Haiku
+   4.5 in us-west-2 (2026-09-11): 2,351 and 3,911 tokens wrote **nothing**, 5,202
+   wrote in full — a real floor of **4,096**, twice the 2,048 the first-party
+   Anthropic docs publish for Haiku. Do not trust a vendor-documented minimum;
+   bracket it. This already produced one false "does not cache" against our own
+   production model.
+
 3. **Reference repo — `aws-samples/sample-strands-agent-with-agentcore`**
    - https://github.com/aws-samples/sample-strands-agent-with-agentcore/commits/main
    - Diff the last 7 days (or "since last research run" — whichever is longer). Identify new patterns, removed approaches, or fixes that map to constructs in this repo: agent setup, tool registration, AgentCore Identity flows, Memory configuration, Gateway/MCP wiring.