From 8e175398ab64232350d42df1201dde11081855fb Mon Sep 17 00:00:00 2001 From: Ivan Morgillo Date: Tue, 7 Jul 2026 09:06:39 +0200 Subject: [PATCH 1/4] feat(compose-agent): write-mode acceptance evals + CI validation (4.4.0) Closes the last 4.3.1-cross-review follow-up: compose-agent had no evals of its own, so its authoring rules were only exercised via the audit's acceptance cases. - skills/compose-agent/evals/evals.json: 5 write-mode cases that ask the skill to WRITE Compose and grade the result. Cases 0-2 mirror the 4.3.x audit rules (cross-phase back-write, Strong Skipping false leads, snapshot self-invalidation); 3-4 cover phase-correct reads and lifecycle-aware Flow collection. - bin/ci: structurally validates every skills//evals/evals.json (JSON, unique numeric ids, required fields, non-empty expectations) and asserts the compose-agent suite keeps covering the three authoring rules, so the authoring path can't drift from the audit scoring path. Negative-tested. - SKILL.md: new Acceptance Evals section. - Versions: compose-agent -> 4.4.0; jetpack-compose-audit unchanged at 4.3.2. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_016ysskXsKpwEbs1n9CX1ScY --- CHANGELOG.md | 11 ++++ README.md | 8 ++- bin/ci | 52 +++++++++++++++ docs/release-notes-4.4.0.md | 32 ++++++++++ .../compose-agent/.claude-plugin/plugin.json | 2 +- .../compose-agent/.cursor-plugin/plugin.json | 2 +- skills/compose-agent/SKILL.md | 6 +- skills/compose-agent/evals/evals.json | 64 +++++++++++++++++++ 8 files changed, 173 insertions(+), 4 deletions(-) create mode 100644 docs/release-notes-4.4.0.md create mode 100644 skills/compose-agent/evals/evals.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 553f132..b867ae7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,17 @@ Full release history for the Compose Skill Suite. The newest release is summarised under **What's new** in the [README](./README.md). +### 4.4.0 — 2026-07-07 + +**`compose-agent` — write-mode acceptance evals.** + +Closes the last follow-up from the 4.3.1 cross-review: the authoring skill had no evals of its own, so its new rules were only exercised through the audit's acceptance cases. + +- **New suite.** `skills/compose-agent/evals/evals.json` — five write-mode cases that ask the skill to *write* Compose and grade the result. Cases 0–2 lock in the 4.3.x authoring rules (cross-phase back-write avoidance, Strong Skipping false-lead avoidance, snapshot self-invalidation avoidance); cases 3–4 cover phase-correct reads and lifecycle-aware Flow collection. Unlike the audit evals (which grade an audit of existing code), these grade generated code. +- **CI enforcement.** `bin/ci` now structurally validates every `skills//evals/evals.json` (JSON, unique numeric ids, required fields, non-empty expectations) and asserts the compose-agent suite keeps covering the three 4.3.x authoring rules — so the authoring path can't silently drift from the audit scoring path. +- **`SKILL.md`.** New *Acceptance Evals* section pointing at the suite. +- **Versions.** `compose-agent` → `4.4.0`. `jetpack-compose-audit` unchanged at `4.3.2`. + ### 4.3.2 — 2026-07-07 **Patch — one missed Strong Skipping caveat.** diff --git a/README.md b/README.md index 16c3b0b..bde2515 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Claude Code plugin

-**`compose-agent` 4.3.2 · 2026-07-07** — Authoring guidance so the coding-agent skill *warns while you write*: **Never Back-Write Across Phases** (layout callbacks / snapshot-collection mutation feeding composition) and **Optimizations That Do Nothing** (the false leads). The 4.3.0 audit catches these after the fact; 4.3.1 stops you writing them. +**`compose-agent` 4.4.0 · 2026-07-07** — Authoring guidance so the coding-agent skill *warns while you write* — **Never Back-Write Across Phases** and **Optimizations That Do Nothing** — now backed by its own [write-mode acceptance evals](./skills/compose-agent/evals/evals.json) (CI-enforced to track the audit's rules). The 4.3.0 audit catches these after the fact; `compose-agent` stops you writing them. **`jetpack-compose-audit` 4.3.2 · 2026-07-07** — Cross-phase back-write detection (axis 3: layout callbacks writing state read in composition), a related composition-phase self-invalidation check (snapshot collections mutated in a composable body), and a **False Leads** scoring guard so the auditor stops crediting no-op "recomposition fixes." Adapted from [`chrisbanes/skills`](https://github.com/chrisbanes/skills) (Apache-2.0). @@ -31,6 +31,12 @@ Authored and cross-reviewed with every frontier model — Claude Opus 4.8, GPT-5 ## What's new +### 4.4.0 — 2026-07-07 + +**`compose-agent` — write-mode acceptance evals.** New [`skills/compose-agent/evals/evals.json`](./skills/compose-agent/evals/evals.json): five cases that ask the skill to *write* Compose and grade the output (cross-phase back-write avoidance, Strong Skipping false leads, snapshot self-invalidation, phase-correct reads, lifecycle-aware Flow collection). `bin/ci` now validates every eval suite's structure and asserts the compose-agent cases keep covering the 4.3.x authoring rules — so the authoring path can't drift from the audit path. `compose-agent` → `4.4.0`; `jetpack-compose-audit` stays `4.3.2`. + +For release detail, see [`docs/release-notes-4.4.0.md`](./docs/release-notes-4.4.0.md). + ### 4.3.2 — 2026-07-07 **Patch.** Fixed one Strong Skipping caveat in `jetpack-compose-audit/scoring.md` that the 4.3.1 sweep missed (its "opt-out path" wording dodged the string match): `@NonSkippableComposable` opts a composable out of *skipping* but does not disable *lambda memoization* — only `@DontMemoize` / SSM-off does. Both skills → `4.3.2`. diff --git a/bin/ci b/bin/ci index 7487da7..af2f1fe 100755 --- a/bin/ci +++ b/bin/ci @@ -162,6 +162,58 @@ const auditVersion = JSON.parse(readFileSync('skills/jetpack-compose-audit/.clau if (!auditSkill.includes(`**Skill version:** ${auditVersion}`)) { throw new Error('jetpack-compose-audit SKILL.md version does not match manifest version'); } + +// Eval suites: structural validation for every skills//evals/evals.json. +for (const evalFile of [...files].filter((file) => file.endsWith('/evals/evals.json'))) { + const skillName = evalFile.split('/')[1]; + let suite; + try { + suite = JSON.parse(readFileSync(evalFile, 'utf8')); + } catch (error) { + throw new Error(`${evalFile} is not valid JSON: ${error.message}`); + } + if (suite.skill_name !== skillName) { + throw new Error(`${evalFile} skill_name must be "${skillName}", got "${suite.skill_name}"`); + } + if (!Array.isArray(suite.evals) || suite.evals.length === 0) { + throw new Error(`${evalFile} must have a non-empty "evals" array`); + } + const seenIds = new Set(); + for (const item of suite.evals) { + if (typeof item.id !== 'number' || seenIds.has(item.id)) { + throw new Error(`${evalFile} eval ids must be unique numbers (offender: ${JSON.stringify(item.id)})`); + } + seenIds.add(item.id); + for (const field of ['prompt', 'expected_output']) { + if (typeof item[field] !== 'string' || !item[field].trim()) { + throw new Error(`${evalFile} eval ${item.id} is missing a non-empty "${field}"`); + } + } + if (!Array.isArray(item.files)) { + throw new Error(`${evalFile} eval ${item.id} must have a "files" array`); + } + if (!Array.isArray(item.expectations) || item.expectations.length === 0) { + throw new Error(`${evalFile} eval ${item.id} must have a non-empty "expectations" array`); + } + } +} + +// compose-agent write-mode evals must keep pace with the 4.3.x authoring rules, +// so the authoring path can't silently drift from the audit scoring path. +const composeEvalsPath = 'skills/compose-agent/evals/evals.json'; +if (!files.has(composeEvalsPath)) { + throw new Error('compose-agent must ship write-mode evals at skills/compose-agent/evals/evals.json'); +} +const composeEvalsText = readFileSync(composeEvalsPath, 'utf8'); +for (const [label, needle] of [ + ['cross-phase back-write', /onSizeChanged|onGloballyPositioned|cross-phase/], + ['Strong Skipping false lead', /Strong Skipping[\s\S]*remember|remember[\s\S]*Strong Skipping/], + ['snapshot self-invalidation', /mutableStateMapOf|mutableStateListOf|self-invalidation/], +]) { + if (!needle.test(composeEvalsText)) { + throw new Error(`compose-agent evals must cover the ${label} authoring rule`); + } +} NODE if command -v claude >/dev/null 2>&1; then diff --git a/docs/release-notes-4.4.0.md b/docs/release-notes-4.4.0.md new file mode 100644 index 0000000..157ec41 --- /dev/null +++ b/docs/release-notes-4.4.0.md @@ -0,0 +1,32 @@ +# Release notes — 4.4.0 (2026-07-07) + +**`compose-agent` only.** `jetpack-compose-audit` unchanged at `4.3.2`. + +Closes the last follow-up flagged across the 4.3.x cross-reviews: `compose-agent` had no evals of its own, so its authoring rules were only ever exercised through the *audit's* acceptance cases (which grade an audit of existing code, not code the skill writes). + +## New: write-mode acceptance evals + +`skills/compose-agent/evals/evals.json` — five cases that hand the skill a *writing* task and grade the Compose it produces: + +- **Case 0 — cross-phase back-write avoidance.** Align a value to a label's measured width without writing `onSizeChanged` state read in composition; expects a layout-phase solution. +- **Case 1 — Strong Skipping false leads.** Make a row skip under SSM; expects *no* `remember`-wrapped callbacks and *no* `remember(index)` on pure expressions — real levers only (stable types, keys, verified with recomposition counts). +- **Case 2 — snapshot self-invalidation avoidance.** Build a per-entry lookup with `remember(entries)`, not by mutating a `mutableStateMapOf` during composition. +- **Case 3 — phase-correct reads.** Feed a per-frame `Float` through a `() -> Float` provider or a layout/draw-phase read, keeping composition stable. +- **Case 4 — lifecycle-aware collection.** Collect a `StateFlow` with `collectAsStateWithLifecycle()`, side-effect free, modifier-hygienic. + +Cases 0–2 mirror the `jetpack-compose-audit` scoring rules 1:1, so the authoring path and the audit path stay in lockstep. + +## CI enforcement + +`bin/ci` now: + +- structurally validates **every** `skills//evals/evals.json` — valid JSON, unique numeric ids, non-empty `prompt` / `expected_output`, `files` array, non-empty `expectations`; +- asserts the compose-agent suite keeps covering the three 4.3.x authoring rules (cross-phase back-write, Strong Skipping false lead, snapshot self-invalidation), so a future edit can't quietly drop coverage. + +## Note + +These evals are acceptance data, not a runner — run a model with the skill loaded against each prompt and check every expectation, the same way `jetpack-compose-audit/evals/evals.json` is used. + +## Versions + +`compose-agent` → **4.4.0** (SKILL + both `plugin.json`). `jetpack-compose-audit` stays at `4.3.2`. diff --git a/skills/compose-agent/.claude-plugin/plugin.json b/skills/compose-agent/.claude-plugin/plugin.json index 96b4634..822dd14 100644 --- a/skills/compose-agent/.claude-plugin/plugin.json +++ b/skills/compose-agent/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "compose-agent", - "version": "4.3.2", + "version": "4.4.0", "description": "Agent skill that helps AI coding assistants write modern Jetpack Compose: correct state, effects, performance-aware modifiers, Navigation 3, Paging 3 in Compose, coroutines on lifecycle, animations, UI tests, focus/keyboard navigation, Compose Multiplatform boundaries, Android launch resources, and idiomatic Kotlin. Targets the mistakes LLMs actually make in Compose code.", "author": { "name": "Ivan Morgillo" diff --git a/skills/compose-agent/.cursor-plugin/plugin.json b/skills/compose-agent/.cursor-plugin/plugin.json index 96b4634..822dd14 100644 --- a/skills/compose-agent/.cursor-plugin/plugin.json +++ b/skills/compose-agent/.cursor-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "compose-agent", - "version": "4.3.2", + "version": "4.4.0", "description": "Agent skill that helps AI coding assistants write modern Jetpack Compose: correct state, effects, performance-aware modifiers, Navigation 3, Paging 3 in Compose, coroutines on lifecycle, animations, UI tests, focus/keyboard navigation, Compose Multiplatform boundaries, Android launch resources, and idiomatic Kotlin. Targets the mistakes LLMs actually make in Compose code.", "author": { "name": "Ivan Morgillo" diff --git a/skills/compose-agent/SKILL.md b/skills/compose-agent/SKILL.md index 8351f17..7720001 100644 --- a/skills/compose-agent/SKILL.md +++ b/skills/compose-agent/SKILL.md @@ -6,7 +6,7 @@ allowed-tools: Read, Glob, Grep, Edit, Write, Bash argument-hint: "[focus area, e.g. 'state', 'effects', 'navigation', 'paging', 'lifecycle', 'animation', 'testing', 'focus', 'kmp']" metadata: author: Ivan Morgillo - version: "4.3.2" + version: "4.4.0" --- # Compose Agent @@ -178,6 +178,10 @@ If the user needs any of the above, narrow the scope and say so. - `references/paging.md` — Paging 3 in Compose: when to page vs plain lists, `collectAsLazyPagingItems`, stable `itemKey`, `LoadState` UI, pull-to-refresh, anti-patterns for paginated lazy lists. - `references/kotlin.md` — Kotlin coding conventions and Android Kotlin style the LLM keeps missing. +## Acceptance Evals + +`evals/evals.json` holds write-mode acceptance cases — prompts that ask this skill to *write* Compose, plus the expectations the produced code must satisfy. They mirror the `jetpack-compose-audit` scoring rules (cross-phase back-writes, Strong Skipping false leads, snapshot self-invalidation, phase-correct reads, lifecycle-aware Flow collection) so the authoring path can't drift from the audit path. Run a model with this skill loaded against each prompt and check every expectation. + ## Primary Sources Every rule in this skill traces back to one of: diff --git a/skills/compose-agent/evals/evals.json b/skills/compose-agent/evals/evals.json new file mode 100644 index 0000000..b543184 --- /dev/null +++ b/skills/compose-agent/evals/evals.json @@ -0,0 +1,64 @@ +{ + "skill_name": "compose-agent", + "description": "Write-mode acceptance evals for compose-agent. Unlike jetpack-compose-audit's evals (which grade an audit of existing code), these grade code the skill is asked to WRITE. Cases 0-2 lock in the 4.3.x authoring rules the audit already enforces (cross-phase back-writes, Strong Skipping false leads, snapshot self-invalidation) so the authoring path can't drift from the scoring path; cases 3-4 cover foundational phase/lifecycle rules. Run a model with the skill loaded against each prompt and check the produced Compose satisfies every expectation.", + "evals": [ + { + "id": 0, + "prompt": "Write a Compose row that shows a `label` and, to its right, a `value` that starts exactly 8dp after the label ends — the value's left edge must track the label's actual measured width for any label length. Target Kotlin 2.1 with Strong Skipping on.", + "expected_output": "A single custom `Layout` (or `SubcomposeLayout`) that measures the label and places the value at `label.width + 8dp`, keeping the measured width inside the layout phase. It must NOT capture the label's width from `onSizeChanged` / `onGloballyPositioned` into a `mutableStateOf` / `mutableIntStateOf` that the value then reads in composition.", + "files": [], + "expectations": [ + "Do not write the label's measured width from onSizeChanged / onGloballyPositioned / onPlaced into state that the value reads in composition (that is a cross-phase back-write: measure -> write -> recompose).", + "Solve it in the layout phase: a custom Layout, Modifier.layout { }, or SubcomposeLayout that places the value using the label's measured width.", + "If BoxWithConstraints / SubcomposeLayout appears, it is used only for parent constraints, never to read the sibling label's measured size.", + "The value's horizontal offset is computed during measure/layout, not read from a composition-phase State." + ] + }, + { + "id": 1, + "prompt": "This screen targets Kotlin 2.1 with Strong Skipping on by default. Make `RowItem` skip recomposition when unrelated screen state changes:\n@Composable fun Rows(items: List, onOpen: (Row) -> Unit) {\n LazyColumn { itemsIndexed(items) { i, row ->\n RowItem(row, isFirst = i == 0, onClick = { onOpen(row) })\n } }\n}", + "expected_output": "Guidance/code that targets the real levers — `Row` being a stable type and `RowItem` restartable/skippable, stable list keys — and explicitly does NOT add ceremony that does nothing under Strong Skipping: no wrapping `onOpen`/`onClick` in `remember`, no `remember(i)` around `i == 0`.", + "files": [], + "expectations": [ + "Do not wrap the onClick / onOpen lambda in remember to 'stabilize' it — under Strong Skipping the compiler already memoizes lambdas passed to composables (even with unstable captures).", + "Do not add remember(i) / remember(index) around the pure `i == 0` expression as a recomposition fix.", + "Point at real levers: Row as a stable type (data class with stable/immutable fields, no var), RowItem being skippable, and a stable key = on itemsIndexed.", + "Recommend verifying with Layout Inspector recomposition counts or Compose compiler reports rather than asserting a win from the pattern." + ] + }, + { + "id": 2, + "prompt": "Write a composable `Legend(entries: List)` that renders each entry as a colored dot plus its label, where each dot's color comes from a lookup keyed by `entry.id` built from `entries`.", + "expected_output": "A composable that derives the lookup as a plain `Map` with `remember(entries) { ... }` (or inline), reading it to render. It must NOT build the lookup by mutating a `mutableStateMapOf` / `mutableStateListOf` inside the composable body and reading it back in the same composition.", + "files": [], + "expectations": [ + "Build the lookup with remember(entries) { entries.associate { it.id to it.color } } (or derive inline) — a plain immutable Map keyed on entry.id.", + "Do not mutate a mutableStateMapOf / mutableStateListOf (put / putAll / add / clear / [k] =) inside the composable body and read it in the same composition (composition-phase self-invalidation).", + "No snapshot-state writes happen during composition." + ] + }, + { + "id": 3, + "prompt": "Write a `Parent` composable that owns a scroll- or animation-driven `progress: Float` in 0f..1f and passes it to a `ProgressBar` child that fills that fraction of its width. Keep recomposition off the per-frame hot path. Target Strong Skipping.", + "expected_output": "Parent hoists/owns the progress; the fast-changing value reaches ProgressBar as a `() -> Float` provider or is read in the layout/draw phase (`Modifier.graphicsLayer { }`, `Modifier.drawBehind { }`, or `Modifier.layout { }`), so composition stays stable each frame.", + "files": [], + "expectations": [ + "The per-frame progress is read in layout/draw (lambda modifier / graphicsLayer / drawBehind / layout) or passed as a () -> Float provider — not as a plain Float read in ProgressBar's composition.", + "ProgressBar does not recompose on every frame of the scroll/animation.", + "State is hoisted to Parent (or a state holder), and any Animatable is held in remember and driven from a LaunchedEffect." + ] + }, + { + "id": 4, + "prompt": "Write a `FeedScreen(viewModel: FeedViewModel)` that renders `viewModel.uiState: StateFlow` (a header plus a list of posts). Assume a normal, non-paged list.", + "expected_output": "Collects the StateFlow with `collectAsStateWithLifecycle()`, keeps the composable side-effect free in composition, forwards a `modifier` to the root, and hoists any UI state correctly.", + "files": [], + "expectations": [ + "Collect the StateFlow with collectAsStateWithLifecycle(), not plain collectAsState().", + "No side effects (I/O, navigation, non-remembered mutation) run directly in the composition body.", + "FeedScreen takes modifier: Modifier = Modifier applied to the outermost layout, and parameter order is data -> modifier -> other.", + "Does not reach for collectAsLazyPagingItems() here — this is a plain StateFlow, not PagingData." + ] + } + ] +} From 98c0d42f0cbcb2659348828b19c22c5ddae369d3 Mon Sep 17 00:00:00 2001 From: Ivan Morgillo Date: Tue, 7 Jul 2026 09:15:29 +0200 Subject: [PATCH 2/4] review r1: per-case CI enforcement, case-0 SubcomposeLayout fix, symmetry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-1 cross-review (Cursor, Antigravity; no blockers): - CI enforcement was cosmetic (Cursor + Antigravity): the coverage regexes ran on the whole file text, so the suite description alone satisfied them and cases could be deleted. Now parse the JSON and require at least one eval CASE to match each rule. Negative-tested: dropping case 0/1/2 each fails CI; 3/4 don't (foundational, not locked — matches docs). Tightened the false-lead needle to `memoiz` (the distinctive token) so case 3's incidental "Strong Skipping"+"remember" no longer satisfies it. - Structural validator now rejects empty-string expectations. - Removed the redundant post-loop re-read (reuse the parsed suite map). - Eval case 0: fixed the SubcomposeLayout contradiction — SubcomposeLayout / custom Layout measuring the label are valid; only BoxWithConstraints is called out (it exposes parent constraints, not a sibling's measured size). - Softened the parity claim: cases 0-2 mirror the audit 1:1 and are CI-locked; 3-4 are foundational, no audit counterpart. Cross-referenced audit cases 12-14 by id. - Symmetry: jetpack-compose-audit SKILL.md gains an Acceptance Evals section too. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_016ysskXsKpwEbs1n9CX1ScY --- bin/ci | 21 +++++++++++++-------- skills/compose-agent/SKILL.md | 2 +- skills/compose-agent/evals/evals.json | 6 +++--- skills/jetpack-compose-audit/SKILL.md | 4 ++++ 4 files changed, 21 insertions(+), 12 deletions(-) diff --git a/bin/ci b/bin/ci index af2f1fe..5d41d8f 100755 --- a/bin/ci +++ b/bin/ci @@ -164,6 +164,7 @@ if (!auditSkill.includes(`**Skill version:** ${auditVersion}`)) { } // Eval suites: structural validation for every skills//evals/evals.json. +const evalSuites = new Map(); for (const evalFile of [...files].filter((file) => file.endsWith('/evals/evals.json'))) { const skillName = evalFile.split('/')[1]; let suite; @@ -195,23 +196,27 @@ for (const evalFile of [...files].filter((file) => file.endsWith('/evals/evals.j if (!Array.isArray(item.expectations) || item.expectations.length === 0) { throw new Error(`${evalFile} eval ${item.id} must have a non-empty "expectations" array`); } + if (!item.expectations.every((expectation) => typeof expectation === 'string' && expectation.trim())) { + throw new Error(`${evalFile} eval ${item.id} expectations must all be non-empty strings`); + } } + evalSuites.set(skillName, suite); } // compose-agent write-mode evals must keep pace with the 4.3.x authoring rules, -// so the authoring path can't silently drift from the audit scoring path. -const composeEvalsPath = 'skills/compose-agent/evals/evals.json'; -if (!files.has(composeEvalsPath)) { +// so the authoring path can't silently drift from the audit scoring path. Checked +// per eval CASE (not whole-file text), so the suite description can't satisfy them. +const composeSuite = evalSuites.get('compose-agent'); +if (!composeSuite) { throw new Error('compose-agent must ship write-mode evals at skills/compose-agent/evals/evals.json'); } -const composeEvalsText = readFileSync(composeEvalsPath, 'utf8'); for (const [label, needle] of [ - ['cross-phase back-write', /onSizeChanged|onGloballyPositioned|cross-phase/], - ['Strong Skipping false lead', /Strong Skipping[\s\S]*remember|remember[\s\S]*Strong Skipping/], + ['cross-phase back-write', /onSizeChanged|onGloballyPositioned|onPlaced|cross-phase/], + ['Strong Skipping false lead', /Strong Skipping[\s\S]*memoiz|memoiz[\s\S]*Strong Skipping/i], ['snapshot self-invalidation', /mutableStateMapOf|mutableStateListOf|self-invalidation/], ]) { - if (!needle.test(composeEvalsText)) { - throw new Error(`compose-agent evals must cover the ${label} authoring rule`); + if (!composeSuite.evals.some((item) => needle.test(JSON.stringify(item)))) { + throw new Error(`compose-agent evals must include a case covering the ${label} authoring rule`); } } NODE diff --git a/skills/compose-agent/SKILL.md b/skills/compose-agent/SKILL.md index 7720001..5af2643 100644 --- a/skills/compose-agent/SKILL.md +++ b/skills/compose-agent/SKILL.md @@ -180,7 +180,7 @@ If the user needs any of the above, narrow the scope and say so. ## Acceptance Evals -`evals/evals.json` holds write-mode acceptance cases — prompts that ask this skill to *write* Compose, plus the expectations the produced code must satisfy. They mirror the `jetpack-compose-audit` scoring rules (cross-phase back-writes, Strong Skipping false leads, snapshot self-invalidation, phase-correct reads, lifecycle-aware Flow collection) so the authoring path can't drift from the audit path. Run a model with this skill loaded against each prompt and check every expectation. +`evals/evals.json` holds write-mode acceptance cases — prompts that ask this skill to *write* Compose, plus the expectations the produced code must satisfy. Cases 0–2 mirror the `jetpack-compose-audit` scoring rules 1:1 (cross-phase back-writes, Strong Skipping false leads, snapshot self-invalidation) — `bin/ci` keeps them in lockstep so the authoring path can't drift from the audit path; cases 3–4 add foundational phase-correct reads and lifecycle-aware Flow collection. Run a model with this skill loaded against each prompt and check every expectation. ## Primary Sources diff --git a/skills/compose-agent/evals/evals.json b/skills/compose-agent/evals/evals.json index b543184..baa7ea6 100644 --- a/skills/compose-agent/evals/evals.json +++ b/skills/compose-agent/evals/evals.json @@ -1,6 +1,6 @@ { "skill_name": "compose-agent", - "description": "Write-mode acceptance evals for compose-agent. Unlike jetpack-compose-audit's evals (which grade an audit of existing code), these grade code the skill is asked to WRITE. Cases 0-2 lock in the 4.3.x authoring rules the audit already enforces (cross-phase back-writes, Strong Skipping false leads, snapshot self-invalidation) so the authoring path can't drift from the scoring path; cases 3-4 cover foundational phase/lifecycle rules. Run a model with the skill loaded against each prompt and check the produced Compose satisfies every expectation.", + "description": "Write-mode acceptance evals for compose-agent. Unlike jetpack-compose-audit's evals (which grade an audit of existing code), these grade code the skill is asked to WRITE. Cases 0-2 are the write-mode mirror of jetpack-compose-audit/evals/evals.json cases 12-14 (cross-phase back-write, Strong Skipping false lead, snapshot self-invalidation) — same rules, opposite direction (write vs audit) — so the authoring path can't drift from the scoring path; cases 3-4 cover foundational phase/lifecycle rules that this skill enforces but the audit does not have dedicated cases for. Run a model with the skill loaded against each prompt and check the produced Compose satisfies every expectation.", "evals": [ { "id": 0, @@ -9,8 +9,8 @@ "files": [], "expectations": [ "Do not write the label's measured width from onSizeChanged / onGloballyPositioned / onPlaced into state that the value reads in composition (that is a cross-phase back-write: measure -> write -> recompose).", - "Solve it in the layout phase: a custom Layout, Modifier.layout { }, or SubcomposeLayout that places the value using the label's measured width.", - "If BoxWithConstraints / SubcomposeLayout appears, it is used only for parent constraints, never to read the sibling label's measured size.", + "Solve it in the layout phase: a custom Layout, Modifier.layout { }, or SubcomposeLayout that measures the label and places the value at label.width + 8dp — these are correct.", + "Do not use BoxWithConstraints as a substitute for measuring the sibling label: it exposes the parent's constraints, not a sibling's measured size.", "The value's horizontal offset is computed during measure/layout, not read from a composition-phase State." ] }, diff --git a/skills/jetpack-compose-audit/SKILL.md b/skills/jetpack-compose-audit/SKILL.md index b5e0a8c..8498e35 100644 --- a/skills/jetpack-compose-audit/SKILL.md +++ b/skills/jetpack-compose-audit/SKILL.md @@ -364,3 +364,7 @@ For medium or large repositories: - `references/canonical-sources.md` — the official URLs every deduction must cite - `references/diagnostics.md` — copy-pasteable Gradle/code snippets for Compose Compiler reports, stability config, baseline profiles, and R8 checks - `scripts/compose-reports.init.gradle` — Gradle init script the skill injects via `--init-script` in Step 4 to generate compiler reports automatically + +## Acceptance Evals + +`evals/evals.json` holds acceptance cases that grade an *audit of existing code* — each `{prompt, expected_output, expectations}` is a snippet plus the findings the audit must produce. Cases 12–14 (cross-phase back-write, Strong Skipping false lead, snapshot self-invalidation) are mirrored, in the opposite direction, by `compose-agent`'s write-mode evals, so the scoring path and the authoring path stay in lockstep. Run a model against each prompt and check every expectation. From d9398d5d908bece655acc79680a171898b9a22c4 Mon Sep 17 00:00:00 2001 From: Ivan Morgillo Date: Tue, 7 Jul 2026 09:27:30 +0200 Subject: [PATCH 3/4] review r2: stronger CI lockstep + doc sync MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-2 cross-review (Codex, Cursor; no blockers): - Codex: the false-lead gate only locked the auto-memoized-lambda subcase, not the remember(index) pure-expression subcase that case 1 also mirrors. Added an explicit subcase check (the false-lead case must contain both `memoiz` and remember(index)). - Cursor: "lockstep" was one-way and one stuffed case could fake coverage. Now (a) each of the 3 rules must be covered by a DISTINCT case (usedIds), and (b) the audit suite must retain cases 12-14 — bidirectional. Negative-tested: dropping compose 0/1/2, stripping the remember(index) token, stuffing all keywords in one case, or dropping audit case 12 each fail CI; 3/4 stay unlocked (foundational). - Cursor: release-notes-4.3.1 "tracked follow-up" note now marked delivered in 4.4.0. - Cursor: audit evals.json description now mentions the 4.4.0 write-mode mirror. - Cursor nit: case 3 expectation no longer assumes Animatable — allows a scroll-only (LazyListState + lambda modifier) solution. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_016ysskXsKpwEbs1n9CX1ScY --- bin/ci | 24 +++++++++++++++++-- docs/release-notes-4.3.1.md | 2 +- skills/compose-agent/evals/evals.json | 2 +- skills/jetpack-compose-audit/evals/evals.json | 2 +- 4 files changed, 25 insertions(+), 5 deletions(-) diff --git a/bin/ci b/bin/ci index 5d41d8f..ffea1ba 100755 --- a/bin/ci +++ b/bin/ci @@ -210,13 +210,33 @@ const composeSuite = evalSuites.get('compose-agent'); if (!composeSuite) { throw new Error('compose-agent must ship write-mode evals at skills/compose-agent/evals/evals.json'); } +// Each rule must be covered by its own DISTINCT case — so one case stuffed with all +// three keywords can't fake coverage. +const usedIds = new Set(); for (const [label, needle] of [ ['cross-phase back-write', /onSizeChanged|onGloballyPositioned|onPlaced|cross-phase/], ['Strong Skipping false lead', /Strong Skipping[\s\S]*memoiz|memoiz[\s\S]*Strong Skipping/i], ['snapshot self-invalidation', /mutableStateMapOf|mutableStateListOf|self-invalidation/], ]) { - if (!composeSuite.evals.some((item) => needle.test(JSON.stringify(item)))) { - throw new Error(`compose-agent evals must include a case covering the ${label} authoring rule`); + const match = composeSuite.evals.find((item) => !usedIds.has(item.id) && needle.test(JSON.stringify(item))); + if (!match) { + throw new Error(`compose-agent evals must include a distinct case covering the ${label} authoring rule`); + } + usedIds.add(match.id); +} +// The Strong Skipping false-lead case must cover BOTH subcases: the auto-memoized +// lambda AND the remember(index) pure-expression no-op. +const falseLeadCase = composeSuite.evals.find((item) => /memoiz/i.test(JSON.stringify(item))); +if (!falseLeadCase || !/remember\(index\)|remember\(i\)/.test(JSON.stringify(falseLeadCase))) { + throw new Error('compose-agent Strong Skipping false-lead case must cover both the auto-memoized-lambda and the remember(index) pure-expression subcases'); +} +// Bidirectional lockstep: the audit must keep the cases (12-14) that these mirror. +const auditSuite = evalSuites.get('jetpack-compose-audit'); +if (auditSuite) { + for (const requiredId of [12, 13, 14]) { + if (!auditSuite.evals.some((item) => item.id === requiredId)) { + throw new Error(`jetpack-compose-audit evals must retain case ${requiredId} (the audit side of the compose-agent write-mode mirror)`); + } } } NODE diff --git a/docs/release-notes-4.3.1.md b/docs/release-notes-4.3.1.md index dad5c86..01d9716 100644 --- a/docs/release-notes-4.3.1.md +++ b/docs/release-notes-4.3.1.md @@ -26,7 +26,7 @@ The same correction lands in `compose-agent/performance.md`: the *Strong Skippin ## Eval coverage -The authoring rules share the audit acceptance evals (`jetpack-compose-audit/evals/evals.json` cases 12–14) — `compose-agent` mirrors the same rubric. A dedicated compose-agent *write-mode* eval harness is a tracked follow-up. +The authoring rules share the audit acceptance evals (`jetpack-compose-audit/evals/evals.json` cases 12–14) — `compose-agent` mirrors the same rubric. A dedicated compose-agent *write-mode* eval harness was **delivered in 4.4.0** (`skills/compose-agent/evals/evals.json`, cases 0–2 mirror audit 12–14, CI-locked). ## Attribution diff --git a/skills/compose-agent/evals/evals.json b/skills/compose-agent/evals/evals.json index baa7ea6..1684c3e 100644 --- a/skills/compose-agent/evals/evals.json +++ b/skills/compose-agent/evals/evals.json @@ -45,7 +45,7 @@ "expectations": [ "The per-frame progress is read in layout/draw (lambda modifier / graphicsLayer / drawBehind / layout) or passed as a () -> Float provider — not as a plain Float read in ProgressBar's composition.", "ProgressBar does not recompose on every frame of the scroll/animation.", - "State is hoisted to Parent (or a state holder), and any Animatable is held in remember and driven from a LaunchedEffect." + "State is hoisted to Parent (or a state holder); if the solution is animation-driven, any Animatable is held in remember and driven from a LaunchedEffect (a scroll-driven solution may instead read LazyListState in a lambda modifier)." ] }, { diff --git a/skills/jetpack-compose-audit/evals/evals.json b/skills/jetpack-compose-audit/evals/evals.json index 04b5042..0cce188 100644 --- a/skills/jetpack-compose-audit/evals/evals.json +++ b/skills/jetpack-compose-audit/evals/evals.json @@ -1,6 +1,6 @@ { "skill_name": "jetpack-compose-audit", - "description": "Acceptance evals for the API-hygiene, effect-correctness, cancellation, and state-boundary behavior introduced in 4.1.x, plus Paging 3 in Compose list/load-state behavior added in 4.2.0 (cases 10-11), plus cross-phase back-write detection and the false-lead scoring guard added in 4.3.0 (cases 12-14; compose-agent gained matching authoring guidance in 4.3.1). These are the audit/review acceptance criteria the skill must preserve; the earlier cases also back compose-agent review behavior. Run a model against each prompt and check it satisfies every expectation.", + "description": "Acceptance evals for the API-hygiene, effect-correctness, cancellation, and state-boundary behavior introduced in 4.1.x, plus Paging 3 in Compose list/load-state behavior added in 4.2.0 (cases 10-11), plus cross-phase back-write detection and the false-lead scoring guard added in 4.3.0 (cases 12-14; compose-agent gained matching authoring guidance in 4.3.1, and a write-mode eval mirror in 4.4.0 — its cases 0-2 correspond to cases 12-14 here). These are the audit/review acceptance criteria the skill must preserve; the earlier cases also back compose-agent review behavior. Run a model against each prompt and check it satisfies every expectation.", "evals": [ { "id": 0, From e102425420da87c6232f4b335eb13ba72562864d Mon Sep 17 00:00:00 2001 From: Ivan Morgillo Date: Tue, 7 Jul 2026 09:38:18 +0200 Subject: [PATCH 4/4] review r3: content-based audit lockstep + coupled subcase check + SSM caveat MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-3 cross-review (Cursor; Codex empty; no blockers — hardening only): - Audit-side lockstep is now content-based, not id-only: audit cases 12/13/14 must still match the cross-phase / false-lead / self-invalidation needles, so hollowing them out fails CI (negative-tested). - The remember(index) subcase check now reuses the case matched for the Strong Skipping rule (matchedByRule map) instead of a fresh /memoiz/ lookup. - Case 1 gains an expectation mirroring audit case 13's last bullet: the manual- remember lever, if mentioned, must be scoped to SSM-off / @DontMemoize, never the default SSM track. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_016ysskXsKpwEbs1n9CX1ScY --- bin/ci | 35 ++++++++++++++++++--------- skills/compose-agent/evals/evals.json | 3 ++- 2 files changed, 25 insertions(+), 13 deletions(-) diff --git a/bin/ci b/bin/ci index ffea1ba..706ad1c 100755 --- a/bin/ci +++ b/bin/ci @@ -210,32 +210,43 @@ const composeSuite = evalSuites.get('compose-agent'); if (!composeSuite) { throw new Error('compose-agent must ship write-mode evals at skills/compose-agent/evals/evals.json'); } -// Each rule must be covered by its own DISTINCT case — so one case stuffed with all -// three keywords can't fake coverage. -const usedIds = new Set(); -for (const [label, needle] of [ +// The three 4.3.x authoring rules, each with the token that identifies its case. +const authoringRules = [ ['cross-phase back-write', /onSizeChanged|onGloballyPositioned|onPlaced|cross-phase/], ['Strong Skipping false lead', /Strong Skipping[\s\S]*memoiz|memoiz[\s\S]*Strong Skipping/i], ['snapshot self-invalidation', /mutableStateMapOf|mutableStateListOf|self-invalidation/], -]) { +]; +// Each rule must be covered by its own DISTINCT case — so one case stuffed with all +// three keywords can't fake coverage. +const usedIds = new Set(); +const matchedByRule = new Map(); +for (const [label, needle] of authoringRules) { const match = composeSuite.evals.find((item) => !usedIds.has(item.id) && needle.test(JSON.stringify(item))); if (!match) { throw new Error(`compose-agent evals must include a distinct case covering the ${label} authoring rule`); } usedIds.add(match.id); + matchedByRule.set(label, match); } // The Strong Skipping false-lead case must cover BOTH subcases: the auto-memoized -// lambda AND the remember(index) pure-expression no-op. -const falseLeadCase = composeSuite.evals.find((item) => /memoiz/i.test(JSON.stringify(item))); -if (!falseLeadCase || !/remember\(index\)|remember\(i\)/.test(JSON.stringify(falseLeadCase))) { +// lambda AND the remember(index) pure-expression no-op. Checked on the same case the +// Strong Skipping rule matched above, not a fresh lookup. +const falseLeadCase = matchedByRule.get('Strong Skipping false lead'); +if (!/remember\(index\)|remember\(i\)/.test(JSON.stringify(falseLeadCase))) { throw new Error('compose-agent Strong Skipping false-lead case must cover both the auto-memoized-lambda and the remember(index) pure-expression subcases'); } -// Bidirectional lockstep: the audit must keep the cases (12-14) that these mirror. +// Bidirectional lockstep: the audit must keep cases 12-14 AND they must still cover +// the same three rules (an id alone is not enough). const auditSuite = evalSuites.get('jetpack-compose-audit'); if (auditSuite) { - for (const requiredId of [12, 13, 14]) { - if (!auditSuite.evals.some((item) => item.id === requiredId)) { - throw new Error(`jetpack-compose-audit evals must retain case ${requiredId} (the audit side of the compose-agent write-mode mirror)`); + const auditMirror = [[12, authoringRules[0]], [13, authoringRules[1]], [14, authoringRules[2]]]; + for (const [id, [label, needle]] of auditMirror) { + const auditCase = auditSuite.evals.find((item) => item.id === id); + if (!auditCase) { + throw new Error(`jetpack-compose-audit evals must retain case ${id} (the audit side of the compose-agent write-mode mirror)`); + } + if (!needle.test(JSON.stringify(auditCase))) { + throw new Error(`jetpack-compose-audit case ${id} must still cover the ${label} rule (mirror of compose-agent write-mode evals)`); } } } diff --git a/skills/compose-agent/evals/evals.json b/skills/compose-agent/evals/evals.json index 1684c3e..5f6d754 100644 --- a/skills/compose-agent/evals/evals.json +++ b/skills/compose-agent/evals/evals.json @@ -23,7 +23,8 @@ "Do not wrap the onClick / onOpen lambda in remember to 'stabilize' it — under Strong Skipping the compiler already memoizes lambdas passed to composables (even with unstable captures).", "Do not add remember(i) / remember(index) around the pure `i == 0` expression as a recomposition fix.", "Point at real levers: Row as a stable type (data class with stable/immutable fields, no var), RowItem being skippable, and a stable key = on itemsIndexed.", - "Recommend verifying with Layout Inspector recomposition counts or Compose compiler reports rather than asserting a win from the pattern." + "Recommend verifying with Layout Inspector recomposition counts or Compose compiler reports rather than asserting a win from the pattern.", + "If the manual-remember lever is mentioned at all, it is correctly scoped to SSM-off or a @DontMemoize path — never presented as helping on the default Strong Skipping track." ] }, {