diff --git a/labs/22-yield/yield/evals/cases/README.md b/labs/22-yield/yield/evals/cases/README.md index 0f606b1b..4d194a53 100644 --- a/labs/22-yield/yield/evals/cases/README.md +++ b/labs/22-yield/yield/evals/cases/README.md @@ -7,9 +7,9 @@ repository and is identified by commit plus SHA-256 in `index.json`. | Case | Thin skill | Yield program | Pinned original | |---|---|---|---| -| GStack review | [SKILL.md](gstack-review/SKILL.md) | [workflow.ts](gstack-review/workflow.ts) | [source](https://github.com/garrytan/gstack/blob/a3259400a366593e0c909dd9ac3e59752efd2488/review/SKILL.md) | -| Anthropic doc co-authoring | [SKILL.md](doc-coauthoring/SKILL.md) | [workflow.ts](doc-coauthoring/workflow.ts) | [source](https://github.com/anthropics/skills/blob/b29e7cf65e5cb78a5ac33d582270551bc74a14eb/skills/doc-coauthoring/SKILL.md) | -| Superpowers systematic debugging | [SKILL.md](systematic-debugging/SKILL.md) | [workflow.ts](systematic-debugging/workflow.ts) | [source](https://github.com/obra/superpowers/blob/44c9b2d6e889982ac18c27d05a19fefe335194e1/skills/systematic-debugging/SKILL.md) | -| Vercel deploy | [SKILL.md](vercel-deploy/SKILL.md) | [workflow.ts](vercel-deploy/workflow.ts) | [source](https://github.com/vercel-labs/agent-skills/blob/7c180d9044c9ae2b442b567aad4e42a28dd5ed62/skills/deploy-to-vercel/SKILL.md) | -| Microsoft MCP builder | [SKILL.md](mcp-builder/SKILL.md) | [workflow.ts](mcp-builder/workflow.ts) | [source](https://github.com/microsoft/skills/blob/4a2873faffc1b101a33a0b59c24713d4ed78142f/.github/skills/mcp-builder/SKILL.md) | -| Trail of Bits actions auditor | [SKILL.md](actions-auditor/SKILL.md) | [workflow.ts](actions-auditor/workflow.ts) | [source](https://github.com/trailofbits/skills/blob/1256982d4d925a0acfe11e26c2253c32052c6247/plugins/agentic-actions-auditor/skills/agentic-actions-auditor/SKILL.md) | +| [GStack review](gstack-review/) | [SKILL.md](gstack-review/SKILL.md) | [workflow.ts](gstack-review/workflow.ts) | [source](https://github.com/garrytan/gstack/blob/a3259400a366593e0c909dd9ac3e59752efd2488/review/SKILL.md) | +| [Anthropic doc co-authoring](doc-coauthoring/) | [SKILL.md](doc-coauthoring/SKILL.md) | [workflow.ts](doc-coauthoring/workflow.ts) | [source](https://github.com/anthropics/skills/blob/b29e7cf65e5cb78a5ac33d582270551bc74a14eb/skills/doc-coauthoring/SKILL.md) | +| [Superpowers systematic debugging](systematic-debugging/) | [SKILL.md](systematic-debugging/SKILL.md) | [workflow.ts](systematic-debugging/workflow.ts) | [source](https://github.com/obra/superpowers/blob/44c9b2d6e889982ac18c27d05a19fefe335194e1/skills/systematic-debugging/SKILL.md) | +| [Vercel deploy](vercel-deploy/) | [SKILL.md](vercel-deploy/SKILL.md) | [workflow.ts](vercel-deploy/workflow.ts) | [source](https://github.com/vercel-labs/agent-skills/blob/7c180d9044c9ae2b442b567aad4e42a28dd5ed62/skills/deploy-to-vercel/SKILL.md) | +| [Microsoft MCP builder](mcp-builder/) | [SKILL.md](mcp-builder/SKILL.md) | [workflow.ts](mcp-builder/workflow.ts) | [source](https://github.com/microsoft/skills/blob/4a2873faffc1b101a33a0b59c24713d4ed78142f/.github/skills/mcp-builder/SKILL.md) | +| [Trail of Bits actions auditor](actions-auditor/) | [SKILL.md](actions-auditor/SKILL.md) | [workflow.ts](actions-auditor/workflow.ts) | [source](https://github.com/trailofbits/skills/blob/1256982d4d925a0acfe11e26c2253c32052c6247/plugins/agentic-actions-auditor/skills/agentic-actions-auditor/SKILL.md) | diff --git a/labs/22-yield/yield/evals/cases/actions-auditor/README.md b/labs/22-yield/yield/evals/cases/actions-auditor/README.md new file mode 100644 index 00000000..e5ff3682 --- /dev/null +++ b/labs/22-yield/yield/evals/cases/actions-auditor/README.md @@ -0,0 +1,34 @@ +# Trail of Bits actions auditor — Yield conversion + +This is an independent, measured conversion of +[Trail of Bits' agentic actions auditor](https://github.com/trailofbits/skills/blob/1256982d4d925a0acfe11e26c2253c32052c6247/plugins/agentic-actions-auditor/skills/agentic-actions-auditor/SKILL.md). +It is not a Trail of Bits artifact. + +We used Yield's [`convert-skill`](../../../examples/convert-skill/) workflow to +split security judgment from repeatable control flow, then reviewed the +checked-in result and measured it against the pinned original. + +## The converted version + +- [`SKILL.md`](SKILL.md) keeps attack-path analysis and finding quality. +- [`workflow.ts`](workflow.ts) owns discovery, coverage, evidence requirements, + report generation, and completion. + +## Result + +| Original | Thin skill | Yield program | Maintained change | +|---:|---:|---:|---:| +| 4,846 tokens | 132 tokens | 222 tokens | **−92.7%** | + +## Benefits in this conversion + +- Workflow files are discovered by a real command instead of inferred. +- Completion requires every discovered workflow to be reviewed. +- Every finding must carry concrete evidence. +- Report generation is an observed command with a checked exit code. + +## Claim boundary + +This result measures source size, not audit coverage or behavioral equivalence. +See the [evaluation methodology](../../README.md) for the evidence required +before making a behavior claim. diff --git a/labs/22-yield/yield/evals/cases/doc-coauthoring/README.md b/labs/22-yield/yield/evals/cases/doc-coauthoring/README.md new file mode 100644 index 00000000..ea82e05f --- /dev/null +++ b/labs/22-yield/yield/evals/cases/doc-coauthoring/README.md @@ -0,0 +1,34 @@ +# Anthropic doc co-authoring — Yield conversion + +This is an independent, measured conversion of +[Anthropic's doc co-authoring skill](https://github.com/anthropics/skills/blob/b29e7cf65e5cb78a5ac33d582270551bc74a14eb/skills/doc-coauthoring/SKILL.md). +It is not an Anthropic artifact. + +We used Yield's [`convert-skill`](../../../examples/convert-skill/) workflow to +split model judgment from repeatable control flow, then reviewed the checked-in +result and measured it against the pinned original. + +## The converted version + +- [`SKILL.md`](SKILL.md) keeps writing judgment, reader perspective, and voice. +- [`workflow.ts`](workflow.ts) owns the questions, outline approval, draft stages, + reader test, saved answers, and completion gate. + +## Result + +| Original | Thin skill | Yield program | Maintained change | +|---:|---:|---:|---:| +| 3,289 tokens | 137 tokens | 209 tokens | **−89.5%** | + +## Benefits in this conversion + +- A returning run can continue from saved audience, outcome, and source context. +- Outline approval is an explicit gate instead of a prose suggestion. +- Reader testing happens before the final revision. +- The model spends its context on the document, not on remembering stage order. + +## Claim boundary + +This result measures source size, not writing quality or behavioral equivalence. +See the [evaluation methodology](../../README.md) for the evidence required +before making a behavior claim. diff --git a/labs/22-yield/yield/evals/cases/gstack-review/README.md b/labs/22-yield/yield/evals/cases/gstack-review/README.md new file mode 100644 index 00000000..41eca414 --- /dev/null +++ b/labs/22-yield/yield/evals/cases/gstack-review/README.md @@ -0,0 +1,34 @@ +# GStack review — Yield conversion + +This is an independent, measured conversion of +[GStack's review skill](https://github.com/garrytan/gstack/blob/a3259400a366593e0c909dd9ac3e59752efd2488/review/SKILL.md). +It is not an upstream GStack artifact. + +We used Yield's [`convert-skill`](../../../examples/convert-skill/) workflow to +split model judgment from repeatable control flow, then reviewed the checked-in +result and measured it against the pinned original. + +## The converted version + +- [`SKILL.md`](SKILL.md) keeps review judgment, severity rules, and finding shape. +- [`workflow.ts`](workflow.ts) owns the diff, tests, typecheck, critical-finding + gate, saved result, and completion. + +## Result + +| Original | Thin skill | Yield program | Maintained change | +|---:|---:|---:|---:| +| 27,162 tokens | 146 tokens | 160 tokens | **−98.9%** | + +## Benefits in this conversion + +- Repository checks run as commands instead of instructions the model must remember. +- A review cannot complete while a critical finding remains. +- The diff and check output become explicit evidence passed into the review. +- The model-facing prompt is small enough to focus on finding real defects. + +## Claim boundary + +The token result measures source size. A separate early GStack behavior study is +summarized in [`results/latest.json`](../../results/latest.json), but its raw +artifact is not yet published and it does not prove general equivalence. diff --git a/labs/22-yield/yield/evals/cases/mcp-builder/README.md b/labs/22-yield/yield/evals/cases/mcp-builder/README.md new file mode 100644 index 00000000..5c52d32e --- /dev/null +++ b/labs/22-yield/yield/evals/cases/mcp-builder/README.md @@ -0,0 +1,34 @@ +# Microsoft MCP builder — Yield conversion + +This is an independent, measured conversion of +[Microsoft's MCP builder skill](https://github.com/microsoft/skills/blob/4a2873faffc1b101a33a0b59c24713d4ed78142f/.github/skills/mcp-builder/SKILL.md). +It is not a Microsoft artifact. + +We used Yield's [`convert-skill`](../../../examples/convert-skill/) workflow to +split tool-design judgment from repeatable control flow, then reviewed the +checked-in result and measured it against the pinned original. + +## The converted version + +- [`SKILL.md`](SKILL.md) keeps tool-design and review judgment. +- [`workflow.ts`](workflow.ts) owns user inputs, the tool-count gate, design + approval, scaffold, tests, evaluations, and completion. + +## Result + +| Original | Thin skill | Yield program | Maintained change | +|---:|---:|---:|---:| +| 2,648 tokens | 131 tokens | 291 tokens | **−84.1%** | + +## Benefits in this conversion + +- The tool surface stays bounded before code generation begins. +- Building requires explicit design approval. +- Scaffold, tests, review, and evaluations happen in a fixed order. +- Completion requires both review and evaluation gates to pass. + +## Claim boundary + +This result measures source size, not MCP server quality or behavioral +equivalence. See the [evaluation methodology](../../README.md) for the evidence +required before making a behavior claim. diff --git a/labs/22-yield/yield/evals/cases/systematic-debugging/README.md b/labs/22-yield/yield/evals/cases/systematic-debugging/README.md new file mode 100644 index 00000000..66e8838c --- /dev/null +++ b/labs/22-yield/yield/evals/cases/systematic-debugging/README.md @@ -0,0 +1,34 @@ +# Systematic debugging — Yield conversion + +This is an independent, measured conversion of +[Superpowers' systematic debugging skill](https://github.com/obra/superpowers/blob/44c9b2d6e889982ac18c27d05a19fefe335194e1/skills/systematic-debugging/SKILL.md). +It is not an upstream Superpowers artifact. + +We used Yield's [`convert-skill`](../../../examples/convert-skill/) workflow to +split diagnostic judgment from repeatable control flow, then reviewed the +checked-in result and measured it against the pinned original. + +## The converted version + +- [`SKILL.md`](SKILL.md) keeps the standard for a falsifiable root-cause hypothesis. +- [`workflow.ts`](workflow.ts) owns reproduction, experiment bounds, approval, + verification, and completion. + +## Result + +| Original | Thin skill | Yield program | Maintained change | +|---:|---:|---:|---:| +| 2,226 tokens | 137 tokens | 226 tokens | **−83.7%** | + +## Benefits in this conversion + +- The failure must reproduce before diagnosis starts. +- A proposed cause must include an executable falsification experiment. +- Applying a fix requires user approval. +- Completion requires the full test suite to pass. + +## Claim boundary + +This result measures source size, not diagnosis quality or behavioral +equivalence. See the [evaluation methodology](../../README.md) for the evidence +required before making a behavior claim. diff --git a/labs/22-yield/yield/evals/cases/vercel-deploy/README.md b/labs/22-yield/yield/evals/cases/vercel-deploy/README.md new file mode 100644 index 00000000..0bbd19d4 --- /dev/null +++ b/labs/22-yield/yield/evals/cases/vercel-deploy/README.md @@ -0,0 +1,34 @@ +# Vercel deploy — Yield conversion + +This is an independent, measured conversion of +[Vercel's deploy skill](https://github.com/vercel-labs/agent-skills/blob/7c180d9044c9ae2b442b567aad4e42a28dd5ed62/skills/deploy-to-vercel/SKILL.md). +It is not a Vercel artifact. + +We used Yield's [`convert-skill`](../../../examples/convert-skill/) workflow to +split deployment explanation from repeatable control flow, then reviewed the +checked-in result and measured it against the pinned original. + +## The converted version + +- [`SKILL.md`](SKILL.md) keeps interpretation of project state and failures. +- [`workflow.ts`](workflow.ts) owns state detection, approval, authentication, + linking, deployment, HTTPS verification, and completion. + +## Result + +| Original | Thin skill | Yield program | Maintained change | +|---:|---:|---:|---:| +| 2,898 tokens | 108 tokens | 290 tokens | **−86.3%** | + +## Benefits in this conversion + +- The deployment command runs only after explicit approval. +- Missing authentication produces an honest blocked outcome. +- A successful command is not enough; the deployed URL must answer over HTTPS. +- Cancellation and failure are recorded separately from success. + +## Claim boundary + +This result measures source size, not deployment reliability or behavioral +equivalence. See the [evaluation methodology](../../README.md) for the evidence +required before making a behavior claim. diff --git a/labs/22-yield/yield/evals/scripts/validate.mjs b/labs/22-yield/yield/evals/scripts/validate.mjs index 651f0075..20ba0986 100644 --- a/labs/22-yield/yield/evals/scripts/validate.mjs +++ b/labs/22-yield/yield/evals/scripts/validate.mjs @@ -24,7 +24,11 @@ for (const item of index.cases) { if (!isSha256(item.source.sha256)) fail(`${item.id}: source sha256 is invalid`) const skill = await readFile(join(root, "cases", item.thin_skill), "utf8") const workflow = await readFile(join(root, "cases", item.workflow), "utf8") + const readme = await readFile(join(root, "cases", item.id, "README.md"), "utf8") if (!skill.trim() || !workflow.trim()) fail(`${item.id}: conversion source is empty`) + for (const required of [item.source.repo, "convert-skill", "](SKILL.md)", "](workflow.ts)"]) { + if (!readme.includes(required)) fail(`${item.id}: README is missing ${required}`) + } localMeasurements.set(item.id, { prompt_tokens: countTokens(skill), workflow_tokens: countTokens(workflow),