Skip to content

README.md

Latest commit

 

History

History
66 lines (48 loc) · 2.13 KB

README.md

File metadata and controls

66 lines (48 loc) · 2.13 KB
Raw

Yield evaluations

These evaluations test Yield itself. They do not compare Yield with another tool or company.

The deterministic suite answers two questions:

  1. Can each checked-in example skill workflow reach its expected final result through every supported SDK?
  2. Does the runtime behave correctly when a run resumes, replays, blocks, or encounters changed code?

Current coverage

  • 10 skill workflow patterns written by this project.
  • 4 SDKs: TypeScript, Python, Go, and Rust.
  • 40 end-to-end workflow tests.
  • 8 runtime checks: response locking and recovery, declared user choices, resume, replay, changed behavior, failed requirements, and changed source.

Run the exact suite and refresh the checked-in result:

cd evals
npm run eval

Check that the published result still matches the current source:

npm test

What a passing result proves

A passing result proves that the tested Yield revision:

  • executes each owned skill workflow test to completed;
  • runs command steps rather than asking the model to invent their outputs;
  • presents requests in the program-defined order;
  • resumes from recorded responses;
  • returns to the same saved step during replay;
  • stops on changed behavior or failed requirements.

What it does not prove

This suite does not prove that Yield is better than prose, that an agent's judgment is correct, or that illustrative commands are production-safe. The fixed test data supplies agent and human responses so the suite can test only the code-controlled workflow layer.

results/latest.json is a compact, website-safe result. Its source hash is computed from the CLI, engine, protocol, SDKs, example workflows, fixtures, and evaluation harness. CI reruns the suite instead of trusting that file alone.

Coding-agent workflow check

The separate agent/ suite runs the same owned workflow through a real coding agent in two forms: a long skill, and a thin skill backed by Yield code. It checks matching step order, gates, responses, and final status. It does not score the agent's domain judgment or claim that one form is better.

npm run eval:agent
npm run test:agent