Skip to content

feat(mcp): include the skill references a task needs, picked by Jev - #22

Merged
Plopix merged 1 commit into
mainfrom
feat/jev-skill-references
Sep 29, 2026
Merged

Plopix merged 1 commit into
mainfrom
feat/jev-skill-references

Conversation

@Plopix

@Plopix Plopix commented Sep 29, 2026

Copy link
Copy Markdown
Member
Q A
Branch? main
Bug fix? no
New feature? yes
BC breaks? no
Fixed tickets —

The skills tool takes an optional task ("add Klarna to my checkout"). TypeSafe's Jev (from #20) picks which references of the requested skills that task needs, and the tool includes them in the same call. Before, the agent got either no references (just their names) or all of them: 144k characters for payments, 150k for content-model.

How it works

  • Each reference is summarized from its own content: its # title, its intro paragraph and its ## headings.
  • Jev gets one request with one yes/no question per reference. The tool keeps those scoring >= 0.5, up to 3.
  • The main skill document is always returned. Picked references are included whole, preceded by a line like References picked for your task: klarna (96%), and the other references are listed by name.
  • Explicit references or includeAllReferences win over task. A blank task, a Jev failure, or nothing picked gives today's output.
  • The skills catalog (virtual:skills) is now injected through the container, so the tool can be tested; Bun can't load the Vite virtual module.

Measured live (bun run bin/eval-reference-selection.ts)

threshold tasks with all needed references avg references returned
0.3 12/12 2.00
0.5 12/12 1.75
0.7 9/12 1.33

Each call is about 1.7k input tokens (roughly $0.00007) and takes about 320ms.

Notes

  • The agent still picks the skills. Jev only chooses among the references of the skills it asked for, and the eval measures exactly that step.
  • Most useful where reference names are ambiguous (content-model, mass-operations). For payments the provider names already speak for themselves.
  • The cap of 3 is full on two tasks. "Landing pages" and "create a shape" each fill all three slots.
  • Shared eval helper. Both eval scripts now use bin/lib/jev-rest.ts, and both need no settings: the account comes from wrangler.jsonc and the token from wrangler login.

bun test passes (162/162), and bun type-check, bun lint and bun run build are clean.

@Plopix
Plopix merged commit c95142e into main Sep 29, 2026
1 check passed
@Plopix
Plopix deleted the feat/jev-skill-references branch September 29, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant