Skip to content
This repository was archived by the owner on Sep 4, 2026. It is now read-only.

test(e2e): prove AskUserQuestion against the real CLI, not a reading of the SDK - #394

Merged
ElbertePlinio merged 1 commit into
mainfrom
feat-364-live-e2e
Jul 27, 2026
Merged

test(e2e): prove AskUserQuestion against the real CLI, not a reading of the SDK#394
ElbertePlinio merged 1 commit into
mainfrom
feat-364-live-e2e

Conversation

@ElbertePlinio

Copy link
Copy Markdown
Member

Closes the gap I flagged when #364 merged: it shipped with thorough unit coverage at every layer, and no way to know it worked.

Nothing in this repo reaches the CLI:

tier what it mocks
claudeBridge.test.ts mocks the SDK, puts a zero-byte claude on PATH
Rust bridge tests spawn #!/bin/sh fakes
VRT runs against tauriMock.ts
bun run e2e mocks invoke

So the one contract that actually matters — that returning {behavior: "allow", updatedInput: {...input, answers}} makes the tool report answers rather than "The user did not answer the questions." — was asserted against my reading of the SDK, never against the SDK.

What this does

Drives the real claude binary through the real bridge, forces an AskUserQuestion, answers it through the same approve op the GUI uses, and asserts the tool's own result text.

Verified load-bearing the only way that counts

answers branch enabled  → PASS: AskUserQuestion was answered and the tool accepted it
answers branch disabled → FAIL: the tool reported that the user did not answer

Disabling the branch in permissionResultForDecision reproduces the real refusal string from the live tool; restoring it passes. That is the first direct evidence #364's fix works.

Gating, per the device tiers beside it

  • Unset gate → SKIP, exit 0. CI and bun run e2e stay green without credentials.
  • Gate set but claude missing → FAIL FAST. A silent pass here would be worse than no test.
  • Costs a real API call; pinned to a cheap model (PICKFORGE_E2E_CLAUDE_MODEL, default haiku) per the repo's dogfooding convention in CLAUDE.md.
  • bun run e2e:claude added; tests/e2e/README.md documents the tier.

One finding worth the README entry

Wait for a raw result message, not turnClosed. The bridge emits turnClosed only when the whole query iterator ends — a session stays open for follow-up sends — so waiting on it hangs until the timeout even though the turn succeeded. That cost a debugging cycle and would cost the next person one too.

Incidental confirmation for #365

The live stream shows system:status and system:thinking_tokens frames arriving and being dropped by the parser's dispatch — direct evidence for the premise behind #392, which handles system/status.

Validation

  • PICKFORGE_E2E_CLAUDE=1 bun run e2e:claude — PASS against the live CLI
  • bun run e2e — PASS (device tier skips, as designed)
  • bunx tsc --noEmit clean; bun run lint clean

Refs #364

…of the SDK

#364 shipped with thorough unit coverage at every layer and no way to know it
worked. Nothing in this repo reaches the CLI: the bridge tests mock the SDK and
put a zero-byte `claude` on PATH, the Rust tests spawn `#!/bin/sh` fakes, VRT
runs against `tauriMock.ts`, and `bun run e2e` mocks `invoke`. The one contract
that matters — that returning `{behavior: "allow", updatedInput: {...input,
answers}}` makes the tool report answers rather than "The user did not answer
the questions." — was asserted against my reading of the SDK.

This drives the real binary through the real bridge, forces an AskUserQuestion,
answers it through the same approve op the GUI uses, and asserts the tool's own
result text.

Verified load-bearing the only way that counts: disabling the answers branch in
`permissionResultForDecision` makes this fail with the real refusal string, and
restoring it makes it pass. That is the first direct evidence the fix works.

Opt-in in the style of the device tiers beside it — unset gate skips and exits
0, so CI stays green without credentials; gate set with `claude` missing fails
fast, because a silent pass would be worse than no test. Pinned to a cheap model
per the repo's dogfooding convention.

One finding worth recording, which cost a debugging cycle and is now in the
README: wait for a raw `result` message, not `turnClosed`. The bridge emits
`turnClosed` only when the whole query iterator ends, because a session stays
open for follow-up sends — waiting on it hangs until timeout even though the
turn succeeded.

Refs #364
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant