This repository was archived by the owner on Sep 4, 2026. It is now read-only.
test(e2e): prove AskUserQuestion against the real CLI, not a reading of the SDK - #394
Merged
Conversation
ElbertePlinio
force-pushed
the
feat-364-live-e2e
branch
from
July 26, 2026 03:09
02f0e16 to
8cc7b80
Compare
…of the SDK #364 shipped with thorough unit coverage at every layer and no way to know it worked. Nothing in this repo reaches the CLI: the bridge tests mock the SDK and put a zero-byte `claude` on PATH, the Rust tests spawn `#!/bin/sh` fakes, VRT runs against `tauriMock.ts`, and `bun run e2e` mocks `invoke`. The one contract that matters — that returning `{behavior: "allow", updatedInput: {...input, answers}}` makes the tool report answers rather than "The user did not answer the questions." — was asserted against my reading of the SDK. This drives the real binary through the real bridge, forces an AskUserQuestion, answers it through the same approve op the GUI uses, and asserts the tool's own result text. Verified load-bearing the only way that counts: disabling the answers branch in `permissionResultForDecision` makes this fail with the real refusal string, and restoring it makes it pass. That is the first direct evidence the fix works. Opt-in in the style of the device tiers beside it — unset gate skips and exits 0, so CI stays green without credentials; gate set with `claude` missing fails fast, because a silent pass would be worse than no test. Pinned to a cheap model per the repo's dogfooding convention. One finding worth recording, which cost a debugging cycle and is now in the README: wait for a raw `result` message, not `turnClosed`. The bridge emits `turnClosed` only when the whole query iterator ends, because a session stays open for follow-up sends — waiting on it hangs until timeout even though the turn succeeded. Refs #364
ElbertePlinio
force-pushed
the
feat-364-live-e2e
branch
from
July 26, 2026 03:16
8cc7b80 to
e9d80fc
Compare
This was referenced Jul 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap I flagged when #364 merged: it shipped with thorough unit coverage at every layer, and no way to know it worked.
Nothing in this repo reaches the CLI:
claudeBridge.test.tsclaudeon PATH#!/bin/shfakestauriMock.tsbun run e2einvokeSo the one contract that actually matters — that returning
{behavior: "allow", updatedInput: {...input, answers}}makes the tool report answers rather than "The user did not answer the questions." — was asserted against my reading of the SDK, never against the SDK.What this does
Drives the real
claudebinary through the real bridge, forces anAskUserQuestion, answers it through the sameapproveop the GUI uses, and asserts the tool's own result text.Verified load-bearing the only way that counts
Disabling the branch in
permissionResultForDecisionreproduces the real refusal string from the live tool; restoring it passes. That is the first direct evidence #364's fix works.Gating, per the device tiers beside it
bun run e2estay green without credentials.claudemissing → FAIL FAST. A silent pass here would be worse than no test.PICKFORGE_E2E_CLAUDE_MODEL, defaulthaiku) per the repo's dogfooding convention inCLAUDE.md.bun run e2e:claudeadded;tests/e2e/README.mddocuments the tier.One finding worth the README entry
Wait for a raw
resultmessage, notturnClosed. The bridge emitsturnClosedonly when the whole query iterator ends — a session stays open for follow-up sends — so waiting on it hangs until the timeout even though the turn succeeded. That cost a debugging cycle and would cost the next person one too.Incidental confirmation for #365
The live stream shows
system:statusandsystem:thinking_tokensframes arriving and being dropped by the parser's dispatch — direct evidence for the premise behind #392, which handlessystem/status.Validation
PICKFORGE_E2E_CLAUDE=1 bun run e2e:claude— PASS against the live CLIbun run e2e— PASS (device tier skips, as designed)bunx tsc --noEmitclean;bun run lintcleanRefs #364