Repository navigation
FAL-311: Add isolated read-only CLI live smoke coverage - #25
daveosterjr wants to merge 6 commits into
Conversation
npm run smoke:live builds the CLI and runs the real dm binary against DM_API_URL with DM_API_KEY in a temporary home directory. A local relay refuses non-allowlisted requests and checks the CLI source header. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Alma: Code and cleanup
Changes need attention.
I found three actionable issues: the integration test needs a prebuilt dist tree that npm test does not create, the account check requires a plan field outside the CLI's current response contract, and setup failures can leave the temporary credential directory behind.
-
medium: Integration test requires a prebuilt dist directory (source)
Expected: The documentednpm testcommand should run from a clean checkout without requiring a prior manual build.
Observed: This test launches the coverage script directly, and that script exits with an error whendist/index.jsis absent.npm testruns Vitest without buildingdist, so the integration case fails on a clean checkout; the CI matrix avoids this only because it builds first.
Suggested fix: Build the CLI as part of this test's setup or make the normal test command build before running Vitest. -
medium: Account smoke requires an undocumented plan field (source)
Expected: The account smoke check should validate fields in the CLI's current account response contract and the ticket's account-check requirement.
Observed: The smoke fails unlessdata.planexists, although the CLI'sAccountResponseand its existing account response fixture contain organization and user fields but no plan. The new synthetic API fixture adds a plan, so it cannot expose this contract mismatch.
Suggested fix: Remove thedata.planrequirement unless the account endpoint contract is updated and verified to guarantee it. -
low: Setup failures leave the temporary credential directory behind (source)
Expected: Temporary credentials and the local relay should be cleaned up if isolated configuration setup fails.
Observed: The temporary directory, encrypted credential file and relay are created before thetry/finallythat closes the relay and removes the directory. If seeding the config or writing the isolation module throws, cleanup is skipped and the temp directory can retain the credential file.
Suggested fix: Put relay and temporary-directory setup inside a cleanup scope that closes the relay when created and removes the temporary directory when available.
Reviewed commit: 08fed505a7cd.
This is a comment review. It does not approve a merge or release; required checks and QA remain separate.
Factory-Review-Run: eb390627dd059fc8c5dbff985fce45e1825866bc12d7ba3bf5d10db342848e09
There was a problem hiding this comment.
Alma: Code and cleanup
No findings in this review.
I reviewed all ten changed files against the acceptance criteria and found no actionable code defect. This was a source review; the authenticated staging smoke on macOS and Windows remains an outstanding human QA check in the added regression case.
Reviewed commit: 1c841d0cc174.
This is a comment review. It does not approve a merge or release; required checks and QA remain separate.
Factory-Review-Run: e80bf51fc6d3de7818667ebbbc50564a296230bbcc7eb52067702c0f89ef5cd3
There was a problem hiding this comment.
Alma: Security
No findings in this review.
No concrete security defects found in the pinned PR. Reviewed all ten changed files and traced temporary credential creation through the real CLI configuration reader, authenticated request construction, the relay's read-only allowlist and estimate-only POST requirement, HTTPS target validation, redirect refusal, child-process isolation, redacted output and normal/error cleanup. The added CI job retains read-only repository permissions and uses synthetic credentials. This was source review only: no tests or live API requests were executed, and authenticated staging plus Windows OS credential-store isolation remain unverified operator checks. Required QA and repository checks still apply.
Reviewed commit: 1c841d0cc174.
This is a comment review. It does not approve a merge or release; required checks and QA remain separate.
Factory-Review-Run: 4b48cd22abbed9a27f331ccc06872ec7bd2240aed23244d345a5f4a68345f5d0
|
Superseded by PR #28, which carries the scoped FAL-311 smoke implementation and Windows fixes and merged into staging as a9bc914 after code review, security review, CI and registered live staging QA passed. Live macOS and Windows proof is recorded on the final reviewed commit 601c114. This closes the duplicate proposal; it does not publish a CLI release. |
FAL-311 adds
npm run smoke:live:coverageto exercise the built CLI's version, account, filters, fields and credit-free property-search estimate against an explicitly selected API. The local relay refuses requests outside the read-only allowlist and validates CLI source, User-Agent and authentication. Each command uses an encrypted temporary configuration with OS credential-store access disabled; missing credentials exit 1 without printing the key.The existing
smoke:liveFactory baseline and packed-install assertions are preserved. The expanded smoke loads its isolation module by file URL, fixing Windows drive paths and temporary paths containing#. The build invokes the installed TypeScript compiler through Node, fixingspawnSync tsc ENOENTon Windows. CI now exercises the smoke's isolation and cleanup on macOS and Windows as well as the existing Linux checks.Validation at
1c841d0cc174fae35ba2a1f1835ae16afcf8585d:npm run check: 90 tests, typecheck, build and public artifact validation passed.npm run test:package: both npm names installed and passed the clean-consumer checks.Authenticated staging passed on macOS/Node 22 at this exact candidate on October 9: all five Commands, four read-only API requests, temporary-home removal, key redaction and reservation release. The API proved source
512f1e55b8b4904068012dd46b5d9fd1fa5f1f6cbefore and after. The dedicated account was made available by separately authorized recovery of the earlier run's one owned test list; ownership, commissioned properties, prospect baseline and independent deletion readback passed before its journal closed. Windows CI uses a synthetic local API, so authenticated Windows staging remains unverified. The existing drafttests/factory/regression/cli-live-smoke-coverage.jsonpreserves that remaining check.This internal maintainer command changes no public API contract; usage and development documentation are updated here. The CLI has no web preview. Repository policy targets
master. Factory now accepts this target for local review only; merges and release remain separate. This PR includes CI changes and requires a person to merge on GitHub.FAL-311
Review repairs: the integration suite builds its own CLI, so
npm testnow passes with nodistdirectory (90 tests). Smoke setup is covered by cleanup and redacted error handling; a fault injected after the encrypted key write now exits 1, removes the temporary home and preserves caller settings. The new cleanup regression failed on the previous code.The account-plan finding was checked against the owning API contract: the published Get Account reference documents
data.plan, and the API OpenAPI schema requires it. The CLI display type lists only the fields it renders, not the complete response. The smoke assertion is retained and its source linked beside it.Factory code and security reviews passed at this head with no findings. The final QA gate remains blocked because the workflow requests a hosted browser preview for CLI; local evidence has not been represented as a hosted QA pass. Recorded staging and recovery result.