Skip to content

FAL-311: Add isolated read-only CLI live smoke coverage - #25

Closed
daveosterjr wants to merge 6 commits into
masterfrom
codex/live-smoke
Closed

daveosterjr wants to merge 6 commits into
masterfrom
codex/live-smoke

Conversation

@daveosterjr

@daveosterjr daveosterjr commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

FAL-311 adds npm run smoke:live:coverage to exercise the built CLI's version, account, filters, fields and credit-free property-search estimate against an explicitly selected API. The local relay refuses requests outside the read-only allowlist and validates CLI source, User-Agent and authentication. Each command uses an encrypted temporary configuration with OS credential-store access disabled; missing credentials exit 1 without printing the key.

The existing smoke:live Factory baseline and packed-install assertions are preserved. The expanded smoke loads its isolation module by file URL, fixing Windows drive paths and temporary paths containing #. The build invokes the installed TypeScript compiler through Node, fixing spawnSync tsc ENOENT on Windows. CI now exercises the smoke's isolation and cleanup on macOS and Windows as well as the existing Linux checks.

Validation at 1c841d0cc174fae35ba2a1f1835ae16afcf8585d:

  • npm run check: 90 tests, typecheck, build and public artifact validation passed.
  • npm run test:package: both npm names installed and passed the clean-consumer checks.
  • All six CI jobs passed: Linux checks/package tests and macOS/Windows smoke tests on Node 22 and 24.
  • The temporary-path integration test failed on the previous implementation and passed after the file-URL fix. The first Windows CI run exposed the compiler launch failure; the final run passes both Windows versions.

Authenticated staging passed on macOS/Node 22 at this exact candidate on October 9: all five Commands, four read-only API requests, temporary-home removal, key redaction and reservation release. The API proved source 512f1e55b8b4904068012dd46b5d9fd1fa5f1f6c before and after. The dedicated account was made available by separately authorized recovery of the earlier run's one owned test list; ownership, commissioned properties, prospect baseline and independent deletion readback passed before its journal closed. Windows CI uses a synthetic local API, so authenticated Windows staging remains unverified. The existing draft tests/factory/regression/cli-live-smoke-coverage.json preserves that remaining check.

This internal maintainer command changes no public API contract; usage and development documentation are updated here. The CLI has no web preview. Repository policy targets master. Factory now accepts this target for local review only; merges and release remain separate. This PR includes CI changes and requires a person to merge on GitHub.

FAL-311

Review repairs: the integration suite builds its own CLI, so npm test now passes with no dist directory (90 tests). Smoke setup is covered by cleanup and redacted error handling; a fault injected after the encrypted key write now exits 1, removes the temporary home and preserves caller settings. The new cleanup regression failed on the previous code.

The account-plan finding was checked against the owning API contract: the published Get Account reference documents data.plan, and the API OpenAPI schema requires it. The CLI display type lists only the fields it renders, not the complete response. The smoke assertion is retained and its source linked beside it.

Factory code and security reviews passed at this head with no findings. The final QA gate remains blocked because the workflow requests a hosted browser preview for CLI; local evidence has not been represented as a hosted QA pass. Recorded staging and recovery result.

daveosterjr and others added 2 commits October 1, 2026 15:25
npm run smoke:live builds the CLI and runs the real dm binary against
DM_API_URL with DM_API_KEY in a temporary home directory. A local relay
refuses non-allowlisted requests and checks the CLI source header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daveosterjr daveosterjr changed the title Add a live, read-only CLI smoke test Add isolated CLI live coverage alongside the existing smoke baseline Oct 9, 2026
@daveosterjr
daveosterjr marked this pull request as draft October 9, 2026 16:52
@daveosterjr daveosterjr changed the title Add isolated CLI live coverage alongside the existing smoke baseline FAL-311: Add isolated read-only CLI live smoke coverage Oct 9, 2026
@daveosterjr
daveosterjr marked this pull request as ready for review October 9, 2026 17:04

@dealmachine-alma dealmachine-alma Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alma: Code and cleanup

Changes need attention.

I found three actionable issues: the integration test needs a prebuilt dist tree that npm test does not create, the account check requires a plan field outside the CLI's current response contract, and setup failures can leave the temporary credential directory behind.

  • medium: Integration test requires a prebuilt dist directory (source)
    Expected: The documented npm test command should run from a clean checkout without requiring a prior manual build.
    Observed: This test launches the coverage script directly, and that script exits with an error when dist/index.js is absent. npm test runs Vitest without building dist, so the integration case fails on a clean checkout; the CI matrix avoids this only because it builds first.
    Suggested fix: Build the CLI as part of this test's setup or make the normal test command build before running Vitest.

  • medium: Account smoke requires an undocumented plan field (source)
    Expected: The account smoke check should validate fields in the CLI's current account response contract and the ticket's account-check requirement.
    Observed: The smoke fails unless data.plan exists, although the CLI's AccountResponse and its existing account response fixture contain organization and user fields but no plan. The new synthetic API fixture adds a plan, so it cannot expose this contract mismatch.
    Suggested fix: Remove the data.plan requirement unless the account endpoint contract is updated and verified to guarantee it.

  • low: Setup failures leave the temporary credential directory behind (source)
    Expected: Temporary credentials and the local relay should be cleaned up if isolated configuration setup fails.
    Observed: The temporary directory, encrypted credential file and relay are created before the try/finally that closes the relay and removes the directory. If seeding the config or writing the isolation module throws, cleanup is skipped and the temp directory can retain the credential file.
    Suggested fix: Put relay and temporary-directory setup inside a cleanup scope that closes the relay when created and removes the temporary directory when available.

Reviewed commit: 08fed505a7cd.

This is a comment review. It does not approve a merge or release; required checks and QA remain separate.

Factory-Review-Run: eb390627dd059fc8c5dbff985fce45e1825866bc12d7ba3bf5d10db342848e09

@dealmachine-alma dealmachine-alma Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alma: Code and cleanup

No findings in this review.

I reviewed all ten changed files against the acceptance criteria and found no actionable code defect. This was a source review; the authenticated staging smoke on macOS and Windows remains an outstanding human QA check in the added regression case.

Reviewed commit: 1c841d0cc174.

This is a comment review. It does not approve a merge or release; required checks and QA remain separate.

Factory-Review-Run: e80bf51fc6d3de7818667ebbbc50564a296230bbcc7eb52067702c0f89ef5cd3

@dealmachine-alma dealmachine-alma Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alma: Security

No findings in this review.

No concrete security defects found in the pinned PR. Reviewed all ten changed files and traced temporary credential creation through the real CLI configuration reader, authenticated request construction, the relay's read-only allowlist and estimate-only POST requirement, HTTPS target validation, redirect refusal, child-process isolation, redacted output and normal/error cleanup. The added CI job retains read-only repository permissions and uses synthetic credentials. This was source review only: no tests or live API requests were executed, and authenticated staging plus Windows OS credential-store isolation remain unverified operator checks. Required QA and repository checks still apply.

Reviewed commit: 1c841d0cc174.

This is a comment review. It does not approve a merge or release; required checks and QA remain separate.

Factory-Review-Run: 4b48cd22abbed9a27f331ccc06872ec7bd2240aed23244d345a5f4a68345f5d0

@daveosterjr

Copy link
Copy Markdown
Contributor Author

Superseded by PR #28, which carries the scoped FAL-311 smoke implementation and Windows fixes and merged into staging as a9bc914 after code review, security review, CI and registered live staging QA passed. Live macOS and Windows proof is recorded on the final reviewed commit 601c114. This closes the duplicate proposal; it does not publish a CLI release.

@daveosterjr daveosterjr closed this Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant