diff --git a/README.md b/README.md index 59bbd62..c4f8d33 100644 --- a/README.md +++ b/README.md @@ -1195,6 +1195,32 @@ The published and deployed checks are release gates. They are expected to fail b published or the documentation deployment reaches production. The scenario catalog is stored in `evals/claude-code-name-lookup.json` so the same prompt variants remain visible and reviewable. +### Testing + +```bash +npm test # Unit and behavioral tests with a mocked API (no credentials) +npm run check # Typecheck, build, tests and the public artifact guard +DM_API_URL=https://api-staging.v2.dealmachine.com/v1 DM_API_KEY= npm run smoke:live:coverage +``` + +`npm test` never calls an API. `smoke:live:coverage` builds the CLI and runs the real `dm` binary against +the API named by `DM_API_URL`, authenticated with `DM_API_KEY`. Both are required; the key is never +printed. Each Command runs with a temporary home directory, so the smoke never reads or changes +your own `dm login` credentials. It runs only read-only, credit-free Commands: + +| Command | Checks | +| ----------------------------------------------------------- | ---------------------------------------------------------- | +| `dm --version` | Prints the package version without a network request | +| `dm account --json` | Organization, user auth type and plan are present | +| `dm filters --source-type properties --per-page 5 --json` | Filters have IDs, names, operators and pagination | +| `dm fields --source-type properties --per-page 5 --json` | Fields have IDs, names and pagination | +| `dm properties search --body --estimate-cost --json` | Returns an estimate with no records and no charged credits | + +Requests pass through a local relay that refuses anything outside those read-only Endpoints +(property search must carry `estimate_cost: true`) and confirms each request sends +`X-DealMachine-Source: cli` and the CLI `User-Agent`. The smoke prints one `PASS` or `FAIL` line per +Command and exits 0 only when every Command passes. + ### Standalone Binary The compiled `dist/index.js` includes a `#!/usr/bin/env node` shebang and is declared in `package.json` under `bin.dm`. When installed globally via npm, it becomes available as `dm` on the PATH. diff --git a/docs/development.md b/docs/development.md index a0c0edf..97826cd 100644 --- a/docs/development.md +++ b/docs/development.md @@ -9,7 +9,7 @@ npm run test:package npm run dev ``` -`build` creates JavaScript, declarations and source maps in `dist`, cleaning old output first. It bundles `playbook/PLAYBOOK.md` for offline agent onboarding. `dev` watches TypeScript; run `build` once before the watcher and again after changing the Playbook. From a second terminal, run `npm start -- --help` or `npm start -- agents playbook`. +`build` creates JavaScript, declarations and source maps in `dist`, cleaning old output first. It runs the installed TypeScript compiler through Node so it also works on Windows without relying on a shell shim. It bundles `playbook/PLAYBOOK.md` for offline agent onboarding. `dev` watches TypeScript; run `build` once before the watcher and again after changing the Playbook. From a second terminal, run `npm start -- --help` or `npm start -- agents playbook`. The [Command reference](commands.md) describes CLI usage and JSON output. Tests under `tests/` exercise request behavior and output. `test:package` installs both npm archives into an empty temporary project and checks the executable, module import and project-local Claude Code Playbook installation. It does not write to your personal agent setup or call the API. @@ -28,3 +28,5 @@ Every `master` push runs the production npm workflow; its publishing job stays s `npm run smoke:live` requires explicit `DM_API_URL` ending in `/v1` and `DM_API_KEY` for an approved QA workspace. It builds this candidate and runs real Commands for account identity, field discovery and a credit-free property estimate. It isolates configuration and the credential store, removes its temporary files and does not read your saved login. Missing configuration exits 2; a failed request or response assertion exits 1. It does not prove writes, paid data, OAuth consent, every Command or the published npm version. Factory registers these as `cli:test:package` and `cli:smoke:live`. Keep executable checks here and the shared regression case, reviewed source version, runner delivery and private credentials in Factory. When adding or changing a Command, assess both suites and update the relevant public behavior check alongside its acceptance criteria. A local fixture pass never substitutes for an authenticated live result. + +`npm run smoke:live:coverage` adds version and filter checks plus request-header and read-only relay assertions. It requires the same approved QA target and key, rejects non-HTTPS remote targets and non-plain `/v1` bases, and explicitly disables the OS credential store in child processes. Its built-CLI integration test uses a synthetic local API and runs in routine CI on Ubuntu with Node 22 and 24. It checks missing credentials, request authentication, unchanged caller settings and temporary-directory cleanup, including paths with spaces and `#`. The isolation module is loaded by file URL so Windows drive letters and URL-significant path characters work. Factory runs the registered live smoke against staging with its approved QA workspace reserved for the run. Full macOS and Windows verification is performed separately by the authorized local coding session or an approved platform runner; routine CI uses no live credentials. diff --git a/package.json b/package.json index 450bd89..e53a8e0 100644 --- a/package.json +++ b/package.json @@ -29,6 +29,7 @@ "eval:cold-start:published": "node scripts/eval-cold-start.mjs published", "eval:cold-start:deployed": "node scripts/eval-cold-start.mjs deployed", "eval:cold-start:all": "npm run build && node scripts/eval-cold-start.mjs all", + "smoke:live:coverage": "npm run build && node scripts/smoke-live-coverage.mjs", "prepublishOnly": "npm run check", "typecheck": "tsc --noEmit --composite false --incremental false", "check": "npm run typecheck && npm run build && npm test && npm run check:artifact", diff --git a/scripts/build.mjs b/scripts/build.mjs index e12b434..9eddd1d 100644 --- a/scripts/build.mjs +++ b/scripts/build.mjs @@ -1,9 +1,11 @@ import { rmSync } from 'node:fs'; import { execFileSync } from 'node:child_process'; +import { createRequire } from 'node:module'; import { fileURLToPath } from 'node:url'; const cwd = fileURLToPath(new URL('..', import.meta.url)); // A previous private build must never leave files in the public npm artifact. rmSync(new URL('../dist', import.meta.url), { recursive: true, force: true }); -execFileSync('tsc', ['--build', '--force'], { cwd, stdio: 'inherit' }); +const compiler = createRequire(import.meta.url).resolve('typescript/bin/tsc'); +execFileSync(process.execPath, [compiler, '--build', '--force'], { cwd, stdio: 'inherit' }); await import('./copy-agent-assets.mjs'); diff --git a/scripts/check-public-artifact.mjs b/scripts/check-public-artifact.mjs index da7df93..9c04ec7 100644 --- a/scripts/check-public-artifact.mjs +++ b/scripts/check-public-artifact.mjs @@ -1,3 +1,4 @@ +import { runNpm } from './run-npm.mjs'; import assert from 'node:assert/strict'; import { execFileSync, spawnSync } from 'node:child_process'; import { readFileSync } from 'node:fs'; @@ -6,7 +7,7 @@ import { fileURLToPath } from 'node:url'; const cli = fileURLToPath(new URL('..', import.meta.url)); const run = (command, args, cwd = cli) => execFileSync(command, args, { cwd, encoding: 'utf8' }); -const [artifact] = JSON.parse(run('npm', ['pack', '--dry-run', '--ignore-scripts', '--json'], cli)); +const [artifact] = JSON.parse(runNpm(['pack', '--dry-run', '--ignore-scripts', '--json'], { cwd: cli, encoding: 'utf8' })); for (const file of artifact.files) { assert.doesNotMatch( file.path, diff --git a/scripts/run-npm.mjs b/scripts/run-npm.mjs new file mode 100644 index 0000000..5854e4a --- /dev/null +++ b/scripts/run-npm.mjs @@ -0,0 +1,8 @@ +import { execFileSync } from 'node:child_process'; + +// npm exposes its JS entrypoint to lifecycle scripts on every supported OS. +// Running that entrypoint avoids trying to execute Windows npm.cmd as a binary. +export function runNpm(args, options) { + if (process.env.npm_execpath) return execFileSync(process.execPath, [process.env.npm_execpath, ...args], options); + return execFileSync('npm', args, options); +} diff --git a/scripts/smoke-live-coverage.mjs b/scripts/smoke-live-coverage.mjs new file mode 100644 index 0000000..ff0260d --- /dev/null +++ b/scripts/smoke-live-coverage.mjs @@ -0,0 +1,313 @@ +#!/usr/bin/env node +/** + * Live CLI smoke: runs the built `dm` binary against a real API with a short + * sequence of read-only, credit-free Commands. + * + * DM_API_URL=https://api-staging.v2.dealmachine.com/v1 DM_API_KEY=... npm run smoke:live:coverage + * + * The CLI runs in a child process with a temporary HOME, so it never reads or + * writes a developer's credentials. Its requests go through a local relay that + * records headers and refuses anything outside a read-only allowlist before it + * reaches the API. The key is never printed. + */ + +import { spawn } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as http from 'node:http'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import { + assertAccountShape, + assertEstimateShape, + assertFieldsShape, + assertFiltersShape, + checkAllowedRequest, + parseJsonOutput, + redact, + seedIsolatedConfig, +} from './smoke-live-lib.mjs'; + +const repoRoot = fileURLToPath(new URL('..', import.meta.url)); +const entrypoint = path.join(repoRoot, 'dist', 'index.js'); +const packageVersion = JSON.parse( + fs.readFileSync(path.join(repoRoot, 'package.json'), 'utf-8') +).version; +const COMMAND_TIMEOUT_MS = 60_000; +const UPSTREAM_TIMEOUT_MS = 45_000; + +const apiUrl = process.env.DM_API_URL?.trim(); +const apiKey = process.env.DM_API_KEY?.trim(); +const missing = [!apiUrl && 'DM_API_URL', !apiKey && 'DM_API_KEY'].filter(Boolean); +if (missing.length > 0) { + console.error(`smoke:live:coverage needs ${missing.join(' and ')}.`); + console.error( + 'Example: DM_API_URL=https://api-staging.v2.dealmachine.com/v1 DM_API_KEY= npm run smoke:live:coverage' + ); + process.exit(1); +} + +let target; +try { + target = new URL(apiUrl); + const loopback = ['localhost', '127.0.0.1', '[::1]'].includes(target.hostname); + if (target.pathname.replace(/\/$/, '') !== '/v1' + || target.username || target.password || target.search || target.hash + || !(target.protocol === 'https:' || (target.protocol === 'http:' && loopback))) { + throw new Error('unsafe API base URL'); + } +} catch { + console.error('DM_API_URL must be a plain HTTPS /v1 base URL (HTTP is allowed only for loopback).'); + process.exit(1); +} +const targetBase = target.toString().replace(/\/+$/, ''); + +if (!fs.existsSync(entrypoint)) { + console.error('dist/index.js is missing. Run `npm run build` first or use `npm run smoke:live:coverage`.'); + process.exit(1); +} + +const secrets = [apiKey, apiKey.slice(0, 20)]; +const say = (line) => console.log(redact(line, secrets)); + +// --------------------------------------------------------------------------- +// Local relay: records each CLI request and forwards only allowlisted ones. +// --------------------------------------------------------------------------- + +const requests = []; + +const relay = http.createServer(async (req, res) => { + const chunks = []; + for await (const chunk of req) chunks.push(chunk); + const rawBody = Buffer.concat(chunks); + let body; + if (rawBody.length > 0) { + try { + body = JSON.parse(rawBody.toString('utf-8')); + } catch { + body = undefined; + } + } + + const record = { + method: req.method, + path: req.url, + source: req.headers['x-dealmachine-source'], + userAgent: req.headers['user-agent'], + hasAuthorization: typeof req.headers.authorization === 'string', + refused: undefined, + status: undefined, + }; + requests.push(record); + + const verdict = checkAllowedRequest(req.method, req.url, body); + if (!verdict.allowed) { + record.refused = verdict.reason; + record.status = 403; + res.writeHead(403, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ error: { code: 'smoke_refused', message: `Live smoke refused ${verdict.reason}` } })); + return; + } + + const headers = {}; + for (const name of ['authorization', 'content-type', 'user-agent', 'x-dealmachine-source', 'accept']) { + if (req.headers[name]) headers[name] = req.headers[name]; + } + + try { + const upstream = await fetch(`${targetBase}${req.url}`, { + method: req.method, + headers, + body: rawBody.length > 0 ? rawBody : undefined, + signal: AbortSignal.timeout(UPSTREAM_TIMEOUT_MS), + redirect: 'error', + }); + const responseBody = Buffer.from(await upstream.arrayBuffer()); + record.status = upstream.status; + res.writeHead(upstream.status, { + 'content-type': upstream.headers.get('content-type') || 'application/json', + }); + res.end(responseBody); + } catch (error) { + record.status = 502; + record.refused = `upstream unreachable (${error instanceof Error ? error.name : 'error'})`; + res.writeHead(502, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ error: { code: 'smoke_upstream', message: 'Live smoke could not reach the API' } })); + } +}); + +let tempHome; +try { + await new Promise((resolve, reject) => { + relay.once('error', reject); + relay.listen(0, '127.0.0.1', resolve); + }); + const relayUrl = `http://127.0.0.1:${relay.address().port}`; + + // --------------------------------------------------------------------------- + // Isolated CLI environment + // --------------------------------------------------------------------------- + + tempHome = fs.mkdtempSync(path.join(os.tmpdir(), 'dm-live-smoke-')); + seedIsolatedConfig(tempHome, apiKey); + // Preserve the existing smoke's explicit OS credential-store isolation. HOME + // alone does not determine os.homedir on every supported platform. + const isolation = path.join(tempHome, 'isolation.mjs'); + fs.writeFileSync(isolation, [ + "import os from 'node:os';", + "import childProcess from 'node:child_process';", + "import { syncBuiltinESMExports } from 'node:module';", + `os.homedir = () => ${JSON.stringify(tempHome)};`, + "childProcess.execFileSync = () => { throw new Error('Personal credential store disabled during smoke'); };", + 'syncBuiltinESMExports();', + ].join('\n'), { mode: 0o600 }); + + const childEnv = { + PATH: process.env.PATH ?? '', + HOME: tempHome, + USERPROFILE: tempHome, + TMPDIR: process.env.TMPDIR ?? os.tmpdir(), + ...(process.env.SystemRoot && { SystemRoot: process.env.SystemRoot }), + ...(process.env.TEMP && { TEMP: process.env.TEMP }), + ...(process.env.TMP && { TMP: process.env.TMP }), + DM_API_URL: relayUrl, + DM_QUIET: '1', + CI: 'true', + NO_COLOR: '1', + FORCE_COLOR: '0', + }; + + function runCli(args) { + return new Promise((resolve) => { + const child = spawn(process.execPath, ['--import', pathToFileURL(isolation).href, entrypoint, ...args], { + cwd: tempHome, + env: childEnv, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let stdout = ''; + let stderr = ''; + let timedOut = false; + child.stdout.on('data', (chunk) => (stdout += chunk)); + child.stderr.on('data', (chunk) => (stderr += chunk)); + const timer = setTimeout(() => { + timedOut = true; + child.kill('SIGKILL'); + }, COMMAND_TIMEOUT_MS); + child.on('close', (code) => { + clearTimeout(timer); + resolve({ code, stdout, stderr, timedOut }); + }); + }); + } + + // --------------------------------------------------------------------------- + // Commands + // --------------------------------------------------------------------------- + + const estimateBody = JSON.stringify({ + locations: [{ type: 'zip_code', code: '78704' }], + pagination: { page: 1, per_page: 1 }, + }); + + const steps = [ + { + name: 'dm --version', + args: ['--version'], + expectRequests: [], + check: ({ stdout }) => { + const version = stdout.trim(); + if (version !== packageVersion) throw new Error(`printed "${version}", expected ${packageVersion}`); + return `version ${version}, no network request`; + }, + }, + { + name: 'dm account --json', + args: ['account', '--json'], + expectRequests: [['GET', '/account']], + check: ({ stdout }) => assertAccountShape(parseJsonOutput(stdout)), + }, + { + name: 'dm filters --source-type properties --per-page 5 --json', + args: ['filters', '--source-type', 'properties', '--per-page', '5', '--json'], + expectRequests: [['GET', '/filters']], + check: ({ stdout }) => assertFiltersShape(parseJsonOutput(stdout)), + }, + { + name: 'dm fields --source-type properties --per-page 5 --json', + args: ['fields', '--source-type', 'properties', '--per-page', '5', '--json'], + expectRequests: [['GET', '/fields']], + check: ({ stdout }) => assertFieldsShape(parseJsonOutput(stdout)), + }, + { + name: 'dm properties search --estimate-cost --json (ZIP 78704)', + args: ['properties', 'search', '--body', estimateBody, '--estimate-cost', '--json'], + expectRequests: [['POST', '/properties/search']], + check: ({ stdout }) => assertEstimateShape(parseJsonOutput(stdout)), + }, + ]; + + function checkRequests(seen, expected) { + const refused = seen.find((request) => request.refused); + if (refused) throw new Error(`relay ${refused.status}: ${refused.refused}`); + const summary = seen.map((r) => `${r.method} ${new URL(r.path, relayUrl).pathname}`); + const wanted = expected.map(([method, pathname]) => `${method} ${pathname}`); + if (summary.join(',') !== wanted.join(',')) { + throw new Error(`requests were [${summary.join(', ')}], expected [${wanted.join(', ')}]`); + } + for (const request of seen) { + if (request.source !== 'cli') { + throw new Error(`x-dealmachine-source was ${JSON.stringify(request.source ?? null)}, expected "cli"`); + } + if (!String(request.userAgent ?? '').startsWith(`dm-cli/${packageVersion}`)) { + throw new Error(`user-agent was ${JSON.stringify(request.userAgent ?? null)}`); + } + if (!request.hasAuthorization) throw new Error('request had no Authorization header'); + if (request.status !== 200) throw new Error(`API returned ${request.status}`); + } + } + + function firstLine(text) { + return String(text).trim().split('\n').find(Boolean)?.slice(0, 200) ?? ''; + } + + say(`Live CLI smoke: dm ${packageVersion} against ${targetBase}`); + let failures = 0; + + for (const step of steps) { + const before = requests.length; + const result = await runCli(step.args); + const seen = requests.slice(before); + let reason; + let passed = false; + try { + if (result.timedOut) throw new Error(`timed out after ${COMMAND_TIMEOUT_MS / 1000}s`); + checkRequests(seen, step.expectRequests); + if (result.code !== 0) { + throw new Error(`exit ${result.code}: ${firstLine(result.stderr) || firstLine(result.stdout)}`); + } + reason = step.check(result); + passed = true; + } catch (error) { + reason = error instanceof Error ? error.message : String(error); + } + if (!passed) failures += 1; + say(`${passed ? 'PASS' : 'FAIL'} ${step.name}: ${reason}`); + } + + const headerNote = requests.length > 0 && requests.every((r) => r.source === 'cli') + ? `; all ${requests.length} API requests carried x-dealmachine-source: cli` + : ''; + say( + failures === 0 + ? `Live smoke passed: ${steps.length} of ${steps.length} Commands${headerNote}.` + : `Live smoke failed: ${failures} of ${steps.length} Commands failed.` + ); + process.exitCode = failures === 0 ? 0 : 1; +} catch (error) { + console.error(redact(`Live smoke failed: ${error instanceof Error ? error.message : String(error)}`, secrets)); + process.exitCode = 1; +} finally { + relay.closeAllConnections(); + await new Promise((resolve) => relay.close(resolve)); + if (tempHome) fs.rmSync(tempHome, { recursive: true, force: true }); +} diff --git a/scripts/smoke-live-lib.mjs b/scripts/smoke-live-lib.mjs new file mode 100644 index 0000000..03fe38d --- /dev/null +++ b/scripts/smoke-live-lib.mjs @@ -0,0 +1,174 @@ +/** + * Helpers for the live CLI smoke (scripts/smoke-live-coverage.mjs). + * + * Kept separate so the credential seeding, request allowlist, redaction and + * response shape checks can be unit tested without contacting an API. + */ + +import { createCipheriv, createHash, randomBytes } from 'node:crypto'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +// These mirror the encrypted-file credential store in src/lib/config.ts. The +// unit test reads a seeded home through the real readConfig, so a format +// change there fails the test instead of silently breaking the smoke. +const FALLBACK_ENCRYPTION_CONTEXT = 'dealmachine-cli-api-key-v1'; +const AES_ALGORITHM = 'aes-256-gcm'; +const AES_IV_BYTES = 12; + +/** + * Write a CLI config under `homeDir` that uses the encrypted-file credential + * store. This never calls the macOS Keychain or Windows DPAPI, so the smoke + * cannot read or overwrite a developer's stored key. + */ +export function seedIsolatedConfig(homeDir, apiKey, identity = {}) { + const configDir = path.join(homeDir, '.dealmachine'); + fs.mkdirSync(configDir, { recursive: true, mode: 0o700 }); + + const username = identity.username ?? safeUsername(); + const hostname = identity.hostname ?? os.hostname(); + const key = createHash('sha256') + .update([FALLBACK_ENCRYPTION_CONTEXT, username, hostname, configDir].join('\0')) + .digest(); + const iv = randomBytes(AES_IV_BYTES); + const cipher = createCipheriv(AES_ALGORITHM, key, iv); + const encrypted = Buffer.concat([cipher.update(apiKey, 'utf-8'), cipher.final()]); + const payload = Buffer.concat([iv, encrypted, cipher.getAuthTag()]).toString('base64'); + + const credentialFile = path.join(configDir, 'api-key.enc'); + fs.writeFileSync(credentialFile, payload, { mode: 0o600, encoding: 'utf-8' }); + fs.writeFileSync( + path.join(configDir, 'config.json'), + JSON.stringify( + { + configVersion: 2, + keyId: 'live-smoke', + organizationId: 0, + organizationName: 'Live smoke', + organizationSlug: '', + credentialStore: 'encrypted-file', + credentialFile, + }, + null, + 2 + ), + { mode: 0o600, encoding: 'utf-8' } + ); + return configDir; +} + +function safeUsername() { + try { + return os.userInfo().username; + } catch { + return 'unknown-user'; + } +} + +/** Replace every occurrence of each secret with a fixed marker. */ +export function redact(text, secrets) { + let output = String(text ?? ''); + for (const secret of secrets) { + if (secret && secret.length >= 4) output = output.split(secret).join('[redacted]'); + } + return output; +} + +/** + * Requests the smoke may forward to the live API. Anything else is refused by + * the local relay before it leaves the machine. + */ +export function checkAllowedRequest(method, pathWithQuery, body) { + const pathname = new URL(pathWithQuery, 'http://relay.local').pathname; + if (method === 'GET' && ['/account', '/fields', '/filters'].includes(pathname)) { + return { allowed: true }; + } + if (method === 'POST' && pathname === '/properties/search') { + if (body && typeof body === 'object' && body.estimate_cost === true) { + return { allowed: true }; + } + return { allowed: false, reason: 'property search without estimate_cost=true' }; + } + return { allowed: false, reason: `${method} ${pathname} is not on the read-only allowlist` }; +} + +/** Parse JSON from CLI stdout. Throws with a short reason. */ +export function parseJsonOutput(stdout) { + try { + return JSON.parse(stdout); + } catch { + throw new Error('stdout was not valid JSON'); + } +} + +function fail(reason) { + throw new Error(reason); +} + +function isObject(value) { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} + +function checkPagination(pagination) { + if (!isObject(pagination)) fail('missing pagination object'); + for (const key of ['page', 'per_page', 'total_results', 'total_pages']) { + if (typeof pagination[key] !== 'number') fail(`pagination.${key} is not a number`); + } +} + +/** Shape checks return a short success reason or throw with the failure. */ +export function assertAccountShape(json) { + const data = isObject(json) ? json.data : undefined; + if (!isObject(data)) fail('missing data object'); + if (!isObject(data.organization)) fail('missing data.organization'); + if (!Number.isFinite(data.organization.id) || data.organization.id <= 0) fail('data.organization.id is not a positive number'); + if (typeof data.organization.name !== 'string') fail('data.organization.name is not a string'); + if (!isObject(data.user) || !['api_key', 'oauth'].includes(data.user.authType)) { + fail('missing data.user.authType'); + } + // The API account contract includes plan even though the CLI's display type only lists the fields it renders. + // https://api.docs.dealmachine.com/api-reference/account/get-account + if (!isObject(data.plan)) fail('missing data.plan'); + return `organization ${data.organization.id}, auth ${data.user.authType}`; +} + +function assertCatalogShape(json, idKey, extraCheck) { + if (!isObject(json) || !Array.isArray(json.data)) fail('missing data array'); + if (json.data.length === 0) fail('data array is empty'); + for (const item of json.data) { + if (!isObject(item) || typeof item[idKey] !== 'string') fail(`item without string ${idKey}`); + if (typeof item.name !== 'string') fail(`${item[idKey]} has no name`); + extraCheck?.(item); + } + checkPagination(json.pagination); + return `${json.data.length} of ${json.pagination.total_results} returned`; +} + +export function assertFiltersShape(json) { + return assertCatalogShape(json, 'filter_id', (item) => { + if (!Array.isArray(item.allowed_operators)) { + fail(`${item.filter_id} has no allowed_operators array`); + } + }); +} + +export function assertFieldsShape(json) { + return assertCatalogShape(json, 'field_id'); +} + +export function assertEstimateShape(json) { + if (!isObject(json)) fail('response is not an object'); + if ('data' in json || 'credits' in json) fail('response contains records or charged credits'); + const estimate = json.estimated_credits; + if (!isObject(estimate)) fail('missing estimated_credits'); + if (!Number.isFinite(estimate.this_page) || estimate.this_page < 0) fail('estimated_credits.this_page is not a nonnegative number'); + if (!Number.isFinite(estimate.total_all_pages) || estimate.total_all_pages < 0) { + fail('estimated_credits.total_all_pages is not a nonnegative number'); + } + if (!isObject(json.totals) || !Number.isFinite(json.totals.properties) || json.totals.properties < 0) { + fail('missing totals.properties'); + } + checkPagination(json.pagination); + return `${json.totals.properties} properties, ${estimate.this_page} credits estimated for page 1`; +} diff --git a/scripts/test-package.mjs b/scripts/test-package.mjs index 80c4158..e398553 100644 --- a/scripts/test-package.mjs +++ b/scripts/test-package.mjs @@ -1,3 +1,4 @@ +import { runNpm } from './run-npm.mjs'; import assert from 'node:assert/strict'; import { execFile, execFileSync } from 'node:child_process'; import { createServer } from 'node:http'; @@ -6,12 +7,13 @@ import { promisify } from 'node:util'; import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { fileURLToPath } from 'node:url'; +import { fileURLToPath, pathToFileURL } from 'node:url'; const root = fileURLToPath(new URL('..', import.meta.url)); const temporary = mkdtempSync(join(tmpdir(), 'dm-package-')); const version = JSON.parse(readFileSync(join(root, 'package.json'), 'utf8')).version; -const run = (command, args, cwd = temporary) => execFileSync(command, args, { cwd, encoding: 'utf8', timeout: 120000 }); +const run = (command, args, cwd = temporary) => (command === 'npm' ? runNpm(args, { cwd, encoding: 'utf8', timeout: 120000 }) : execFileSync(command, args, { cwd, encoding: 'utf8', timeout: 120000 })); +const dm = args => run('npm', ['exec', '--offline', '--', 'dm', ...args]); try { run('npm', ['run', 'build'], root); run(process.execPath, ['scripts/check-public-artifact.mjs'], root); @@ -21,18 +23,18 @@ try { }); writeFileSync(join(temporary, 'package.json'), JSON.stringify({ name: 'dm-install-check', private: true })); run('npm', ['install', '--ignore-scripts', '--no-audit', '--no-fund', ...archives]); - const binary = join(temporary, 'node_modules/.bin/dm'); - assert.equal(run(binary, ['--version']).trim(), version); + const binary = join(temporary, 'node_modules/dealmachine/bin/dm.js'); + assert.equal(dm(['--version']).trim(), version); for (const entry of ['@dealmachine/cli/dist/index.js', 'dealmachine/bin/dm.js']) { assert.equal(run(process.execPath, [join(temporary, 'node_modules', entry), '--version']).trim(), version); } - assert.match(run(binary, ['--help']), /properties/); - assert.match(run(binary, ['prospects', '--help']), /add/); - assert.match(run(binary, ['webhooks', '--help']), /events/); - const playbook = JSON.parse(run(binary, ['agents', 'playbook', '--json'])); + assert.match(dm(['--help']), /properties/); + assert.match(dm(['prospects', '--help']), /add/); + assert.match(dm(['webhooks', '--help']), /events/); + const playbook = JSON.parse(dm(['agents', 'playbook', '--json'])); assert.match(playbook.content, /name: dealmachine/); assert.match(playbook.content, /specific name always uses person enrichment/); - run(binary, ['agents', 'install', 'claude-code', '--project', '--json']); + dm(['agents', 'install', 'claude-code', '--project', '--json']); assert.equal(readFileSync(join(temporary, '.claude/skills/dealmachine/SKILL.md'), 'utf8'), playbook.content); const program = run(process.execPath, ['--input-type=module', '--eval', "const { program } = await import('@dealmachine/cli/dist/index.js'); console.log(program.name());"]); assert.equal(program.trim(), 'dm'); @@ -49,7 +51,7 @@ try { "childProcess.execFileSync = () => { throw new Error('OS credential store disabled in package fixture'); };", 'syncBuiltinESMExports();', ].join('\n')); - run(process.execPath, ['--import', credentialIsolation, '--input-type=module', '--eval', + run(process.execPath, ['--import', pathToFileURL(credentialIsolation).href, '--input-type=module', '--eval', "const { writeConfig } = await import('@dealmachine/cli/dist/lib/config.js'); writeConfig({ apiKey: 'dm-package-test', keyId: 'fixture', organizationId: 1, organizationName: 'Package fixture', organizationSlug: 'fixture' });", ]); const fixtureConfig = JSON.parse(readFileSync(join(fixtureHome, '.dealmachine/config.json'), 'utf8')); @@ -79,7 +81,7 @@ try { [['lists', 'import', '123', '--ids', '456'], '/lists/123/import'], ]) { for (const optOut of [false, true]) { - await execute(process.execPath, ['--import', credentialIsolation, binary, ...args, '--json', ...(optOut ? ['--no-prospects'] : [])], { + await execute(process.execPath, ['--import', pathToFileURL(credentialIsolation).href, binary, ...args, '--json', ...(optOut ? ['--no-prospects'] : [])], { cwd: temporary, timeout: 10000, env: { ...process.env, DM_API_URL: `http://127.0.0.1:${server.address().port}/v1`, DM_API_KEY: 'unrelated-environment-key' }, diff --git a/tests/factory/FAL-311.json b/tests/factory/FAL-311.json new file mode 100644 index 0000000..f5d9189 --- /dev/null +++ b/tests/factory/FAL-311.json @@ -0,0 +1,71 @@ +{ + "version": 1, + "issue": "FAL-311", + "criteria": [ + { + "id": "AC-1", + "text": "** Scenario 1: Missing credentials\nGIVEN either DM_API_URL or DM_API_KEY is missing,\nWHEN I run the live smoke command,\nTHEN it exits with code 1 and names the missing variable,\nAND it does not print the API key." + }, + { + "id": "AC-2", + "text": "** Scenario 2: Read-only live commands\nGIVEN DM_API_URL and DM_API_KEY are set,\nWHEN I run the live smoke command,\nTHEN it runs the built CLI version, account, filters, fields and estimated-cost property search checks,\nAND the local relay blocks requests outside the approved read-only list and checks the source header, User-Agent and authentication." + }, + { + "id": "AC-3", + "text": "** Scenario 3: Isolated credentials\nGIVEN I run the live smoke command,\nWHEN it prepares CLI configuration,\nTHEN it uses a temporary home and does not access the operator's Keychain or saved CLI settings.\n---\nFiled by Alma because this work had no Linear ticket. Source: [dealmachine-cli #25]()" + } + ], + "checks": [ + { + "title": "CLI missing credentials and isolation regression", + "surface": "cli", + "repository": "cli", + "preconditions": [ + "Run the reviewed CLI candidate with the approved Factory executor." + ], + "steps": [ + "Run the CLI check suite, including the built-CLI smoke regressions." + ], + "expected": "Missing environment variables exit 1 without printing secrets or making requests; temporary credential setup, failure cleanup and caller settings isolation pass.", + "criterionId": "AC-1", + "execution": "automated", + "automation": { + "taskId": "cli:check" + } + }, + { + "title": "Built CLI against read-only staging API", + "surface": "cli", + "repository": "cli", + "preconditions": [ + "Run the reviewed CLI candidate with the approved Factory executor." + ], + "steps": [ + "Run the expanded live smoke on the dedicated staging fixture." + ], + "expected": "Five built Commands pass, four allowlisted API requests pass header and authentication checks, cost estimation charges no credits, and temporary credentials are removed.", + "criterionId": "AC-2", + "execution": "automated", + "automation": { + "taskId": "cli:smoke:live:coverage" + } + }, + { + "title": "CLI missing credentials and isolation regression", + "surface": "cli", + "repository": "cli", + "preconditions": [ + "Run the reviewed CLI candidate with the approved Factory executor." + ], + "steps": [ + "Run the CLI check suite, including the built-CLI smoke regressions." + ], + "expected": "Missing environment variables exit 1 without printing secrets or making requests; temporary credential setup, failure cleanup and caller settings isolation pass.", + "criterionId": "AC-3", + "execution": "automated", + "automation": { + "taskId": "cli:check" + } + } + ] +} diff --git a/tests/factory/regression/cli-live-smoke-coverage.json b/tests/factory/regression/cli-live-smoke-coverage.json new file mode 100644 index 0000000..7e12760 --- /dev/null +++ b/tests/factory/regression/cli-live-smoke-coverage.json @@ -0,0 +1,26 @@ +[ + { + "id": "cli-live-smoke-coverage", + "revision": 1, + "spec": { + "title": "CLI expanded live smoke and Windows credential isolation", + "surface": "cli", + "repository": "cli", + "preconditions": [ + "Use the exact reviewed FAL-311 PR candidate with npm ci and a supported Node version.", + "The QA operator has an approved staging API URL ending in /v1 and a dedicated QA API key through the approved private runtime; never paste it into evidence.", + "Run on macOS and Windows; preserve the caller's existing CLI settings and OS credential store. No production target, account write or paid lookup is permitted." + ], + "steps": [ + "Run npm run check and npm run test:package. Verify the built-CLI integration tests and the unchanged package baseline pass.", + "Without DM_API_URL, then without DM_API_KEY, run npm run smoke:live:coverage. Each attempt exits 1, names the missing variable, makes no API request and prints no key.", + "On the approved staging target, run npm run smoke:live:coverage on macOS and Windows. Verify all five Commands pass: version, account, filters, fields and property-search cost estimate; request source and User-Agent checks pass and no records or charged credits are returned.", + "Confirm the caller's saved CLI configuration and OS credential-store entries are unchanged and the smoke's temporary directory is removed. Record the operating system, Node version, source commit, target environment and redacted command results." + ], + "expected": "Every command, isolation and cleanup assertion passes on the exact candidate on both operating systems. Missing configuration refuses before network use. Only four read-only API requests occur; property search uses estimate_cost. Live proof stays separate from the synthetic integration test and original three-Command Factory baseline. Any missing platform, credential or failed step remains unverified.", + "criterionId": "FAL-311-AC-1-3", + "execution": "human-required", + "executionReason": "Factory runs the registered staging smoke with the approved dedicated credential. Full macOS and Windows isolation proof comes from the authorized local coding session or an approved platform runner; Factory's Linux result alone does not verify both platforms." + } + } +] diff --git a/tests/scripts/smokeLive.integration.test.ts b/tests/scripts/smokeLive.integration.test.ts new file mode 100644 index 0000000..78244d7 --- /dev/null +++ b/tests/scripts/smokeLive.integration.test.ts @@ -0,0 +1,133 @@ +import { execFile } from 'node:child_process'; +import { createServer } from 'node:http'; +import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { promisify } from 'node:util'; +import { pathToFileURL } from 'node:url'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +const execute = promisify(execFile); +const syntheticKey = 'dm_sk_synthetic_smoke_integration_only'; +const pagination = { page: 1, per_page: 5, total_results: 1, total_pages: 1 }; +const requests: Array<{ method: string; path: string; source?: string; authorization?: string; agent?: string; body: unknown }> = []; +const server = createServer(async (request, response) => { + const chunks: Buffer[] = []; + for await (const chunk of request) chunks.push(Buffer.from(chunk)); + const raw = Buffer.concat(chunks).toString(); + const path = new URL(request.url!, 'http://localhost').pathname; + requests.push({ method: request.method!, path, source: request.headers['x-dealmachine-source'] as string, + authorization: request.headers.authorization, agent: request.headers['user-agent'], body: raw ? JSON.parse(raw) : undefined }); + const data = path === '/v1/account' ? { data: { organization: { id: 1, name: 'Synthetic QA' }, user: { authType: 'api_key' }, plan: { name: 'QA' } } } + : path === '/v1/fields' ? { data: [{ field_id: 'property_id', name: 'Property ID' }], pagination } + : path === '/v1/filters' ? { data: [{ filter_id: 'property_id', name: 'Property ID', allowed_operators: ['eq'] }], pagination } + : path === '/v1/properties/search' ? { totals: { properties: 1 }, estimated_credits: { this_page: 1, total_all_pages: 1 }, pagination } + : { error: 'Unexpected request' }; + response.writeHead(path.startsWith('/v1/') ? 200 : 404, { 'content-type': 'application/json' }); + response.end(JSON.stringify(data)); +}); +let target: string; +let callerHome: string; +let callerSentinel: string; +let smokeTemporary: string; + +beforeAll(async () => { + await execute(process.execPath, ['scripts/build.mjs'], { timeout: 60_000 }); + await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)); + const address = server.address() as { port: number }; + target = `http://127.0.0.1:${address.port}/v1`; + callerHome = await mkdtemp(join(tmpdir(), 'dm-smoke-caller-')); + await mkdir(join(callerHome, '.dealmachine')); + callerSentinel = join(callerHome, '.dealmachine', 'config.json'); + await writeFile(callerSentinel, 'leave caller settings unchanged'); + // Exercise URL-significant characters as well as spaces in the import path. + smokeTemporary = await mkdtemp(join(tmpdir(), 'dm-smoke-tmp #')); +}, 60_000); +afterAll(async () => { + await new Promise(resolve => server.close(() => resolve())); + await rm(callerHome, { recursive: true, force: true }); + await rm(smokeTemporary, { recursive: true, force: true }); +}); + +async function smoke(apiUrl?: string, apiKey?: string, preload?: string) { + try { + const result = await execute(process.execPath, [...(preload ? ['--import', pathToFileURL(preload).href] : []), 'scripts/smoke-live-coverage.mjs'], { + env: { PATH: process.env.PATH, HOME: callerHome, USERPROFILE: callerHome, ...(process.env.SystemRoot && { SystemRoot: process.env.SystemRoot }), + TMPDIR: smokeTemporary, TEMP: smokeTemporary, TMP: smokeTemporary, + ...(apiUrl !== undefined && { DM_API_URL: apiUrl }), ...(apiKey !== undefined && { DM_API_KEY: apiKey }) }, + timeout: 15_000, + }); + return { ...result, code: 0 }; + } catch (error) { + return error as { stdout: string; stderr: string; code: number }; + } +} + +describe('built CLI live smoke', () => { + it('removes temporary credentials and closes the relay when config setup fails', async () => { + const preload = join(callerHome, 'fail-config-write.mjs'); + await writeFile(preload, ` + import fs from 'node:fs'; + import { basename, dirname, join } from 'node:path'; + import { syncBuiltinESMExports } from 'node:module'; + const write = fs.writeFileSync; + fs.writeFileSync = (file, ...args) => { + if (basename(String(file)) === 'config.json' && fs.existsSync(join(dirname(String(file)), 'api-key.enc'))) { + throw new Error('injected config setup failure ${syntheticKey}'); + } + return write(file, ...args); + }; + syncBuiltinESMExports(); + `); + requests.length = 0; + const result = await smoke(target, syntheticKey, preload); + expect(result.code).toBe(1); + expect(result.stderr).toContain('injected config setup failure'); + expect(await readdir(smokeTemporary)).toEqual([]); + expect(result.stdout + result.stderr).not.toContain(syntheticKey); + expect(await readFile(callerSentinel, 'utf8')).toBe('leave caller settings unchanged'); + expect(requests).toHaveLength(0); + }); + + it('exercises all read-only Commands through the relay and leaves caller settings unchanged', async () => { + requests.length = 0; + const result = await smoke(target, syntheticKey); + expect(result.code, result.stderr + result.stdout).toBe(0); + expect(result.stdout).toContain('5 of 5 Commands'); + expect(result.stdout + result.stderr).not.toContain(syntheticKey); + expect(requests.map(request => `${request.method} ${request.path}`)).toEqual([ + 'GET /v1/account', 'GET /v1/filters', 'GET /v1/fields', 'POST /v1/properties/search', + ]); + for (const request of requests) { + expect(request.source).toBe('cli'); + expect(request.agent).toMatch(/^dm-cli\//); + expect(request.authorization).toBe(`Bearer ${syntheticKey}`); + } + expect(requests.at(-1)?.body).toMatchObject({ estimate_cost: true }); + expect(await readFile(callerSentinel, 'utf8')).toBe('leave caller settings unchanged'); + expect(await readdir(join(callerHome, '.dealmachine'))).toEqual(['config.json']); + expect(await readdir(smokeTemporary)).toEqual([]); + }); + + it.each([['DM_API_URL', undefined, syntheticKey], ['DM_API_KEY', 'http://127.0.0.1/v1', undefined]])( + 'names missing %s without using saved credentials', async (name, apiUrl, apiKey) => { + requests.length = 0; + const result = await smoke(apiUrl, apiKey); + expect(result.code).toBe(1); + expect(result.stderr).toContain(name); + expect(result.stdout + result.stderr).not.toContain(syntheticKey); + expect(requests).toHaveLength(0); + }, + ); + + it('rejects credentials, query strings, fragments and incorrect API paths before starting Commands', async () => { + for (const invalid of [target.replace('http://', 'http://user:password@'), `${target}?unsafe=1`, `${target}#fragment`, target.replace('/v1', '/other')]) { + requests.length = 0; + const result = await smoke(invalid, syntheticKey); + expect(result.code).toBe(1); + expect(result.stderr).toContain('DM_API_URL must be a plain HTTPS /v1 base URL'); + expect(requests).toHaveLength(0); + expect(result.stdout + result.stderr).not.toContain('user:password'); + } + }); +}); diff --git a/tests/scripts/smokeLive.test.ts b/tests/scripts/smokeLive.test.ts new file mode 100644 index 0000000..c5758f5 --- /dev/null +++ b/tests/scripts/smokeLive.test.ts @@ -0,0 +1,115 @@ +import { describe, it, expect, vi, afterAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as nodeOs from 'node:os'; +import * as path from 'node:path'; + +const { tempHome } = vi.hoisted(() => { + const { mkdtempSync } = require('node:fs') as typeof import('node:fs'); + const { tmpdir } = require('node:os') as typeof import('node:os'); + const { join } = require('node:path') as typeof import('node:path'); + return { tempHome: mkdtempSync(join(tmpdir(), 'dm-smoke-test-')) }; +}); + +vi.mock('node:os', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, homedir: () => tempHome }; +}); + +// The smoke must never reach the macOS Keychain or Windows DPAPI. +vi.mock('node:child_process', () => ({ + execFileSync: vi.fn(() => { + throw new Error('credential store must not be used'); + }), +})); + +import { + assertAccountShape, + assertEstimateShape, + assertFieldsShape, + assertFiltersShape, + checkAllowedRequest, + redact, + seedIsolatedConfig, +} from '../../scripts/smoke-live-lib.mjs'; +import { readConfig } from '../../src/lib/config'; + +const pagination = { page: 1, per_page: 5, total_results: 10, total_pages: 2 }; + +afterAll(() => { + fs.rmSync(tempHome, { recursive: true, force: true }); +}); + +describe('live smoke helpers', () => { + it('seeds a config the real CLI reads without a system credential store', async () => { + expect(nodeOs.homedir()).toBe(tempHome); + seedIsolatedConfig(tempHome, 'dm_sk_test_example_key'); + + const config = readConfig(); + expect(config?.apiKey).toBe('dm_sk_test_example_key'); + const persisted = JSON.parse( + fs.readFileSync(path.join(tempHome, '.dealmachine', 'config.json'), 'utf-8') + ); + expect(persisted.credentialStore).toBe('encrypted-file'); + expect(persisted.apiKey).toBeUndefined(); + const { execFileSync } = await import('node:child_process'); + expect(execFileSync).not.toHaveBeenCalled(); + }); + + it('redacts every occurrence of each secret', () => { + expect(redact('key abc12345 and abc12345 again', ['abc12345'])).toBe( + 'key [redacted] and [redacted] again' + ); + expect(redact('unchanged', ['', undefined as unknown as string])).toBe('unchanged'); + }); + + it('allows only read-only requests', () => { + expect(checkAllowedRequest('GET', '/account', undefined).allowed).toBe(true); + expect(checkAllowedRequest('GET', '/filters?source_type=properties', undefined).allowed).toBe(true); + expect(checkAllowedRequest('GET', '/fields?per_page=5', undefined).allowed).toBe(true); + expect( + checkAllowedRequest('POST', '/properties/search', { estimate_cost: true }).allowed + ).toBe(true); + expect(checkAllowedRequest('POST', '/properties/search', {}).allowed).toBe(false); + expect( + checkAllowedRequest('POST', '/properties/search', { estimate_cost: 'true' }).allowed + ).toBe(false); + expect(checkAllowedRequest('DELETE', '/account', undefined).allowed).toBe(false); + expect(checkAllowedRequest('POST', '/lists', { name: 'x' }).allowed).toBe(false); + }); + + it('checks account, catalog and estimate shapes', () => { + expect( + assertAccountShape({ + data: { + organization: { id: 1, name: 'Org' }, + user: { id: null, authType: 'api_key' }, + plan: { name: 'Plan' }, + }, + }) + ).toContain('organization 1'); + expect(() => assertAccountShape({ data: { organization: { id: '1' } } })).toThrow( + 'data.organization.id' + ); + + expect( + assertFiltersShape({ + data: [{ filter_id: 'f1', name: 'Filter', allowed_operators: ['eq'] }], + pagination, + }) + ).toBe('1 of 10 returned'); + expect(() => + assertFiltersShape({ data: [{ filter_id: 'f1', name: 'Filter' }], pagination }) + ).toThrow('allowed_operators'); + expect(() => assertFieldsShape({ data: [], pagination })).toThrow('empty'); + + const estimate = { + totals: { properties: 12, people: 20 }, + pagination, + estimated_credits: { this_page: 3, total_all_pages: 40 }, + }; + expect(assertEstimateShape(estimate)).toContain('12 properties'); + expect(() => assertEstimateShape({ ...estimate, data: [], credits: { used: 1 } })).toThrow( + 'charged credits' + ); + }); +});