From 8f873de17de47190a33bffe572be568efdb331b9 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 22:25:27 +0000 Subject: [PATCH 001/134] feat: document env portability trust model --- docs/env.md | 30 ++ ...ortability-and-secret-management-design.md | 400 ++++++++++++++++++ ...-portability-and-secret-management-plan.md | 278 ++++++++++++ src/commands/env.ts | 54 +++ src/lib/hack-env.ts | 50 +++ tests/env-backend-command.test.ts | 107 +++++ 6 files changed, 919 insertions(+) create mode 100644 docs/plans/2026-03-13-env-portability-and-secret-management-design.md create mode 100644 docs/plans/2026-03-13-env-portability-and-secret-management-plan.md diff --git a/docs/env.md b/docs/env.md index 3314d7971..02905d506 100644 --- a/docs/env.md +++ b/docs/env.md @@ -2,6 +2,28 @@ hack supports a project-scoped env contract (shareable, no values) plus safe secret storage for local development. +## Trust model at a glance + +Hack currently has three env layers, and they should be understood separately: + +- Contract: `.hack/hack.env.json` defines what keys exist and how they should materialize locally. +- Local values: `.hack/.env` stores local plaintext values for `plain_env`. +- Local secrets: the configured secret backend stores secret values for keys that should not live in `.hack/.env`. + +Current status: + +- The contract is portable and safe to commit. +- Local values and local secrets are not automatically portable across machines. +- Remote env portability is a follow-on design, not the current default. + +Planned direction: + +- portable env values will live in an immutable encrypted bundle artifact +- bundle access will be controlled through explicit project-key sharing +- applying a bundle will still write values back into `.hack/.env` and the configured local secret backend so existing runtime behavior stays intact + +See `docs/plans/2026-03-13-env-portability-and-secret-management-design.md` for the full portable env and key-sharing model. + ## Files and storage - `.hack/hack.env.json` (committed): declares env vars, required vs optional, per-service scope, and where values should come from. @@ -34,6 +56,7 @@ hack supports a project-scoped env contract (shareable, no values) plus safe sec ``` Fields: + - `key`: uppercase snake-case env var name (e.g. `AWS_PROFILE`). - `required`: if true, `hack up/run/restart` fails when missing (for targeted services). - `source`: @@ -46,6 +69,7 @@ Fields: - `hack env list [--json] [--show-secrets]` - shows contract + resolution state + - includes a storage summary for the committed contract, local `.hack/.env`, configured secret backend, and current portable-state status - exits `1` if required vars are missing - `hack env set KEY=VALUE` - writes to `.hack/.env` @@ -59,6 +83,7 @@ Fields: - sets global backend strategy for multi-node/env secret storage Notes: + - `hack env set` also supports interactive prompting when `KEY` or `VALUE` is omitted. - Keychain mode uses service name `hack-` (project name from `.hack/hack.config.json`). - Encrypted-file mode uses `HACK_SECRETS_FILE_KEY` (if set) before keychain key lookup (`hack-secrets-backend/encrypted-file-key`). @@ -89,6 +114,7 @@ Global config supports backend selection for secret portability: ``` Fields: + - `backend`: `keychain` | `encrypted_file` | `cloud`. - `allowEnvAuthRefs`: allow `env:VAR_NAME` auth/secret references for node/controller workflows. - `encryptedFile.path`: target path for encrypted local store mode. @@ -106,22 +132,26 @@ When you run `hack up`, `hack restart`, or `hack run`, hack: 4. Invokes `docker compose` with an environment that includes resolved values (including secret backend values). Security posture: + - Secret values are never written into `.hack/` YAML files. - Plain env values live in `.hack/.env` (expected to be gitignored in most repos). ## Remote node secret behavior Current behavior (today): + - Node auth tokens and extension auth refs are stored through the configured secret backend. - `env:VAR_NAME` auth refs are allowed only when `controlPlane.secrets.allowEnvAuthRefs=true`. - Remote dispatch/workspace bootstrap does **not** automatically copy all host env values to remote nodes. Recommended setup: + - Keep controller secrets in `keychain` or `encrypted_file` backend (not shell env). - Configure each remote node to use the same secret backend strategy where possible. - Use project env contract (`.hack/hack.env.json`) to make required keys explicit before runs. Planned direction: + - Add an explicit host-to-node secret sync command with least-privilege scoping, encrypted payload delivery, and audit events per run. ## Daemon/gateway API (UI integration) diff --git a/docs/plans/2026-03-13-env-portability-and-secret-management-design.md b/docs/plans/2026-03-13-env-portability-and-secret-management-design.md new file mode 100644 index 000000000..29664444c --- /dev/null +++ b/docs/plans/2026-03-13-env-portability-and-secret-management-design.md @@ -0,0 +1,400 @@ +# Env Portability And Secret Management Design + +## Context + +Hack already has the beginnings of a useful env system: + +- `.hack/hack.env.json` defines the committed project env contract +- `.hack/.env` stores local non-secret values +- the configured secret backend stores local secret values +- `hack env` can list, set, unset, and resolve values for runtime injection + +What is still missing is an explicit product model for portability, sharing, recovery, and operator trust. Today the user can understand the local contract, but not yet the full lifecycle of: + +- where portable values would live +- how encrypted values are modeled +- how another human or machine should gain access +- how key loss or rotation should behave +- how all of this preserves `.env` compatibility instead of replacing it + +This design defines that product boundary for the full stream. It treats the existing remote encrypted env portability plan as one child slice inside the larger env portability and secret-management program. + +## Goals + +- Make the storage model explicit for committed, local, and portable env state. +- Define a portable project env artifact schema that is independent from any one device or local keychain. +- Define a separate encrypted key-sharing model so access is explicit and reviewable. +- Make rotation and recovery behavior safe enough to explain before it ships. +- Preserve `.env` compatibility and existing backend choices instead of forcing a new runtime model. +- Break the work into coherent child streams that can be implemented independently. + +## Non-Goals + +- Do not make local-only Hack usage require Hack account auth. +- Do not turn the broker into a plaintext env editor. +- Do not silently copy host env values to remote nodes. +- Do not auto-share env access to every org or team member. +- Do not replace `.hack/hack.env.json` with a value-bearing committed file. + +## Approved Product Rules + +### Local remains the default + +The current local behavior remains valid: + +- `.hack/hack.env.json` is the committed contract +- `.hack/.env` remains the local plaintext materialization for non-secret values +- the configured secret backend remains the local materialization for secret values +- `hack up`, `hack run`, and `hack restart` continue resolving local values first + +Portable env management is opt-in per project binding. If a user never enables it, Hack should continue behaving as a local-only tool. + +### Portability is a separate value artifact + +The committed contract and the portable value artifact are different objects: + +- `.hack/hack.env.json` describes what keys exist and how they should materialize locally +- the portable artifact stores encrypted values plus ownership and lineage metadata + +This avoids putting secrets in git while still making the portable state explicit and inspectable. + +### Shared or remote state requires Hack account ownership + +Portable env bundles are owned by a Hack scope: + +- user +- organization +- team + +Provider identity is not the owner. A GitHub login method, Linear profile, or local keychain identity is not the durable ownership layer for portable env state. + +## Explicit Storage Model + +Hack should describe env storage in three layers. + +### 1. Committed contract + +File: + +- `.hack/hack.env.json` + +Purpose: + +- declares required keys +- declares service scoping +- declares local materialization intent + +This file stays safe to commit because it contains no values. + +### 2. Local materialization + +Files/backends: + +- `.hack/.env` for `plain_env` +- configured secret backend for secret material + +Purpose: + +- stores the values a local machine actually uses for runtime injection +- stays compatible with existing Docker Compose env handling and local debugging workflows + +### 3. Portable encrypted state + +Artifact: + +- `hack.project-env-bundle/v1` + +Purpose: + +- stores an immutable encrypted snapshot of project env values +- supports broker storage and explicit export/import +- is portable across machines because it does not rely on one local keychain as the only durable copy + +Portable state must never be described as "the `.env` file in the cloud." It is an encrypted artifact that can be applied back into `.hack/.env` and the local secret backend. + +## Portable Artifact Schema + +The portable value artifact is one immutable bundle version. It may live in the broker and may also be exportable as a file for backup or manual transport. + +Recommended schema: + +```json +{ + "$schema": "https://schemas.hack/hack.project-env-bundle.schema.json", + "kind": "hack.project-env-bundle", + "version": 1, + "bundleId": "peb_01JQ...", + "projectBindingKey": "user:usr_123/my-project/default", + "environment": "default", + "ownerScope": { + "type": "user", + "id": "usr_123" + }, + "createdAt": "2026-03-13T12:00:00.000Z", + "createdBy": "usr_123", + "supersedesBundleId": null, + "contract": { + "path": ".hack/hack.env.json", + "version": 1, + "digest": "sha256:2e7d..." + }, + "crypto": { + "scheme": "project-key-envelope-v1", + "projectKeyId": "pek_01JQ...", + "bundleKeyId": "bek_01JQ...", + "bundleKeyWrap": { + "wrapType": "project_key", + "wrappedBundleKey": "base64..." + }, + "entryAlgorithm": "aes-256-gcm" + }, + "entries": [ + { + "key": "DATABASE_URL", + "required": true, + "services": ["api", "worker"], + "materialization": { + "type": "secret_backend", + "backendHint": "preserve_current" + }, + "ciphertext": "base64...", + "iv": "base64...", + "tag": "base64...", + "digest": "sha256:8b1a..." + }, + { + "key": "AWS_PROFILE", + "required": true, + "services": ["api"], + "materialization": { + "type": "plain_env" + }, + "ciphertext": "base64...", + "iv": "base64...", + "tag": "base64...", + "digest": "sha256:3a7d..." + } + ] +} +``` + +### Schema rules + +- Every bundle version is immutable. +- All values are encrypted in the bundle, including values that will later be written into `.hack/.env`. +- `materialization.type` tells Hack where to place the decrypted value during apply: + - `plain_env` writes to `.hack/.env` + - `secret_backend` writes to the configured secret backend +- `backendHint` is advisory only. It should never force a backend switch during apply. +- `projectBindingKey` is stable across versions and is the join point for bundle history, key shares, and audit. + +## Encrypted Entry And Key Model + +The stream needs two different key concepts. + +### Project key + +The project key is a long-lived wrapping key for one project binding. It is not the value-encryption key for every entry directly. Its job is to make sharing, rotation, and recovery explicit. + +Responsibilities: + +- wraps each bundle version key +- is the object shared with collaborators or recovery targets +- can be rotated without forcing an operator to hand-edit every env value + +### Bundle key + +Each bundle version gets a fresh random bundle key. + +Responsibilities: + +- encrypts each entry payload in the bundle +- is wrapped by the project key +- is discarded from plaintext after publish/apply completes + +This separation keeps snapshot publication simple and makes project-key sharing possible without treating one giant ciphertext blob as the only unit of control. + +## Manual Project-Key Sharing Model + +Sharing access to a portable project env must be manual and additive. The portable bundle and the key share record are separate objects. + +Recommended key-share schema: + +```json +{ + "$schema": "https://schemas.hack/hack.project-env-key-share.schema.json", + "kind": "hack.project-env-key-share", + "version": 1, + "shareId": "pks_01JQ...", + "projectBindingKey": "user:usr_123/my-project/default", + "projectKeyId": "pek_01JQ...", + "recipient": { + "type": "user", + "id": "usr_456" + }, + "wrapMethod": "x25519-sealed-box-v1", + "wrappedProjectKey": "base64...", + "createdAt": "2026-03-13T12:15:00.000Z", + "createdBy": "usr_123", + "revokedAt": null, + "lastUsedAt": null +} +``` + +### Sharing rules + +- Sharing happens one recipient at a time. +- The broker may store key-share records, but it should not auto-grant them based on org membership alone. +- A user or team receiving access gets a wrapped project key, not plaintext env values. +- Revocation removes future access by invalidating the share record and blocking fresh unwrap operations. It does not rewrite old local materializations that a recipient already applied. + +Manual sharing is the explicit trust boundary. If a project owner never creates a share, nobody else gets access by default. + +## Rotation And Recovery Behavior + +The program needs three different operator actions, with different UX and audit implications. + +### 1. Value rotation + +Use when actual env values changed. + +Behavior: + +- publish a new immutable bundle version +- mint a fresh bundle key +- keep the same project key unless crypto rotation is also requested + +### 2. Share rotation + +Use when recipients change. + +Behavior: + +- add or revoke key-share records +- do not rewrite bundle values +- do not require env republish unless an operator explicitly wants a new bundle + +### 3. Project-key rotation + +Use when the project key itself is suspected compromised or when the owner wants to re-anchor trust. + +Behavior: + +- mint a new project key +- re-wrap the latest active bundle keys to the new project key +- require explicit regeneration of all active key-share records +- keep prior project-key lineage in audit metadata + +### Recovery policy + +Hack should not silently escrow plaintext secrets. Recovery must be explicit. + +Required rule: + +- portable env enablement must require at least one recovery path before the operator can delete the only local authority + +Allowed recovery paths: + +- an encrypted recovery package exported by the operator +- a second admin/owner key share +- a designated recovery recipient created manually + +Unsafe behavior to reject: + +- deleting the last active owner share with no recovery path +- rotating the project key while leaving no valid recipient or backup +- revoking the last usable recovery share without an explicit forced override + +## `.env` Compatibility And Backend UX + +Portability should preserve the current local runtime model instead of replacing it. + +### Apply behavior + +When a portable bundle is applied locally: + +- entries with `materialization.type = plain_env` are written to `.hack/.env` +- entries with `materialization.type = secret_backend` are written to the configured secret backend +- the contract file remains `.hack/hack.env.json` + +This means portability changes where values are sourced from, not how local services consume them. + +### UX rule + +Hack should always show three things separately: + +- contract location +- local value location +- portable copy state + +Example user-facing explanations: + +- `AWS_PROFILE`: + - contract: `.hack/hack.env.json` + - local value: `.hack/.env` + - portable copy: enabled, latest bundle `peb_...` +- `DATABASE_URL`: + - contract: `.hack/hack.env.json` + - local value: `keychain` + - portable copy: enabled, protected by project key `pek_...` + +That separation is what makes the trust model understandable. + +## Child Streams + +The program should be implemented as five child streams. + +### Stream 1: Current-state trust model and operator UX + +Deliverables: + +- clearer docs for current local storage +- CLI or UI surfaces that explain contract vs local vs portable state +- explicit trust-boundary copy for backends and remote ownership + +### Stream 2: Portable bundle registry and project binding + +Deliverables: + +- broker schema for bundle versions and bindings +- CLI publish/list/pull/apply flows +- immutable version lineage and audit metadata + +This stream is the direct successor to the existing remote encrypted env portability plan. + +### Stream 3: Manual project-key sharing + +Deliverables: + +- key-share schema and broker routes +- share and revoke flows +- audit for recipient grants and use + +### Stream 4: Rotation and recovery safeguards + +Deliverables: + +- project-key rotation +- recovery package flow +- last-share / last-recovery-path guardrails + +### Stream 5: `.env` compatibility and desktop/backend UX + +Deliverables: + +- consistent apply behavior into `.hack/.env` and configured secret backend +- visibility-first CLI and macOS UX +- explicit backend hints and warnings when apply target differs from publish + +## Open Questions + +- Should export/import use the same portable bundle JSON as broker storage, or a wrapped archive that contains both bundle metadata and selected key shares? +- Do we want a single default environment first, or named environments such as `default`, `staging`, and `prod` in the first release? +- Should org-owned project keys require more than one active admin share before rotation/revocation is allowed? +- How much audit retention is required before remote portability is safe to recommend broadly? + +## Links + +- `docs/env.md` +- `docs/plans/2026-03-06-remote-encrypted-project-env-portability-plan.md` +- `docs/plans/2026-03-06-hack-account-auth-and-provider-ux-design.md` diff --git a/docs/plans/2026-03-13-env-portability-and-secret-management-plan.md b/docs/plans/2026-03-13-env-portability-and-secret-management-plan.md new file mode 100644 index 000000000..fb6bca6c0 --- /dev/null +++ b/docs/plans/2026-03-13-env-portability-and-secret-management-plan.md @@ -0,0 +1,278 @@ +# Env Portability And Secret Management Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Deliver portable project env management as an explicit, auditable product made of immutable encrypted bundles, manual project-key sharing, safe rotation and recovery, and preserved local `.env` compatibility. + +**Architecture:** Keep the current local env contract and local materialization model intact, then add an owner-scoped broker registry for immutable portable bundles plus separate key-share records. Use bundle keys for per-version encryption, a project key for access control and recovery, and visibility-first CLI/macOS UX so users can always tell contract, local materialization, and portable state apart. + +**Tech Stack:** Bun, TypeScript, Hack CLI `env` commands, auth-broker, Better Auth ownership, Drizzle/Postgres, macOS SwiftUI follow-on UX, existing secret backend abstractions. + +--- + +### Task 1: Lock current-state storage and trust-model UX + +**Files:** + +- Modify: `docs/env.md` +- Modify: `src/commands/env.ts` +- Modify: `src/lib/hack-env.ts` +- Test: `tests/env-command.test.ts` or the nearest existing env command test file after inspection + +**Step 1: Write the failing test** + +Add env command coverage for an explain/status surface that distinguishes: + +- committed contract location +- local plaintext value location +- local secret backend location +- portable-state availability + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/env-command.test.ts` Expected: FAIL because the env command output does not yet expose the full trust model. + +**Step 3: Write minimal implementation** + +Add a visibility-first env status/explain output so operators can see where each value lives and what trust boundary applies before any remote portability is enabled. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/env-command.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add docs/env.md src/commands/env.ts src/lib/hack-env.ts tests/env-command.test.ts +git commit -m "feat: explain env storage and trust boundaries" +``` + +### Task 2: Add portable bundle and binding schema + +**Files:** + +- Modify: `packages/db/src/schema/core.ts` +- Modify: `services/auth-broker/src/db/schema.ts` +- Create: `services/auth-broker/src/modules/project-envs/service.ts` +- Test: `services/auth-broker/tests/index.test.ts` +- Docs: `docs/plans/2026-03-06-remote-encrypted-project-env-portability-plan.md` + +**Step 1: Write the failing test** + +Add broker storage tests for: + +- bundle create/list/read by owner scope +- immutable version supersede behavior +- project binding create/update/delete +- cross-owner access rejection + +**Step 2: Run test to verify it fails** + +Run: `bun test --cwd services/auth-broker tests/index.test.ts` Expected: FAIL with missing schema or storage behavior. + +**Step 3: Write minimal implementation** + +Implement the portable bundle registry so one project binding can own a sequence of immutable encrypted bundle versions. + +**Step 4: Run test to verify it passes** + +Run: `bun test --cwd services/auth-broker tests/index.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add packages/db/src/schema/core.ts services/auth-broker/src/db/schema.ts services/auth-broker/src/modules/project-envs/service.ts services/auth-broker/tests/index.test.ts docs/plans/2026-03-06-remote-encrypted-project-env-portability-plan.md +git commit -m "feat: add portable env bundle storage" +``` + +### Task 3: Add broker and CLI publish or pull or apply flows + +**Files:** + +- Create: `services/auth-broker/src/modules/project-envs/plugin.ts` +- Modify: `services/auth-broker/src/app.ts` +- Modify: `services/auth-broker/src/modules/better-auth/session.ts` +- Modify: `src/commands/env.ts` +- Modify: `src/mcp/server.ts` +- Test: `services/auth-broker/tests/index.test.ts` +- Test: `tests/env-command.test.ts` +- Test: `tests/mcp.test.ts` + +**Step 1: Write the failing test** + +Add coverage for: + +- `hack env remote publish` +- `hack env remote list` +- `hack env remote pull` +- `hack env remote apply` +- `hack env remote bind` +- signed-out and cross-owner rejection cases + +**Step 2: Run test to verify it fails** + +Run: `bun test --cwd services/auth-broker tests/index.test.ts && bun test tests/env-command.test.ts tests/mcp.test.ts` Expected: FAIL with route-not-found or unknown-command errors. + +**Step 3: Write minimal implementation** + +Implement explicit operator-driven flows for publish, fetch, bind, and apply. Do not add background sync or remote-node fan-out in this slice. + +**Step 4: Run test to verify it passes** + +Run: `bun test --cwd services/auth-broker tests/index.test.ts && bun test tests/env-command.test.ts tests/mcp.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add services/auth-broker/src/modules/project-envs/plugin.ts services/auth-broker/src/app.ts services/auth-broker/src/modules/better-auth/session.ts src/commands/env.ts src/mcp/server.ts services/auth-broker/tests/index.test.ts tests/env-command.test.ts tests/mcp.test.ts +git commit -m "feat: add portable env publish and apply flows" +``` + +### Task 4: Add manual project-key sharing + +**Files:** + +- Modify: `packages/db/src/schema/core.ts` +- Modify: `services/auth-broker/src/db/schema.ts` +- Modify: `services/auth-broker/src/modules/project-envs/service.ts` +- Modify: `services/auth-broker/src/modules/project-envs/plugin.ts` +- Modify: `src/commands/env.ts` +- Test: `services/auth-broker/tests/index.test.ts` +- Test: `tests/env-command.test.ts` + +**Step 1: Write the failing test** + +Add coverage for: + +- create key share for one recipient +- list active shares +- revoke share +- block access after revocation +- reject implicit org-wide sharing when no explicit share exists + +**Step 2: Run test to verify it fails** + +Run: `bun test --cwd services/auth-broker tests/index.test.ts && bun test tests/env-command.test.ts` Expected: FAIL because project-key share records do not exist yet. + +**Step 3: Write minimal implementation** + +Implement explicit key-share records and CLI flows such as: + +- `hack env remote share` +- `hack env remote shares` +- `hack env remote revoke-share` + +Keep the unit of sharing as the project key, not plaintext values. + +**Step 4: Run test to verify it passes** + +Run: `bun test --cwd services/auth-broker tests/index.test.ts && bun test tests/env-command.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add packages/db/src/schema/core.ts services/auth-broker/src/db/schema.ts services/auth-broker/src/modules/project-envs/service.ts services/auth-broker/src/modules/project-envs/plugin.ts src/commands/env.ts services/auth-broker/tests/index.test.ts tests/env-command.test.ts +git commit -m "feat: add manual project key sharing" +``` + +### Task 5: Add rotation and recovery safeguards + +**Files:** + +- Modify: `services/auth-broker/src/modules/project-envs/service.ts` +- Modify: `services/auth-broker/src/modules/project-envs/plugin.ts` +- Modify: `src/commands/env.ts` +- Modify: `src/lib/secret-store.ts` +- Test: `services/auth-broker/tests/index.test.ts` +- Test: `tests/env-command.test.ts` + +**Step 1: Write the failing test** + +Add coverage for: + +- project-key rotation +- recovery package export metadata +- preventing deletion of the last recovery path +- preventing revocation of the last owner share without force + +**Step 2: Run test to verify it fails** + +Run: `bun test --cwd services/auth-broker tests/index.test.ts && bun test tests/env-command.test.ts` Expected: FAIL because rotation and recovery guardrails are not yet implemented. + +**Step 3: Write minimal implementation** + +Implement separate commands for value rotation, share changes, and project-key rotation, plus explicit recovery-path validation before destructive actions are allowed. + +**Step 4: Run test to verify it passes** + +Run: `bun test --cwd services/auth-broker tests/index.test.ts && bun test tests/env-command.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add services/auth-broker/src/modules/project-envs/service.ts services/auth-broker/src/modules/project-envs/plugin.ts src/commands/env.ts src/lib/secret-store.ts services/auth-broker/tests/index.test.ts tests/env-command.test.ts +git commit -m "feat: add env key rotation and recovery safeguards" +``` + +### Task 6: Add `.env` compatibility and visibility-first desktop UX + +**Files:** + +- Modify: `src/commands/env.ts` +- Modify: `apps/macos/Packages/Features/DashboardFeature/Sources/DashboardFeature/ProjectDetailView.swift` +- Modify: `apps/macos/Packages/Features/DashboardFeature/Sources/DashboardFeature/SettingsOverlayView.swift` +- Modify: `apps/macos/Packages/Services/HackCLI/Sources/HackCLIService/HackCLIClient.swift` +- Modify: `apps/macos/Packages/Shared/Models/Sources/HackDesktopModels/Models.swift` +- Test: `tests/env-command.test.ts` +- Test: `apps/macos/Packages/Shared/Models/Tests/HackDesktopModelsTests/` + +**Step 1: Write the failing test** + +Add coverage for: + +- apply writing `plain_env` entries into `.hack/.env` +- apply writing secret entries into the configured backend +- warnings when local backend differs from publish-time hint +- desktop visibility of contract, local materialization, and portable state + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/env-command.test.ts && swift test --package-path apps/macos` Expected: FAIL with missing compatibility or model coverage. + +**Step 3: Write minimal implementation** + +Implement compatibility-first apply logic and visibility-first UI. Do not add inline plaintext editing in desktop for remote bundles. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/env-command.test.ts && swift test --package-path apps/macos` Expected: PASS + +**Step 5: Commit** + +```bash +git add src/commands/env.ts apps/macos/Packages/Features/DashboardFeature/Sources/DashboardFeature/ProjectDetailView.swift apps/macos/Packages/Features/DashboardFeature/Sources/DashboardFeature/SettingsOverlayView.swift apps/macos/Packages/Services/HackCLI/Sources/HackCLIService/HackCLIClient.swift apps/macos/Packages/Shared/Models/Sources/HackDesktopModels/Models.swift tests/env-command.test.ts apps/macos/Packages/Shared/Models/Tests/HackDesktopModelsTests +git commit -m "feat: preserve env compatibility in portable env UX" +``` + +## Recommended Execution Order + +1. Ship Task 1 first so the current trust model is understandable before remote state expands. +2. Reuse the existing remote env portability work as the starting point for Tasks 2 and 3. +3. Keep key sharing separate from base bundle publication so the access model stays reviewable. +4. Land rotation and recovery only after the share model is in place. +5. Add desktop UX after the CLI and broker contracts stabilize. + +## Follow-On Ticket Titles + +- Clarify env storage and trust boundaries in CLI and docs +- Add portable env bundle registry and project binding +- Add portable env publish or pull or apply command flow +- Add manual project-key sharing and revocation +- Add env key rotation and recovery safeguards +- Add compatibility-first portable env desktop UX + +## Links + +- `docs/plans/2026-03-13-env-portability-and-secret-management-design.md` +- `docs/plans/2026-03-06-remote-encrypted-project-env-portability-plan.md` +- `docs/env.md` diff --git a/src/commands/env.ts b/src/commands/env.ts index 844d0f8da..03c217f53 100644 --- a/src/commands/env.ts +++ b/src/commands/env.ts @@ -12,6 +12,7 @@ import { optJson, optPath, optProject } from "../cli/options.ts"; import { PROJECT_ENV_FILENAME } from "../constants.ts"; import { readControlPlaneConfig } from "../control-plane/sdk/config.ts"; import { updateGlobalConfig } from "../lib/config.ts"; +import type { HackEnvStorageSummary } from "../lib/hack-env.ts"; import { removeDotEnvKey, resolveHackEnv, @@ -242,6 +243,7 @@ const handleEnvList: CommandHandlerFor = async ({ `${JSON.stringify( { project: projectName, + storage: serializeEnvStorageForJson({ storage: resolved.storage }), vars: resolved.values.map((v) => ({ key: v.key, required: v.required, @@ -269,6 +271,30 @@ const handleEnvList: CommandHandlerFor = async ({ return 0; } + await display.kv({ + title: "Env storage", + entries: [ + [ + "contract", + `${resolved.storage.contract.path} (committed contract, no values)`, + ], + [ + "local_plaintext", + `${resolved.storage.localPlaintext.path} (${resolved.storage.localPlaintext.exists ? "present" : "missing"} local plaintext for plain_env)`, + ], + [ + "local_secrets", + `${formatSecretStoreDescriptor({ descriptor: resolved.storage.localSecrets })} (local secret backend)`, + ], + [ + "portable_state", + `${resolved.storage.portableState.status}: ${resolved.storage.portableState.message}`, + ], + ], + }); + + await display.section("Resolved env vars"); + for (const v of resolved.values) { const value = v.source === "keychain" && !showSecrets && v.value !== null @@ -292,6 +318,34 @@ const handleEnvList: CommandHandlerFor = async ({ return 0; }; +function serializeEnvStorageForJson(input: { + readonly storage: HackEnvStorageSummary; +}) { + return { + contract: { + path: input.storage.contract.path, + trust_model: input.storage.contract.trustModel, + }, + local_plaintext: { + path: input.storage.localPlaintext.path, + exists: input.storage.localPlaintext.exists, + trust_model: input.storage.localPlaintext.trustModel, + }, + local_secrets: { + backend: input.storage.localSecrets.backend, + location: input.storage.localSecrets.location, + mode: input.storage.localSecrets.mode, + provider: input.storage.localSecrets.provider ?? null, + trust_model: input.storage.localSecrets.trustModel, + }, + portable_state: { + status: input.storage.portableState.status, + trust_model: input.storage.portableState.trustModel, + message: input.storage.portableState.message, + }, + }; +} + const handleEnvSet: CommandHandlerFor = async ({ ctx, args, diff --git a/src/lib/hack-env.ts b/src/lib/hack-env.ts index 7f301c030..c157b12c0 100644 --- a/src/lib/hack-env.ts +++ b/src/lib/hack-env.ts @@ -7,6 +7,7 @@ import { import { parseDotEnv } from "./env.ts"; import { readTextFile, writeTextFileIfChanged } from "./fs.ts"; import { getString, isRecord } from "./guards.ts"; +import type { SecretStoreDescriptor } from "./secret-store.ts"; import { resolveSecretStore } from "./secret-store.ts"; export const HACK_ENV_VERSION = 1 as const; @@ -47,12 +48,37 @@ export type HackEnvResolveResult = { readonly contractPath: string; readonly contractExists: boolean; readonly contractParseError?: string; + readonly envPath: string; + readonly envExists: boolean; readonly contract: HackEnvContract; + readonly storage: HackEnvStorageSummary; readonly values: readonly HackEnvValueState[]; readonly missingRequired: readonly HackEnvValueState[]; readonly envForCompose: Readonly>; }; +export type HackEnvPortableStateSummary = { + readonly status: "not_configured"; + readonly trustModel: "local_only"; + readonly message: string; +}; + +export type HackEnvStorageSummary = { + readonly contract: { + readonly path: string; + readonly trustModel: "committed_no_values"; + }; + readonly localPlaintext: { + readonly path: string; + readonly exists: boolean; + readonly trustModel: "gitignored_plaintext"; + }; + readonly localSecrets: SecretStoreDescriptor & { + readonly trustModel: "local_secret_backend"; + }; + readonly portableState: HackEnvPortableStateSummary; +}; + export async function readHackEnvContract(opts: { readonly projectDir: string; }): Promise { @@ -101,6 +127,7 @@ export async function resolveHackEnv(opts: { const envPath = resolve(opts.projectDir, PROJECT_ENV_FILENAME); const envText = await readTextFile(envPath); + const envExists = envText !== null; const dotenv = envText ? parseDotEnv(envText) : {}; const envForCompose: Record = {}; @@ -170,7 +197,30 @@ export async function resolveHackEnv(opts: { contractPath: read.path, contractExists: read.exists, ...(read.parseError ? { contractParseError: read.parseError } : {}), + envPath, + envExists, contract, + storage: { + contract: { + path: read.path, + trustModel: "committed_no_values", + }, + localPlaintext: { + path: envPath, + exists: envExists, + trustModel: "gitignored_plaintext", + }, + localSecrets: { + ...secretStore.descriptor, + trustModel: "local_secret_backend", + }, + portableState: { + status: "not_configured", + trustModel: "local_only", + message: + "Project env values are not portable across machines by default. Portable encrypted bundles are not configured yet.", + }, + }, values, missingRequired, envForCompose, diff --git a/tests/env-backend-command.test.ts b/tests/env-backend-command.test.ts index bd67629e3..4600b1693 100644 --- a/tests/env-backend-command.test.ts +++ b/tests/env-backend-command.test.ts @@ -258,6 +258,113 @@ test("env set --secret stores value using encrypted backend", async () => { expect(apiKey?.value).toBe("super-secret"); }); +test("env list --json explains storage and trust boundaries", async () => { + if (!tempDir) { + throw new Error("Missing temp dir"); + } + process.env.HACK_SECRETS_FILE_KEY = "env-storage-summary-key"; + const projectRoot = resolve(tempDir, "storage-repo"); + const projectDir = resolve(projectRoot, ".hack"); + const encryptedStorePath = resolve(tempDir, "storage-secrets.enc.json"); + await mkdir(projectDir, { recursive: true }); + await writeFile(resolve(projectDir, "docker-compose.yml"), "services: {}\n"); + await writeFile(resolve(projectDir, ".env"), 'PUBLIC_TOKEN="hello world"\n'); + await writeFile( + resolve(projectDir, "hack.config.json"), + `${JSON.stringify( + { + name: "storage-summary-project", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { + path: encryptedStorePath, + }, + }, + }, + }, + null, + 2 + )}\n` + ); + await writeFile( + resolve(projectDir, "hack.env.json"), + `${JSON.stringify( + { + version: 1, + vars: [ + { + key: "PUBLIC_TOKEN", + required: true, + source: "plain_env", + }, + { + key: "DATABASE_URL", + required: true, + source: "keychain", + }, + ], + }, + null, + 2 + )}\n` + ); + + const result = await runHack({ + args: ["env", "list", "--json"], + env: { + ...process.env, + HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", + HACK_SECRETS_FILE_KEY: "env-storage-summary-key", + }, + cwd: projectRoot, + }); + + expect(result.exitCode).toBe(1); + + const json = JSON.parse(result.stdout) as { + readonly storage?: { + readonly contract?: { + readonly path: string; + readonly trust_model: string; + }; + readonly local_plaintext?: { + readonly path: string; + readonly exists: boolean; + readonly trust_model: string; + }; + readonly local_secrets?: { + readonly backend: string; + readonly location: string; + readonly mode: string; + readonly trust_model: string; + }; + readonly portable_state?: { + readonly status: string; + readonly trust_model: string; + readonly message: string; + }; + }; + }; + + expect(json.storage?.contract?.path).toBe( + resolve(projectDir, "hack.env.json") + ); + expect(json.storage?.contract?.trust_model).toBe("committed_no_values"); + expect(json.storage?.local_plaintext?.path).toBe(resolve(projectDir, ".env")); + expect(json.storage?.local_plaintext?.exists).toBe(true); + expect(json.storage?.local_plaintext?.trust_model).toBe( + "gitignored_plaintext" + ); + expect(json.storage?.local_secrets?.backend).toBe("encrypted_file"); + expect(json.storage?.local_secrets?.location).toBe(encryptedStorePath); + expect(json.storage?.local_secrets?.mode).toBe("native"); + expect(json.storage?.local_secrets?.trust_model).toBe("local_secret_backend"); + expect(json.storage?.portable_state?.status).toBe("not_configured"); + expect(json.storage?.portable_state?.trust_model).toBe("local_only"); + expect(json.storage?.portable_state?.message).toContain("not portable"); +}); + async function runHack(input: { readonly args: readonly string[]; readonly env: NodeJS.ProcessEnv; From f8192a53cd7f36a94594fe71843c8133c2febfc0 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 22:30:38 +0000 Subject: [PATCH 002/134] docs: define first-class github workflows --- docs/cli.md | 6 + docs/extensions.md | 64 ++++ ...-13-github-first-class-workflows-design.md | 237 ++++++++++++++ ...03-13-github-first-class-workflows-plan.md | 299 ++++++++++++++++++ 4 files changed, 606 insertions(+) create mode 100644 docs/plans/2026-03-13-github-first-class-workflows-design.md create mode 100644 docs/plans/2026-03-13-github-first-class-workflows-plan.md diff --git a/docs/cli.md b/docs/cli.md index d46d6c85c..247f0f85c 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -959,6 +959,12 @@ GitHub profile precedence for `--pr`: 2. Project `controlPlane.routing.overrides.github.profile`. 3. Global `controlPlane.extensions["dance.hack.github"].config.defaultProfile`. +Scope note for `--pr`: + +- `hack dispatch run --pr` currently covers the PR update workflow only: push the branch, then create or update the PR. +- It does not imply broader GitHub workflow support such as review submission, standalone PR comments, merge actions, or repo-admin actions. +- The broader first-class GitHub workflow model lives in the GitHub extension docs and `docs/plans/2026-03-13-github-first-class-workflows-design.md`, and is intentionally sequenced beyond the current `--pr` path. + #### hack dispatch status Usage: `hack dispatch status [--json]` diff --git a/docs/extensions.md b/docs/extensions.md index b4c8b0a81..05103107b 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -273,6 +273,70 @@ Notes: - macOS desktop shows read-only system Git identity (host-level inherited Git/gh context) separately from remote OAuth/App accounts to keep local gateway identity and remote auth routing boundaries explicit. - `auth.hack` and `auth.hack.gy` are routed by global Caddy to the daemon auth listener (`127.0.0.1:7790` by default), so this endpoint can also host future provider callbacks/hooks. +Workflow scope: + +- Current shipped GitHub workflow support is intentionally narrow: + - auth/profile connection and selection + - PR create/update via `hack x github pr-upsert` +- The first-class GitHub workflow model for the current milestone is bounded to: + - review intake and review decision + - PR update and readiness management + - PR-level conversation comments + - PR-adjacent repo handoff actions +- The initial set is outcome-based rather than command-based so CLI, desktop, and agent surfaces can share the same product boundary. +- Until those additional flows land, this section should not be read as claiming those commands already exist. + +Initial supported set for this milestone: + +1. Review intake for a relevant PR +2. Review decision submission with a summary body +3. PR creation/update plus draft-ready state changes +4. Top-level PR comments +5. Read-only repo-context handoff actions tied to a PR + +Initial workflow definitions: + +- Review intake and review decision + - Outcome: identify a PR needing action, inspect summary state, and submit `approve`, `comment`, or `request_changes`. +- PR update and readiness management + - Outcome: create or update a PR, keep title/body/base accurate, and move between draft and ready states. +- Conversation comments + - Outcome: leave top-level PR discussion comments without doing a full review submission. +- PR-adjacent repo handoff actions + - Outcome: move from PR context to local repo/branch context without manually reconstructing refs. + +Explicitly out of the initial GitHub scope: + +- inline diff review comments and thread resolution +- labels, assignees, milestones, and project edits +- merge, merge queue, auto-merge, and rebase controls +- rerunning GitHub Actions jobs +- GitHub Issues, Projects, Discussions, releases, or repo admin surfaces + +Success criteria by workflow class: + +- Review intake and review decision + - A reviewer can identify a PR that needs action and submit `approve`, `comment`, or `request_changes` from Hack. +- PR update and readiness management + - An author can create or correct a PR state, including draft and ready transitions, from Hack. +- Conversation comments + - A user can leave a contextual PR comment without switching to GitHub web. +- PR-adjacent repo handoff actions + - A user can move from PR context to local repo context without manual copy-paste of refs. + +Recommended implementation order: + +1. shared PR/review read models +2. review decision submission +3. PR metadata update plus draft/ready transitions +4. standalone PR comment creation +5. repo-context handoff actions + +Detailed design and sequencing live in: + +- `docs/plans/2026-03-13-github-first-class-workflows-design.md` +- `docs/plans/2026-03-13-github-first-class-workflows-plan.md` + ### Linear extension (`hack x linear`) Linear integration supports multiple auth profiles, project-level binding, and manual sync in both diff --git a/docs/plans/2026-03-13-github-first-class-workflows-design.md b/docs/plans/2026-03-13-github-first-class-workflows-design.md new file mode 100644 index 000000000..7fab9c98f --- /dev/null +++ b/docs/plans/2026-03-13-github-first-class-workflows-design.md @@ -0,0 +1,237 @@ +# GitHub First-Class Workflows Design + +## Context + +Hack already has a GitHub extension, but its current surface is mostly auth/profile plumbing plus a narrow `pr-upsert` write path. That is enough to prove connectivity, but not enough to define what "GitHub support" means as a product capability. Without an explicit workflow set, future work risks growing as disconnected commands and one-off UI affordances. + +This ticket defines the first-class GitHub workflow boundary for the current milestone so implementation can sequence intentionally around user outcomes instead of command accumulation. + +## Goals + +- Define a prioritized, bounded initial GitHub workflow set. +- Make the supported user outcomes explicit for reviews, PR updates, comments, and adjacent repo actions. +- Separate the initial release scope from later expansion so product and engineering can say no clearly. +- Provide enough sequencing guidance that implementation can proceed in phases. + +## Non-Goals + +- Defining every future GitHub action Hack may ever support. +- Matching the full GitHub web UI. +- Covering GitHub Issues, Projects, Discussions, releases, or repo administration in the initial set. +- Designing the final CLI verbs or macOS UI in detail. + +## Approaches Considered + +### 1. Command-inventory definition + +Define scope around concrete commands such as `pr-upsert`, `review-submit`, and `comment-create`. + +Pros: + +- Easy to tie directly to implementation +- Low ambiguity for CLI work + +Cons: + +- Locks product language to current transport details +- Makes cross-surface planning harder for macOS, CLI, and agents + +### 2. Workflow-class definition (recommended) + +Define scope around user jobs such as "review a PR" or "update my PR", then map commands and UI to those jobs. + +Pros: + +- Keeps the capability user-outcome focused +- Gives implementation freedom across CLI, desktop, and agents +- Makes later expansion easier to reason about + +Cons: + +- Requires a little more up-front definition work + +### 3. Broad GitHub workbench + +Treat GitHub as a general remote workbench and include most adjacent actions now. + +Pros: + +- Ambitious and flexible + +Cons: + +- Too large for the current milestone +- Makes "first-class" indistinguishable from "everything" +- Weakens implementation sequencing + +## Recommendation + +Adopt the workflow-class model. The first-class GitHub surface should be defined by a small number of high-value jobs that users can complete end-to-end inside Hack. Commands, desktop affordances, and agent tools should then map onto those workflows. + +## Proposed First-Class Workflow Set + +### Priority 1: Review Intake and Review Decision + +**User outcome:** A user can see which PR needs their attention, inspect the decision context, and submit a review outcome without falling back to GitHub web for the common case. + +**Initial support** + +- List relevant PRs for the current user or current repo context. +- Open a PR summary with: + - title, author, branch/base + - review state + - mergeability/check status summary + - changed file summary + - unresolved conversation/thread counts +- Submit a review decision: + - `approve` + - `comment` + - `request_changes` +- Include one summary review body with the decision. + +**Explicitly out of initial scope** + +- Inline diff comment authoring +- Multi-comment review drafts +- Thread resolution management +- Batch review across many repositories + +**Success criteria** + +- A reviewer can identify a PR requiring action in one entry flow. +- A reviewer can make an approve/comment/request-changes decision from Hack. +- The review action returns the resulting GitHub review state and PR URL/id so downstream tooling can continue. + +### Priority 2: PR Update and Readiness Management + +**User outcome:** An author can create or keep a PR accurate and move it between draft and ready states without using the GitHub web UI. + +**Initial support** + +- Create or update a PR for the current branch/repo context. +- Update title, body, and base branch. +- Transition draft <-> ready for review. +- Read current PR metadata before mutating it. +- Optionally add a single top-level comment when performing a PR update. + +**Explicitly out of initial scope** + +- Label, assignee, milestone, or project edits +- Auto-generated release notes +- Merge queue controls +- Auto-rebase, auto-merge, or branch protection bypass + +**Success criteria** + +- A branch with an existing or missing PR can be brought to a correct GitHub PR state from Hack. +- An author can mark a PR ready or return it to draft. +- The update flow is idempotent enough to support agent-driven use. + +### Priority 3: Conversation Comments + +**User outcome:** A user can leave PR-level discussion comments when they do not need a full review submission. + +**Initial support** + +- Create a top-level PR comment. +- Reply to an existing discussion target only if GitHub exposes a clear non-inline comment target already represented in Hack state. +- Show enough surrounding PR metadata so comments are not sent blind. + +**Explicitly out of initial scope** + +- Full threaded review comment authoring on diffs +- Comment editing and deletion +- Emoji reactions +- Issue-wide/general GitHub social surface + +**Success criteria** + +- A user can post a contextual PR comment without switching tools. +- Comment creation can be attached to either standalone discussion or a PR update flow. +- The result includes a durable remote identifier for later correlation. + +### Priority 4: Adjacent Repo Actions + +**User outcome:** A user can move cleanly between GitHub PR context and local repo work without re-entering context manually. + +**Initial support** + +- Resolve the current branch to its PR, or a PR to its repo/branch refs. +- Open or hand off to the local repo/branch context: + - checkout or branch/worktree handoff target + - canonical PR URL + - head/base refs + - changed file list summary +- Show CI/check status summary as read-only context when acting on a PR. + +**Explicitly out of initial scope** + +- Merging a PR +- Rebasing or updating branches on GitHub +- Rerunning GitHub Actions jobs +- Repo settings, secrets, releases, tags, or admin actions + +**Success criteria** + +- A user can pivot from PR context to local repo context with no manual copy-paste of branch/ref data. +- Review and PR update workflows can reuse the same repo-context resolution primitives. +- Repo actions remain supportive, not a second unbounded GitHub product surface. + +## Initial Supported Set + +The initial GitHub capability set is explicitly limited to: + +1. Review intake for a relevant PR +2. Review decision submission with a summary body +3. PR creation/update plus draft-ready state changes +4. Top-level PR comments +5. Read-only repo-context handoff actions tied to a PR + +Everything else is out unless it is necessary to make one of those five workflows function end-to-end. + +## Later Expansion + +Later GitHub phases may add: + +- Inline diff comments and thread resolution +- PR metadata enrichment such as labels, assignees, milestones, and linked issues +- Merge and queue actions +- CI control actions such as rerun/retry +- Issue-centric workflows +- Repo maintenance/admin workflows +- Multi-repo inbox and batch review surfaces + +These should remain separate expansion decisions, not implicit additions to the initial milestone. + +## Sequencing Guidance + +Implementation should land in this order: + +1. Shared PR/review read models +2. Review decision submission +3. PR metadata update and draft/ready transitions +4. Standalone comment creation +5. Repo-context handoff affordances + +That order keeps the highest-value workflows intact while forcing a shared read layer before write surfaces multiply. + +## Product Boundary Rules + +- Every new GitHub action must map to one of the workflow classes above or be deferred. +- If an action does not complete a clear user outcome, it is not first-class. +- Repo actions stay subordinate to PR/review workflows unless a separate milestone expands them. +- Initial support favors summary-level actions over fine-grained GitHub UI parity. + +## Acceptance Mapping + +### The initial GitHub capability set is explicit and bounded + +Yes. The initial set is limited to review intake/decision, PR updates, PR comments, and PR-adjacent repo handoff actions. + +### Each workflow has a clear user outcome + +Yes. Each workflow class above includes a single primary outcome and explicit success criteria. + +### The scope is concrete enough to drive implementation sequencing + +Yes. The workflow set is prioritized, split into initial vs later, and ordered for phased implementation. diff --git a/docs/plans/2026-03-13-github-first-class-workflows-plan.md b/docs/plans/2026-03-13-github-first-class-workflows-plan.md new file mode 100644 index 000000000..18b337dd1 --- /dev/null +++ b/docs/plans/2026-03-13-github-first-class-workflows-plan.md @@ -0,0 +1,299 @@ +# GitHub First-Class Workflows Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Implement the initial first-class GitHub workflow set for reviews, PR updates, comments, and PR-adjacent repo actions in a phased, testable way. + +**Architecture:** Build a shared GitHub PR/review domain layer first, then layer workflow-specific command/UI entry points on top. Keep write operations narrowly aligned to the defined workflow classes so the product surface stays bounded. + +**Tech Stack:** Bun, TypeScript, Hack CLI extension framework, Bun test, markdown docs + +--- + +### Task 1: Document the product boundary in public docs + +**Files:** + +- Modify: `docs/extensions.md` +- Modify: `docs/cli.md` +- Test: `docs/plans/2026-03-13-github-first-class-workflows-design.md` + +**Step 1: Write the failing documentation expectation** + +Add a short checklist to the task branch notes describing the public-doc gap: + +- GitHub docs only describe auth/profile management plus `pr-upsert` +- There is no explicit workflow taxonomy for reviews, comments, or repo handoff + +**Step 2: Verify the gap exists** + +Run: `grep -n "GitHub extension" -n docs/extensions.md && grep -n "review" docs/extensions.md` Expected: GitHub auth/pr-upsert docs exist, but workflow taxonomy is missing or incomplete. + +**Step 3: Add minimal public-facing scope language** + +Document: + +- supported initial workflow classes +- explicit non-goals for the first milestone +- pointer from command docs to the workflow model + +**Step 4: Verify the docs render cleanly** + +Run: `git diff --check` Expected: no whitespace or patch-format errors. + +**Step 5: Commit** + +```bash +git add docs/extensions.md docs/cli.md docs/plans/2026-03-13-github-first-class-workflows-design.md +git commit -m "docs: define first-class github workflow scope" +``` + +### Task 2: Add shared PR/review read primitives + +**Files:** + +- Modify: `src/control-plane/extensions/github/client.ts` +- Modify: `src/control-plane/extensions/github/commands.ts` +- Test: `tests/github-client.test.ts` +- Test: `tests/github-commands.test.ts` + +**Step 1: Write the failing tests** + +Add tests covering: + +- fetching a PR summary by repo + number or branch +- listing relevant PRs for a repo/user context +- normalizing review/check/thread summary payloads + +**Step 2: Run targeted tests to verify failure** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts` Expected: FAIL because review/summary read APIs do not exist yet. + +**Step 3: Implement the minimal shared read layer** + +Add client helpers and command plumbing for: + +- PR summary fetch +- relevant PR listing +- review/check/thread summary normalization + +**Step 4: Re-run tests** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts` Expected: PASS for the new read-path coverage. + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/github/client.ts src/control-plane/extensions/github/commands.ts tests/github-client.test.ts tests/github-commands.test.ts +git commit -m "feat(github): add shared pr review read primitives" +``` + +### Task 3: Implement review decision submission + +**Files:** + +- Modify: `src/control-plane/extensions/github/client.ts` +- Modify: `src/control-plane/extensions/github/commands.ts` +- Test: `tests/github-client.test.ts` +- Test: `tests/github-commands.test.ts` +- Modify: `docs/extensions.md` +- Modify: `docs/cli.md` + +**Step 1: Write the failing tests** + +Add tests covering: + +- `approve`, `comment`, and `request_changes` review submission +- required summary body validation +- stable success payload formatting + +**Step 2: Run targeted tests to verify failure** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts` Expected: FAIL because review submission endpoints and command parsing are missing. + +**Step 3: Implement the minimal review submission flow** + +Add: + +- client call for GitHub review submission +- command parsing and output formatting +- docs for the new supported review workflow + +**Step 4: Re-run tests** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts` Expected: PASS for review submission coverage. + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/github/client.ts src/control-plane/extensions/github/commands.ts tests/github-client.test.ts tests/github-commands.test.ts docs/extensions.md docs/cli.md +git commit -m "feat(github): add review decision workflow" +``` + +### Task 4: Expand PR update flow beyond upsert + +**Files:** + +- Modify: `src/control-plane/extensions/github/client.ts` +- Modify: `src/control-plane/extensions/github/commands.ts` +- Test: `tests/github-client.test.ts` +- Test: `tests/github-commands.test.ts` +- Modify: `docs/extensions.md` +- Modify: `docs/cli.md` + +**Step 1: Write the failing tests** + +Add tests covering: + +- PR metadata read-before-write behavior +- title/body/base updates +- draft -> ready and ready -> draft transitions +- optional comment attach-on-update behavior + +**Step 2: Run targeted tests to verify failure** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts` Expected: FAIL because the state-transition surface is incomplete. + +**Step 3: Implement the minimal PR update workflow** + +Extend the current `pr-upsert` path or split it into clearer commands, but keep support limited to: + +- create/update +- draft/ready transitions +- optional top-level comment + +**Step 4: Re-run tests** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts` Expected: PASS for the expanded PR update coverage. + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/github/client.ts src/control-plane/extensions/github/commands.ts tests/github-client.test.ts tests/github-commands.test.ts docs/extensions.md docs/cli.md +git commit -m "feat(github): expand pr update workflow" +``` + +### Task 5: Add standalone PR comment workflow + +**Files:** + +- Modify: `src/control-plane/extensions/github/client.ts` +- Modify: `src/control-plane/extensions/github/commands.ts` +- Test: `tests/github-client.test.ts` +- Test: `tests/github-commands.test.ts` +- Modify: `docs/extensions.md` +- Modify: `docs/cli.md` + +**Step 1: Write the failing tests** + +Add tests covering: + +- creating a top-level PR comment +- validating repo/pr target resolution +- emitting durable remote identifiers in output + +**Step 2: Run targeted tests to verify failure** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts` Expected: FAIL because standalone comment workflow is not yet exposed. + +**Step 3: Implement the minimal standalone comment flow** + +Support only top-level PR comments in this task. Do not add full inline diff-thread comment support. + +**Step 4: Re-run tests** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts` Expected: PASS for standalone comment coverage. + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/github/client.ts src/control-plane/extensions/github/commands.ts tests/github-client.test.ts tests/github-commands.test.ts docs/extensions.md docs/cli.md +git commit -m "feat(github): add standalone pr comments" +``` + +### Task 6: Add PR-adjacent repo handoff actions + +**Files:** + +- Modify: `src/control-plane/extensions/github/client.ts` +- Modify: `src/control-plane/extensions/github/commands.ts` +- Modify: `src/lib/project.ts` +- Test: `tests/github-client.test.ts` +- Test: `tests/github-commands.test.ts` +- Test: `tests/project-execution-routing.test.ts` +- Modify: `docs/extensions.md` +- Modify: `docs/cli.md` + +**Step 1: Write the failing tests** + +Add tests covering: + +- resolving PR head/base refs from GitHub state +- mapping a PR to a local branch/worktree handoff target +- presenting changed-file and checks summary as read-only repo context + +**Step 2: Run targeted tests to verify failure** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts tests/project-execution-routing.test.ts` Expected: FAIL because PR-to-local handoff actions are not implemented. + +**Step 3: Implement the minimal repo handoff flow** + +Support: + +- PR/ref resolution +- local branch/worktree handoff target generation +- PR URL and changed-file/check summary output + +Do not add merge, rebase, rerun, or admin actions. + +**Step 4: Re-run tests** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts tests/project-execution-routing.test.ts` Expected: PASS for the handoff coverage. + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/github/client.ts src/control-plane/extensions/github/commands.ts src/lib/project.ts tests/github-client.test.ts tests/github-commands.test.ts tests/project-execution-routing.test.ts docs/extensions.md docs/cli.md +git commit -m "feat(github): add pr repo handoff actions" +``` + +### Task 7: Run full quality gates for the GitHub workflow milestone + +**Files:** + +- Modify: `docs/extensions.md` +- Modify: `docs/cli.md` +- Modify: `src/control-plane/extensions/github/client.ts` +- Modify: `src/control-plane/extensions/github/commands.ts` +- Modify: `tests/github-client.test.ts` +- Modify: `tests/github-commands.test.ts` + +**Step 1: Run formatter/linter** + +Run: `bun x ultracite fix` Expected: formatting and autofix pass cleanly. + +**Step 2: Run focused tests** + +Run: `bun test tests/github-client.test.ts tests/github-commands.test.ts tests/project-execution-routing.test.ts` Expected: PASS. + +**Step 3: Run broader repository validation** + +Run: `bun x ultracite check` Expected: PASS with no new lint or formatting regressions. + +**Step 4: Review scope against the design doc** + +Verify each implemented action still maps to: + +- review intake/decision +- PR update +- PR comments +- PR-adjacent repo handoff + +Expected: no extra GitHub surface added beyond the design boundary. + +**Step 5: Commit** + +```bash +git add docs/extensions.md docs/cli.md src/control-plane/extensions/github/client.ts src/control-plane/extensions/github/commands.ts tests/github-client.test.ts tests/github-commands.test.ts tests/project-execution-routing.test.ts +git commit -m "feat(github): deliver first-class workflow milestone" +``` From e22be091523d36ffea35ef3423504ce4e5bd99f0 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 22:34:11 +0000 Subject: [PATCH 003/134] fix: explain env storage without contract --- src/commands/env.ts | 14 +- tests/env-backend-command.test.ts | 513 +++++++++++++++++------------- 2 files changed, 304 insertions(+), 223 deletions(-) diff --git a/src/commands/env.ts b/src/commands/env.ts index 03c217f53..5d031d3c9 100644 --- a/src/commands/env.ts +++ b/src/commands/env.ts @@ -264,13 +264,6 @@ const handleEnvList: CommandHandlerFor = async ({ return resolved.missingRequired.length > 0 ? 1 : 0; } - if (resolved.contract.vars.length === 0) { - logger.info({ - message: `No ${project.projectDir}/hack.env.json contract found (or it has no vars).`, - }); - return 0; - } - await display.kv({ title: "Env storage", entries: [ @@ -293,6 +286,13 @@ const handleEnvList: CommandHandlerFor = async ({ ], }); + if (resolved.contract.vars.length === 0) { + logger.info({ + message: `No ${project.projectDir}/hack.env.json contract found (or it has no vars).`, + }); + return 0; + } + await display.section("Resolved env vars"); for (const v of resolved.values) { diff --git a/tests/env-backend-command.test.ts b/tests/env-backend-command.test.ts index 4600b1693..8e09194b5 100644 --- a/tests/env-backend-command.test.ts +++ b/tests/env-backend-command.test.ts @@ -130,240 +130,321 @@ test("env backend use encrypted_file can provision a stable key file", async () expect(configText).toContain(keyPath); }); -test("env backend use cloud requires provider and stores cloud settings", async () => { - const invalid = await runHack({ - args: ["env", "backend", "use", "cloud"], - env: { - ...process.env, - HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", - }, - }); - expect(invalid.exitCode).toBe(1); - expect(`${invalid.stdout}\n${invalid.stderr}`).toContain( - "Cloud backend requires --provider" - ); +test( + "env backend use cloud requires provider and stores cloud settings", + { timeout: 10_000 }, + async () => { + const invalid = await runHack({ + args: ["env", "backend", "use", "cloud"], + env: { + ...process.env, + HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", + }, + }); + expect(invalid.exitCode).toBe(1); + expect(`${invalid.stdout}\n${invalid.stderr}`).toContain( + "Cloud backend requires --provider" + ); - const valid = await runHack({ - args: [ - "env", - "backend", - "use", - "cloud", - "--provider", - "aws", - "--secret-project", - "dev-account", - "--secret-prefix", - "hack-cli", - "--json", - ], - env: { - ...process.env, - HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", - }, - }); - expect(valid.exitCode).toBe(0); - const json = JSON.parse(valid.stdout) as { - readonly backend: string; - readonly cloud: { - readonly provider?: string; - readonly project?: string; - readonly secretPrefix: string; + const valid = await runHack({ + args: [ + "env", + "backend", + "use", + "cloud", + "--provider", + "aws", + "--secret-project", + "dev-account", + "--secret-prefix", + "hack-cli", + "--json", + ], + env: { + ...process.env, + HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", + }, + }); + expect(valid.exitCode).toBe(0); + const json = JSON.parse(valid.stdout) as { + readonly backend: string; + readonly cloud: { + readonly provider?: string; + readonly project?: string; + readonly secretPrefix: string; + }; }; - }; - expect(json.backend).toBe("cloud"); - expect(json.cloud.provider).toBe("aws"); - expect(json.cloud.project).toBe("dev-account"); - expect(json.cloud.secretPrefix).toBe("hack-cli"); -}); - -test("env set --secret stores value using encrypted backend", async () => { - if (!tempDir) { - throw new Error("Missing temp dir"); + expect(json.backend).toBe("cloud"); + expect(json.cloud.provider).toBe("aws"); + expect(json.cloud.project).toBe("dev-account"); + expect(json.cloud.secretPrefix).toBe("hack-cli"); } - process.env.HACK_SECRETS_FILE_KEY = "env-backend-command-key"; - const projectRoot = resolve(tempDir, "repo"); - const projectDir = resolve(projectRoot, ".hack"); - await mkdir(projectDir, { recursive: true }); - await writeFile(resolve(projectDir, "docker-compose.yml"), "services: {}\n"); - await writeFile( - resolve(projectDir, "hack.config.json"), - `${JSON.stringify( - { - name: "env-backend-project", - controlPlane: { - secrets: { - backend: "encrypted_file", - encryptedFile: { - path: resolve(tempDir, "secrets.enc.json"), +); + +test( + "env set --secret stores value using encrypted backend", + { timeout: 20_000 }, + async () => { + if (!tempDir) { + throw new Error("Missing temp dir"); + } + process.env.HACK_SECRETS_FILE_KEY = "env-backend-command-key"; + const projectRoot = resolve(tempDir, "repo"); + const projectDir = resolve(projectRoot, ".hack"); + await mkdir(projectDir, { recursive: true }); + await writeFile( + resolve(projectDir, "docker-compose.yml"), + "services: {}\n" + ); + await writeFile( + resolve(projectDir, "hack.config.json"), + `${JSON.stringify( + { + name: "env-backend-project", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { + path: resolve(tempDir, "secrets.enc.json"), + }, }, }, }, - }, - null, - 2 - )}\n` - ); - await writeFile( - resolve(projectDir, "hack.env.json"), - `${JSON.stringify( - { - version: 1, - vars: [ - { - key: "API_KEY", - required: false, - source: "keychain", - }, - ], - }, - null, - 2 - )}\n` - ); - - const setResult = await runHack({ - args: ["env", "set", "--secret", "API_KEY=super-secret"], - env: { - ...process.env, - HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", - HACK_SECRETS_FILE_KEY: "env-backend-command-key", - }, - cwd: projectRoot, - }); - expect(setResult.exitCode).toBe(0); - expect(`${setResult.stdout}\n${setResult.stderr}`).toContain( - "encrypted_file" - ); + null, + 2 + )}\n` + ); + await writeFile( + resolve(projectDir, "hack.env.json"), + `${JSON.stringify( + { + version: 1, + vars: [ + { + key: "API_KEY", + required: false, + source: "keychain", + }, + ], + }, + null, + 2 + )}\n` + ); - const listResult = await runHack({ - args: ["env", "list", "--show-secrets", "--json"], - env: { - ...process.env, - HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", - HACK_SECRETS_FILE_KEY: "env-backend-command-key", - }, - cwd: projectRoot, - }); - expect(listResult.exitCode).toBe(0); - const listJson = JSON.parse(listResult.stdout) as { - readonly vars: ReadonlyArray<{ - readonly key: string; - readonly value: string | null; - readonly source: string; - }>; - }; - const apiKey = listJson.vars.find((entry) => entry.key === "API_KEY"); - expect(apiKey?.source).toBe("keychain"); - expect(apiKey?.value).toBe("super-secret"); -}); + const setResult = await runHack({ + args: ["env", "set", "--secret", "API_KEY=super-secret"], + env: { + ...process.env, + HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", + HACK_SECRETS_FILE_KEY: "env-backend-command-key", + }, + cwd: projectRoot, + }); + expect(setResult.exitCode).toBe(0); + expect(`${setResult.stdout}\n${setResult.stderr}`).toContain( + "encrypted_file" + ); -test("env list --json explains storage and trust boundaries", async () => { - if (!tempDir) { - throw new Error("Missing temp dir"); + const listResult = await runHack({ + args: ["env", "list", "--show-secrets", "--json"], + env: { + ...process.env, + HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", + HACK_SECRETS_FILE_KEY: "env-backend-command-key", + }, + cwd: projectRoot, + }); + expect(listResult.exitCode).toBe(0); + const listJson = JSON.parse(listResult.stdout) as { + readonly vars: ReadonlyArray<{ + readonly key: string; + readonly value: string | null; + readonly source: string; + }>; + }; + const apiKey = listJson.vars.find((entry) => entry.key === "API_KEY"); + expect(apiKey?.source).toBe("keychain"); + expect(apiKey?.value).toBe("super-secret"); } - process.env.HACK_SECRETS_FILE_KEY = "env-storage-summary-key"; - const projectRoot = resolve(tempDir, "storage-repo"); - const projectDir = resolve(projectRoot, ".hack"); - const encryptedStorePath = resolve(tempDir, "storage-secrets.enc.json"); - await mkdir(projectDir, { recursive: true }); - await writeFile(resolve(projectDir, "docker-compose.yml"), "services: {}\n"); - await writeFile(resolve(projectDir, ".env"), 'PUBLIC_TOKEN="hello world"\n'); - await writeFile( - resolve(projectDir, "hack.config.json"), - `${JSON.stringify( - { - name: "storage-summary-project", - controlPlane: { - secrets: { - backend: "encrypted_file", - encryptedFile: { - path: encryptedStorePath, +); + +test( + "env list --json explains storage and trust boundaries", + { timeout: 20_000 }, + async () => { + if (!tempDir) { + throw new Error("Missing temp dir"); + } + process.env.HACK_SECRETS_FILE_KEY = "env-storage-summary-key"; + const projectRoot = resolve(tempDir, "storage-repo"); + const projectDir = resolve(projectRoot, ".hack"); + const encryptedStorePath = resolve(tempDir, "storage-secrets.enc.json"); + await mkdir(projectDir, { recursive: true }); + await writeFile( + resolve(projectDir, "docker-compose.yml"), + "services: {}\n" + ); + await writeFile( + resolve(projectDir, ".env"), + 'PUBLIC_TOKEN="hello world"\n' + ); + await writeFile( + resolve(projectDir, "hack.config.json"), + `${JSON.stringify( + { + name: "storage-summary-project", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { + path: encryptedStorePath, + }, }, }, }, - }, - null, - 2 - )}\n` - ); - await writeFile( - resolve(projectDir, "hack.env.json"), - `${JSON.stringify( - { - version: 1, - vars: [ - { - key: "PUBLIC_TOKEN", - required: true, - source: "plain_env", - }, - { - key: "DATABASE_URL", - required: true, - source: "keychain", - }, - ], - }, - null, - 2 - )}\n` - ); + null, + 2 + )}\n` + ); + await writeFile( + resolve(projectDir, "hack.env.json"), + `${JSON.stringify( + { + version: 1, + vars: [ + { + key: "PUBLIC_TOKEN", + required: true, + source: "plain_env", + }, + { + key: "DATABASE_URL", + required: true, + source: "keychain", + }, + ], + }, + null, + 2 + )}\n` + ); - const result = await runHack({ - args: ["env", "list", "--json"], - env: { - ...process.env, - HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", - HACK_SECRETS_FILE_KEY: "env-storage-summary-key", - }, - cwd: projectRoot, - }); + const result = await runHack({ + args: ["env", "list", "--json"], + env: { + ...process.env, + HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", + HACK_SECRETS_FILE_KEY: "env-storage-summary-key", + }, + cwd: projectRoot, + }); - expect(result.exitCode).toBe(1); + expect(result.exitCode).toBe(1); - const json = JSON.parse(result.stdout) as { - readonly storage?: { - readonly contract?: { - readonly path: string; - readonly trust_model: string; - }; - readonly local_plaintext?: { - readonly path: string; - readonly exists: boolean; - readonly trust_model: string; - }; - readonly local_secrets?: { - readonly backend: string; - readonly location: string; - readonly mode: string; - readonly trust_model: string; - }; - readonly portable_state?: { - readonly status: string; - readonly trust_model: string; - readonly message: string; + const json = JSON.parse(result.stdout) as { + readonly storage?: { + readonly contract?: { + readonly path: string; + readonly trust_model: string; + }; + readonly local_plaintext?: { + readonly path: string; + readonly exists: boolean; + readonly trust_model: string; + }; + readonly local_secrets?: { + readonly backend: string; + readonly location: string; + readonly mode: string; + readonly trust_model: string; + }; + readonly portable_state?: { + readonly status: string; + readonly trust_model: string; + readonly message: string; + }; }; }; - }; - expect(json.storage?.contract?.path).toBe( - resolve(projectDir, "hack.env.json") - ); - expect(json.storage?.contract?.trust_model).toBe("committed_no_values"); - expect(json.storage?.local_plaintext?.path).toBe(resolve(projectDir, ".env")); - expect(json.storage?.local_plaintext?.exists).toBe(true); - expect(json.storage?.local_plaintext?.trust_model).toBe( - "gitignored_plaintext" - ); - expect(json.storage?.local_secrets?.backend).toBe("encrypted_file"); - expect(json.storage?.local_secrets?.location).toBe(encryptedStorePath); - expect(json.storage?.local_secrets?.mode).toBe("native"); - expect(json.storage?.local_secrets?.trust_model).toBe("local_secret_backend"); - expect(json.storage?.portable_state?.status).toBe("not_configured"); - expect(json.storage?.portable_state?.trust_model).toBe("local_only"); - expect(json.storage?.portable_state?.message).toContain("not portable"); -}); + expect(json.storage?.contract?.path).toBe( + resolve(projectDir, "hack.env.json") + ); + expect(json.storage?.contract?.trust_model).toBe("committed_no_values"); + expect(json.storage?.local_plaintext?.path).toBe( + resolve(projectDir, ".env") + ); + expect(json.storage?.local_plaintext?.exists).toBe(true); + expect(json.storage?.local_plaintext?.trust_model).toBe( + "gitignored_plaintext" + ); + expect(json.storage?.local_secrets?.backend).toBe("encrypted_file"); + expect(json.storage?.local_secrets?.location).toBe(encryptedStorePath); + expect(json.storage?.local_secrets?.mode).toBe("native"); + expect(json.storage?.local_secrets?.trust_model).toBe( + "local_secret_backend" + ); + expect(json.storage?.portable_state?.status).toBe("not_configured"); + expect(json.storage?.portable_state?.trust_model).toBe("local_only"); + expect(json.storage?.portable_state?.message).toContain("not portable"); + } +); + +test( + "env list shows storage summary even when no contract exists", + { timeout: 20_000 }, + async () => { + if (!tempDir) { + throw new Error("Missing temp dir"); + } + process.env.HACK_SECRETS_FILE_KEY = "env-storage-no-contract-key"; + const projectRoot = resolve(tempDir, "no-contract-repo"); + const projectDir = resolve(projectRoot, ".hack"); + const encryptedStorePath = resolve(tempDir, "no-contract-secrets.enc.json"); + await mkdir(projectDir, { recursive: true }); + await writeFile( + resolve(projectDir, "docker-compose.yml"), + "services: {}\n" + ); + await writeFile( + resolve(projectDir, "hack.config.json"), + `${JSON.stringify( + { + name: "no-contract-project", + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { + path: encryptedStorePath, + }, + }, + }, + }, + null, + 2 + )}\n` + ); + + const result = await runHack({ + args: ["env", "list"], + env: { + ...process.env, + HACK_GLOBAL_CONFIG_PATH: tempGlobalConfigPath ?? "", + HACK_LOGGER: "console", + HACK_SECRETS_FILE_KEY: "env-storage-no-contract-key", + }, + cwd: projectRoot, + }); + + expect(result.exitCode).toBe(0); + expect(result.stdout).toContain("Env storage"); + expect(result.stdout).toContain(resolve(projectDir, "hack.env.json")); + expect(result.stdout).toContain(resolve(projectDir, ".env")); + expect(result.stdout).toContain("not_configured"); + expect(result.stderr).toContain("No"); + expect(result.stderr).toContain("hack.env.json contract found"); + } +); async function runHack(input: { readonly args: readonly string[]; From 1f984b2c9edfa028406312d95d00b9e758e13cba Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 22:37:19 +0000 Subject: [PATCH 004/134] docs: add tickets normalization core plan --- ...03-13-tickets-normalization-core-design.md | 426 ++++++++++++++++++ ...6-03-13-tickets-normalization-core-plan.md | 382 ++++++++++++++++ 2 files changed, 808 insertions(+) create mode 100644 docs/plans/2026-03-13-tickets-normalization-core-design.md create mode 100644 docs/plans/2026-03-13-tickets-normalization-core-plan.md diff --git a/docs/plans/2026-03-13-tickets-normalization-core-design.md b/docs/plans/2026-03-13-tickets-normalization-core-design.md new file mode 100644 index 000000000..dfc62c2c6 --- /dev/null +++ b/docs/plans/2026-03-13-tickets-normalization-core-design.md @@ -0,0 +1,426 @@ +# Tickets Normalization Core Design + +## Program Outcome + +This program ticket does not attempt to land the full normalization rewrite in one branch. Its implementation outcome is: + +- a repo-visible design that makes storage direction, sync direction, and normalized entities explicit +- a concrete child-workstream breakdown for storage, provenance, sync, and documents +- hidden-ref ticket records that preserve the work breakdown inside Hack Tickets itself + +The functional implementation should land incrementally through the child tickets rather than as an all-at-once migration under `HACK-431`. + +## Context + +Hack Tickets already has a useful local-first shape: + +- append-only JSONL events under `.hack/tickets/events/` +- git portability via the dedicated tickets ref managed by `src/control-plane/extensions/tickets/tickets-git-channel.ts` +- an in-memory materialized store in `src/control-plane/extensions/tickets/store.ts` +- sync metadata already embedded in ticket records through `external*`, checkpoint, conflict, and review-note events + +That is enough for manual workflows and the current Linear integration, but it leaves a structural gap: + +- the normalized ticket entity is implicit in event reducers instead of being a first-class model +- provenance is spread across ad hoc fields instead of a consistent identity/change-authority model +- materialization is process-local rather than durable +- richer ticket documents are flattened into `body` instead of being tracked as typed projections + +This stream should make Tickets the canonical normalized work model without losing the properties that already matter: + +- local-first operation +- git portability +- hidden-ref sync +- append-only auditability +- compatibility with current ticket commands and integrations + +## Goals + +- Define an explicit normalized ticket entity and provenance model. +- Introduce a durable SQLite projection fed by the append-only journal. +- Keep the journal and hidden-ref sync as the portable source format. +- Make sync direction and conflict semantics explicit and idempotent. +- Add a path for markdown-backed ticket documents and specs as first-class records. + +## Non-Goals + +- Replacing git-backed event storage with SQLite as the transport format. +- Building every downstream integration in this ticket. +- Changing the current CLI UX beyond what is needed to expose the normalized model. +- Introducing rich-text or binary document storage in the first normalization pass. + +## Current State + +Today the source of truth is effectively: + +1. append JSONL events to `.hack/tickets/events/events-YYYY-MM.jsonl` +2. normalize and sync those logs through `tickets-git-channel.ts` +3. read all events into memory and materialize `TicketSummary`, comments, review notes, checkpoints, and conflicts in `store.ts` + +That model is simple and portable, but it conflates three concerns: + +- journal transport +- normalized domain state +- process-local query projection + +The result is that sync and provenance rules exist, but they are encoded as reducer behavior instead of a documented durable model. + +## Approaches Considered + +### 1. Keep JSONL-only and improve reducer structure + +Pros: + +- lowest migration cost +- preserves current portability + +Cons: + +- no durable local projection +- expensive cold reads keep growing with log size +- provenance and document support stay implicit + +### 2. Make SQLite the new primary store and export JSONL from it + +Pros: + +- simpler local querying +- easier relational modeling + +Cons: + +- portability becomes derived instead of native +- hidden-ref sync becomes a secondary export path +- raises trust questions around journal determinism and merge behavior + +### 3. Keep the append-only journal as canonical, add a deterministic SQLite projection, and model normalized entities/provenance explicitly + +Pros: + +- preserves local-first git portability +- gives durable query performance +- makes sync semantics and document support explicit +- allows idempotent rebuild from journal at any time + +Cons: + +- two storage layers to maintain +- needs careful projection versioning and replay semantics + +### Recommendation + +Choose option 3. + +The journal should remain the canonical transport and audit format. SQLite should be a deterministic projection cache plus query layer, never the only copy of portable ticket state. + +## Proposed Model + +### Source-of-truth direction + +The storage direction should be explicit: + +1. Commands and integrations append normalized events to the journal. +2. The journal is the portable canonical record synced through the hidden git ref. +3. SQLite is rebuilt or incrementally updated from journal events. +4. Read paths prefer SQLite once present, with replay fallback if projection state is absent or stale. + +This keeps portability intact while removing the requirement that every read rebuild state from scratch. + +### Layering + +#### 1. Journal layer + +Stored under `.hack/tickets/events/` as append-only JSONL segments. + +Responsibilities: + +- preserve all state transitions as immutable facts +- remain mergeable and git-syncable +- carry enough event metadata for deterministic projection + +Required additions: + +- explicit journal envelope version +- stable event ids for idempotent replay +- causality metadata for external sync events +- typed document events instead of overloading `ticket.updated` body changes + +#### 2. Normalized domain layer + +New first-class model for: + +- ticket identity +- ticket fields +- provenance and authority +- external links and remote mappings +- typed documents +- sync checkpoints and conflicts + +This layer is conceptual and code-level. It should not depend on whether the backing read path is replay or SQLite. + +#### 3. SQLite projection layer + +Stored locally under `.hack/tickets/` as a rebuildable projection database. + +Responsibilities: + +- durable materialized reads +- indexed lookup by ticket id, external identity, status, assignee, project, and updated time +- document lookup and search metadata +- replay bookkeeping for incremental projection + +SQLite is not a sync target. If deleted, it must be recreated from the journal without losing correctness. + +## Normalized Entities + +### Ticket + +Core ticket row with stable Hack identity: + +- `ticket_id` +- `project_id` +- `project_name` +- `status` +- `title` +- `current_document_id` +- `created_at` +- `updated_at` +- `closed_at` + +This should no longer carry every remote-specific field directly. Remote linkage belongs in provenance tables. + +### Ticket provenance + +Explicit provenance should answer: + +- where this ticket originated +- which system owns which field +- which remote objects correspond to this local ticket +- which event last changed a field + +Recommended normalized concepts: + +- `ticket_origins` + - first local source for a ticket such as `hack` or `linear` +- `ticket_remotes` + - one row per provider/profile/remote object link +- `ticket_field_authority` + - per-field authority rule such as `local`, `remote`, `append_only`, `derived`, `review_required` +- `ticket_field_versions` + - last-applied value, source event id, actor, and timestamp for conflict detection/audit + +This replaces the current mixed use of `owner`, `source`, and `external*` fields as the only provenance representation. + +### Ticket documents + +The current `body` field should become a convenience projection over a typed document model. + +Recommended normalized concepts: + +- `ticket_documents` + - immutable markdown snapshots with `document_id`, `ticket_id`, `kind`, `content_sha256`, `content`, `created_at`, `actor` +- `ticket_document_links` + - logical roles such as `description`, `spec`, `notes`, `handoff` + +`kind` should start with: + +- `description` +- `spec` +- `notes` + +This keeps markdown-backed ticket docs portable while allowing richer behavior later. + +### Activity streams + +Retain append-only records for: + +- comments +- review notes +- sync checkpoints +- sync conflicts +- conflict resolutions + +These can remain event-backed and project into dedicated SQLite tables with stable ids. + +## Journal Envelope + +Every event should project from a normalized journal envelope: + +- `eventId` +- `schemaVersion` +- `eventType` +- `ticketId` +- `occurredAt` +- `recordedAt` +- `actor` +- `sourceSystem` +- `sourceOperation` +- `idempotencyKey` +- `causationId` +- `correlationId` +- `payload` + +Notes: + +- `eventId` and `idempotencyKey` must make replay safe. +- `occurredAt` preserves remote source ordering when known. +- `recordedAt` preserves local append time. +- `sourceOperation` distinguishes local edits, remote pulls, webhook applies, and repair flows. + +## SQLite Projection + +### Proposed tables + +- `projection_meta` +- `journal_events` +- `tickets` +- `ticket_origins` +- `ticket_remotes` +- `ticket_field_authority` +- `ticket_field_versions` +- `ticket_documents` +- `ticket_document_links` +- `ticket_comments` +- `ticket_review_notes` +- `ticket_sync_checkpoints` +- `ticket_sync_conflicts` + +### Projection rules + +- Journal replay is strictly append-only by `eventId`. +- `journal_events` stores applied ids and replay metadata. +- Rebuild mode drops and recreates derived tables from journal files. +- Incremental mode applies only unseen events. +- Projection version changes trigger a rebuild. + +### Read-path rules + +- CLI read commands should prefer SQLite snapshots. +- If SQLite is missing or projection metadata is incompatible, rebuild from journal automatically. +- A JSONL-only fallback should remain available for recovery and tests. + +## Sync Semantics + +### Portable sync + +Portable sync remains journal-first: + +- sync pushes the journal through the hidden ref +- no SQLite files are pushed +- peers rebuild their projection locally after fetch or on first read + +### Idempotent external sync + +External sync must treat journal append as the local mutation boundary. + +Rules: + +- remote changes map to deterministic journal events +- repeated delivery of the same remote mutation must reuse the same `idempotencyKey` or remote event identity +- applying the same remote change twice must be a no-op at the projection layer +- outbound sync checkpoints must record the last remote cursor or updated-at observed per remote link + +### Conflict semantics + +Conflicts should become explicit policy instead of inferred behavior. + +Per field, choose one of: + +- `local` +- `remote` +- `append_only` +- `derived` +- `review_required` + +Initial policy: + +- comments: `append_only` +- review notes: `local` +- sync checkpoints: `derived` +- conflict records: `derived` +- title/status/assignee/description: provider-specific, but persisted as explicit authority rules per remote link + +When a field marked `review_required` diverges, the system should: + +1. append a conflict event +2. leave the current materialized value unchanged +3. require an explicit resolution event + +## Markdown-Backed Ticket Documents + +Markdown-backed docs should be first-class without breaking the current CLI shape. + +### Storage model + +- keep documents as UTF-8 markdown in the journal payload and SQLite projection +- expose the current description as the compatibility `body` projection +- allow additional docs like `spec` and `notes` to exist alongside the default description + +### File materialization + +The first pass should not require checked-in document files on the main branch. + +Instead: + +- documents live in the ticket journal and SQLite projection +- optional file export/import can come later as a projection or command + +This preserves hidden-ref portability and avoids creating a second source of truth immediately. + +## Migration Strategy + +### Phase 1 + +- Introduce normalized domain types and journal envelope versioning. +- Keep current JSONL file layout. +- Add SQLite projection schema and rebuild path. +- Backfill current `TicketSummary` reads from the projection adapter. + +### Phase 2 + +- Migrate external sync logic to provenance and authority tables. +- Introduce typed document events and compatibility body projection. + +### Phase 3 + +- Add richer queries and optional document-oriented commands or exports. + +## Child Workstreams + +This program should be decomposed into four child tickets: + +1. `T-00001` Storage: normalized entity schema plus SQLite projection and replay. +2. `T-00002` Provenance: ticket origin, remote identity, field authority, and field version audit model. +3. `T-00003` Sync: idempotent external sync, checkpoint semantics, and conflict policy. +4. `T-00004` Documents: markdown-backed description/spec/notes support and compatibility projection. + +### Execution order + +- Start with `T-00001` because it defines the normalized storage and projection contract. +- Take `T-00002` next because provenance and field authority depend on the storage shape. +- After `T-00001` and `T-00002`, `T-00003` and `T-00004` can proceed independently. + +### Program completion semantics + +`HACK-431` is complete when this design is landed, the implementation plan is landed, and the child tickets exist with the dependencies above. The code changes for normalization itself belong to the child tickets. + +## Risks + +- SQLite can accidentally become the de facto source of truth if write paths bypass the journal. +- Rebuild logic must be deterministic across machines and Bun versions. +- Existing integrations may implicitly rely on ad hoc `external*` fields and need a compatibility layer. +- Document typing can become over-designed if the first pass tries to support too many document kinds. + +## Validation + +The implementation should prove: + +- deleting the SQLite file does not lose state +- replaying the same journal twice is idempotent +- hidden-ref sync transfers all portable ticket state without the SQLite database +- conflicting remote updates create explicit conflict rows/events +- description/spec markdown documents round-trip through journal, projection, and CLI reads + +## Recommended Next Step + +Implement the program as journal-first normalization, not as a database migration away from git-backed tickets. diff --git a/docs/plans/2026-03-13-tickets-normalization-core-plan.md b/docs/plans/2026-03-13-tickets-normalization-core-plan.md new file mode 100644 index 000000000..6ba5a5c8f --- /dev/null +++ b/docs/plans/2026-03-13-tickets-normalization-core-plan.md @@ -0,0 +1,382 @@ +# Tickets Normalization Core Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +## Program Scope + +This plan is the implementation handoff for `HACK-431`, not an instruction to land the entire normalization effort in one branch. The branch-level deliverable for the program ticket is: + +- land this plan and the paired design doc in the repo +- create and link the child tickets in Hack Tickets +- identify the execution order so the normalization work can proceed incrementally + +**Goal:** Define and implement a journal-first normalized ticket model with a durable SQLite projection, explicit provenance semantics, idempotent sync behavior, and markdown-backed ticket documents. + +**Architecture:** Keep `.hack/tickets/events/*.jsonl` and hidden-ref sync as the canonical portable record. Introduce normalized ticket/provenance/document domain types and a rebuildable SQLite projection under `.hack/tickets/`, then migrate current read paths and sync logic onto that model behind compatibility adapters. + +**Tech Stack:** Bun, TypeScript, `bun:sqlite`, current tickets extension modules in `src/control-plane/extensions/tickets/`, current Linear sync code in `src/control-plane/extensions/linear/commands.ts`, Bun tests. + +--- + +## Child Ticket Mapping + +- `T-00001`: Implement normalized tickets storage and SQLite projection +- `T-00002`: Implement normalized ticket provenance and field authority model +- `T-00003`: Implement idempotent ticket sync and explicit conflict semantics +- `T-00004`: Implement markdown-backed ticket documents and spec support + +Recommended execution order: + +- `T-00001` +- `T-00002` +- `T-00003` and `T-00004` in parallel after the first two land + +Program closeout checklist: + +- design doc committed +- implementation plan committed +- `T-00001` through `T-00004` created with dependency edges +- next execution step set to `T-00001` + +--- + +### Task 1: Define the normalized tickets domain module + +**Files:** + +- Create: `src/control-plane/extensions/tickets/domain.ts` +- Modify: `src/control-plane/extensions/tickets/store.ts` +- Modify: `src/control-plane/extensions/tickets/util.ts` +- Test: `tests/tickets-store.test.ts` + +**Step 1: Write the failing test** + +Add assertions in `tests/tickets-store.test.ts` for a normalized domain adapter that can represent: + +- stable ticket identity +- provenance metadata +- typed ticket documents +- explicit authority/conflict state + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/tickets-store.test.ts` Expected: FAIL because the normalized domain module and adapter do not exist yet. + +**Step 3: Write minimal implementation** + +Create `src/control-plane/extensions/tickets/domain.ts` with: + +- journal envelope types +- normalized ticket types +- provenance types +- document types +- compatibility helpers that map the new normalized model to current `TicketSummary` + +Keep `store.ts` using current public return shapes while routing internal state through the new types. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/tickets-store.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/tickets/domain.ts src/control-plane/extensions/tickets/store.ts src/control-plane/extensions/tickets/util.ts tests/tickets-store.test.ts +git commit -m "feat: define normalized tickets domain model" +``` + +### Task 2: Add journal envelope versioning and deterministic replay ids + +**Files:** + +- Modify: `src/control-plane/extensions/tickets/store.ts` +- Modify: `src/control-plane/extensions/tickets/tickets-git-channel.ts` +- Modify: `src/control-plane/extensions/tickets/util.ts` +- Test: `tests/tickets-store.test.ts` +- Test: `tests/tickets-git-channel.test.ts` + +**Step 1: Write the failing test** + +Add tests proving: + +- every written event carries schema version and stable replay identity +- replaying duplicate external events is ignored +- log normalization preserves the new envelope fields + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/tickets-store.test.ts tests/tickets-git-channel.test.ts` Expected: FAIL because existing event writes do not include the full normalized envelope. + +**Step 3: Write minimal implementation** + +Update event creation helpers so every journal event records: + +- `eventId` +- `schemaVersion` +- `occurredAt` +- `recordedAt` +- `sourceSystem` +- `sourceOperation` +- `idempotencyKey` +- `causationId` +- `correlationId` + +Keep backward-compatible replay support for legacy events during migration. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/tickets-store.test.ts tests/tickets-git-channel.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/tickets/store.ts src/control-plane/extensions/tickets/tickets-git-channel.ts src/control-plane/extensions/tickets/util.ts tests/tickets-store.test.ts tests/tickets-git-channel.test.ts +git commit -m "feat: version ticket journal envelopes" +``` + +### Task 3: Implement the SQLite projection + +**Files:** + +- Create: `src/control-plane/extensions/tickets/sqlite-projection.ts` +- Modify: `src/control-plane/extensions/tickets/store.ts` +- Modify: `src/control-plane/extensions/tickets/extension.ts` +- Test: `tests/tickets-store.test.ts` +- Test: `tests/tickets-extension.test.ts` + +**Step 1: Write the failing test** + +Add tests proving: + +- the projection database is created under `.hack/tickets/` +- reads come back after process restart without replaying the full reducer in memory +- deleting the SQLite file triggers a rebuild from journal files + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/tickets-store.test.ts tests/tickets-extension.test.ts` Expected: FAIL because there is no SQLite projection layer yet. + +**Step 3: Write minimal implementation** + +Create `src/control-plane/extensions/tickets/sqlite-projection.ts` using `bun:sqlite` with: + +- projection metadata table +- journal event bookkeeping +- tickets table +- activity tables for comments, review notes, checkpoints, and conflicts + +Update `store.ts` so writes still append to journal first, then project into SQLite. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/tickets-store.test.ts tests/tickets-extension.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/tickets/sqlite-projection.ts src/control-plane/extensions/tickets/store.ts src/control-plane/extensions/tickets/extension.ts tests/tickets-store.test.ts tests/tickets-extension.test.ts +git commit -m "feat: add tickets sqlite projection" +``` + +### Task 4: Normalize provenance and remote identity + +**Files:** + +- Create: `src/control-plane/extensions/tickets/provenance.ts` +- Modify: `src/control-plane/extensions/tickets/store.ts` +- Modify: `src/control-plane/extensions/linear/commands.ts` +- Test: `tests/tickets-store.test.ts` +- Test: `tests/linear-commands.test.ts` + +**Step 1: Write the failing test** + +Add tests that require: + +- one ticket to link to one or more remote provider objects +- per-field authority to be explicit +- sync conflict creation to cite the last known local and remote field versions + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/tickets-store.test.ts tests/linear-commands.test.ts` Expected: FAIL because provenance is still inferred from ad hoc fields. + +**Step 3: Write minimal implementation** + +Create `src/control-plane/extensions/tickets/provenance.ts` with: + +- remote link types +- authority enums +- field version records +- helpers for mapping current `external*` values into the normalized provenance model + +Update Linear sync code to read and write through the new provenance layer instead of relying only on top-level `external*` fields. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/tickets-store.test.ts tests/linear-commands.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/tickets/provenance.ts src/control-plane/extensions/tickets/store.ts src/control-plane/extensions/linear/commands.ts tests/tickets-store.test.ts tests/linear-commands.test.ts +git commit -m "feat: normalize ticket provenance" +``` + +### Task 5: Add typed markdown-backed ticket documents + +**Files:** + +- Create: `src/control-plane/extensions/tickets/documents.ts` +- Modify: `src/control-plane/extensions/tickets/store.ts` +- Modify: `src/control-plane/extensions/tickets/commands.ts` +- Modify: `src/tui/tickets-tui.ts` +- Test: `tests/tickets-store.test.ts` +- Test: `tests/tickets-extension.test.ts` + +**Step 1: Write the failing test** + +Add tests proving: + +- a ticket can have a `description` document and a `spec` document +- the current `body` field remains available as the compatibility projection of the active description document +- document updates append immutable document events instead of mutating prior content + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/tickets-store.test.ts tests/tickets-extension.test.ts` Expected: FAIL because ticket documents are still just a mutable `body` field. + +**Step 3: Write minimal implementation** + +Create `src/control-plane/extensions/tickets/documents.ts` with: + +- document kind enum +- immutable document snapshot types +- compatibility helpers for `body` + +Update commands and TUI reads so they can show the active description now and later expose additional document kinds safely. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/tickets-store.test.ts tests/tickets-extension.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/tickets/documents.ts src/control-plane/extensions/tickets/store.ts src/control-plane/extensions/tickets/commands.ts src/tui/tickets-tui.ts tests/tickets-store.test.ts tests/tickets-extension.test.ts +git commit -m "feat: add markdown-backed ticket documents" +``` + +### Task 6: Make external sync idempotent against the normalized model + +**Files:** + +- Modify: `src/control-plane/extensions/linear/commands.ts` +- Modify: `src/control-plane/extensions/tickets/store.ts` +- Modify: `src/control-plane/extensions/tickets/provenance.ts` +- Test: `tests/linear-commands.test.ts` +- Test: `tests/tickets-store.test.ts` + +**Step 1: Write the failing test** + +Add tests for: + +- duplicate webhook or poll deliveries +- repeated outbound checkpoint updates +- review-required conflicts when authority cannot resolve a field automatically + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/linear-commands.test.ts tests/tickets-store.test.ts` Expected: FAIL because idempotency and authority are not fully normalized yet. + +**Step 3: Write minimal implementation** + +Update Linear sync handlers so remote operations derive deterministic idempotency keys and only append journal events when the remote mutation is genuinely new. Persist checkpoint and conflict semantics against the provenance model. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/linear-commands.test.ts tests/tickets-store.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add src/control-plane/extensions/linear/commands.ts src/control-plane/extensions/tickets/store.ts src/control-plane/extensions/tickets/provenance.ts tests/linear-commands.test.ts tests/tickets-store.test.ts +git commit -m "feat: make ticket sync idempotent" +``` + +### Task 7: Validate hidden-ref portability and rebuild behavior end to end + +**Files:** + +- Modify: `tests/tickets-git-channel.test.ts` +- Modify: `tests/tickets-extension.test.ts` +- Modify: `docs/guides/tickets.md` + +**Step 1: Write the failing test** + +Add coverage that proves: + +- the hidden-ref sync moves journal files only +- peers rebuild local projection state after sync +- deleting local SQLite state does not affect synced ticket history + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/tickets-git-channel.test.ts tests/tickets-extension.test.ts` Expected: FAIL until projection rebuild and docs are aligned. + +**Step 3: Write minimal implementation** + +Adjust tests and docs so the portability contract is explicit: JSONL journal is portable, SQLite is local and rebuildable, and read paths repair projection state automatically. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/tickets-git-channel.test.ts tests/tickets-extension.test.ts` Expected: PASS + +**Step 5: Commit** + +```bash +git add tests/tickets-git-channel.test.ts tests/tickets-extension.test.ts docs/guides/tickets.md +git commit -m "docs: clarify normalized tickets portability" +``` + +### Task 8: Full verification + +**Files:** + +- Verify all touched tickets and Linear files +- Modify: docs if verification reveals contract drift + +**Step 1: Run verification gates** + +Run: + +```bash +bun test tests/tickets-store.test.ts tests/tickets-extension.test.ts tests/tickets-git-channel.test.ts tests/linear-commands.test.ts +bun run check +``` + +Expected: PASS + +**Step 2: Perform manual sanity checks** + +Verify: + +- `hack tickets create/list/show/update/status` still work +- journal files remain under `.hack/tickets/events/` +- projection rebuild happens when the SQLite file is removed +- remote sync behavior remains idempotent for repeated inbound changes +- description/spec documents render correctly through the current CLI compatibility path + +**Step 3: Commit final integrated changes** + +```bash +git add src/control-plane/extensions/tickets/domain.ts src/control-plane/extensions/tickets/sqlite-projection.ts src/control-plane/extensions/tickets/provenance.ts src/control-plane/extensions/tickets/documents.ts src/control-plane/extensions/tickets/store.ts src/control-plane/extensions/tickets/tickets-git-channel.ts src/control-plane/extensions/tickets/commands.ts src/control-plane/extensions/tickets/util.ts src/control-plane/extensions/tickets/extension.ts src/control-plane/extensions/linear/commands.ts src/tui/tickets-tui.ts tests/tickets-store.test.ts tests/tickets-extension.test.ts tests/tickets-git-channel.test.ts tests/linear-commands.test.ts docs/guides/tickets.md +git commit -m "feat: normalize tickets storage and sync" +``` + +Plan complete and saved to `docs/plans/2026-03-13-tickets-normalization-core-plan.md`. Two execution options: + +**1. Subagent-Driven (this session)** - I dispatch fresh subagent per task, review between tasks, fast iteration + +**2. Parallel Session (separate)** - Open new session with executing-plans, batch execution with checkpoints + +**Which approach?** From b192f04a0febd134f0b477d00d6d913ebc23da3e Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 22:37:32 +0000 Subject: [PATCH 005/134] feat(tickets): normalize ticket replay metadata --- .../extensions/tickets/domain.ts | 336 ++++++++++++++++++ src/control-plane/extensions/tickets/store.ts | 309 ++++++++++------ .../extensions/tickets/tickets-git-channel.ts | 33 +- src/control-plane/extensions/tickets/util.ts | 6 + tests/tickets-git-channel.test.ts | 106 +++++- tests/tickets-store.test.ts | 206 +++++++++++ 6 files changed, 892 insertions(+), 104 deletions(-) create mode 100644 src/control-plane/extensions/tickets/domain.ts diff --git a/src/control-plane/extensions/tickets/domain.ts b/src/control-plane/extensions/tickets/domain.ts new file mode 100644 index 000000000..4e11cd67e --- /dev/null +++ b/src/control-plane/extensions/tickets/domain.ts @@ -0,0 +1,336 @@ +import { sha256Hex } from "./util.ts"; + +export type TicketStatus = "open" | "in_progress" | "blocked" | "done"; + +export type TicketSummaryCompatibility = { + readonly ticketId: string; + readonly title: string; + readonly body?: string; + readonly status: TicketStatus; + readonly createdAt: string; + readonly updatedAt: string; + readonly dependsOn: readonly string[]; + readonly blocks: readonly string[]; + readonly owner: string; + readonly source: string; + readonly assignee?: string; + readonly tags: readonly string[]; + readonly externalSystem?: string; + readonly externalId?: string; + readonly externalKey?: string; + readonly externalUrl?: string; + readonly externalProjectId?: string; + readonly externalProjectName?: string; + readonly externalTeamId?: string; + readonly projectId?: string; + readonly projectName?: string; +}; + +export type TicketMetadataValue = + | string + | number + | boolean + | null + | readonly TicketMetadataValue[] + | { readonly [key: string]: TicketMetadataValue }; + +export type TicketSyncCheckpointCompatibility = { + readonly checkpointId: string; + readonly ticketId: string; + readonly provider: string; + readonly profileId?: string; + readonly direction?: string; + readonly remoteCursor?: string; + readonly remoteUpdatedAt?: string; + readonly localUpdatedAt?: string; + readonly actor: string; + readonly createdAt: string; +}; + +export type TicketSyncConflictCompatibility = { + readonly conflictId: string; + readonly ticketId: string; + readonly provider: string; + readonly field: string; + readonly status: "open" | "resolved"; + readonly authority?: string; + readonly summary?: string; + readonly localValue?: TicketMetadataValue; + readonly remoteValue?: TicketMetadataValue; + readonly createdAt: string; + readonly updatedAt: string; + readonly resolution?: "accept_local" | "accept_remote" | "merged" | "ignore"; + readonly resolutionSummary?: string; + readonly resolvedAt?: string; + readonly resolvedBy?: string; +}; + +export type NormalizedTicketIdentity = { + readonly ticketId: string; + readonly projectId?: string; + readonly projectName?: string; +}; + +export type TicketOrigin = { + readonly owner: string; + readonly source: string; + readonly system: string; +}; + +export type TicketRemoteLink = { + readonly provider: string; + readonly remoteId?: string; + readonly remoteKey?: string; + readonly remoteUrl?: string; + readonly projectId?: string; + readonly projectName?: string; + readonly teamId?: string; +}; + +export type TicketDocumentKind = "description" | "spec" | "notes"; +export type TicketDocumentRole = TicketDocumentKind | "handoff"; + +export type TicketDocument = { + readonly documentId: string; + readonly ticketId: string; + readonly kind: TicketDocumentKind; + readonly role: TicketDocumentRole; + readonly content: string; + readonly contentSha256: string; + readonly createdAt: string; + readonly updatedAt: string; +}; + +export type TicketFieldState = { + readonly field: string; + readonly authority: string; + readonly conflictIds: readonly string[]; +}; + +export type NormalizedTicket = { + readonly identity: NormalizedTicketIdentity; + readonly title: string; + readonly status: TicketStatus; + readonly createdAt: string; + readonly updatedAt: string; + readonly dependsOn: readonly string[]; + readonly blocks: readonly string[]; + readonly assignee?: string; + readonly tags: readonly string[]; + readonly provenance: { + readonly origin: TicketOrigin; + readonly remotes: readonly TicketRemoteLink[]; + }; + readonly documents: readonly TicketDocument[]; + readonly fieldStates: readonly TicketFieldState[]; + readonly sync: { + readonly checkpoints: readonly TicketSyncCheckpointCompatibility[]; + readonly conflicts: readonly TicketSyncConflictCompatibility[]; + }; +}; + +export function createNormalizedTicket(input: { + readonly ticket: TicketSummaryCompatibility; + readonly syncCheckpoints?: readonly TicketSyncCheckpointCompatibility[]; + readonly conflicts?: readonly TicketSyncConflictCompatibility[]; +}): NormalizedTicket { + const sourceSystem = inferSourceSystem({ ticket: input.ticket }); + const documents = buildDocuments({ ticket: input.ticket }); + const remotes = buildRemoteLinks({ + ticket: input.ticket, + sourceSystem, + }); + const checkpoints = input.syncCheckpoints ?? []; + const conflicts = input.conflicts ?? []; + + return { + identity: { + ticketId: input.ticket.ticketId, + ...(input.ticket.projectId ? { projectId: input.ticket.projectId } : {}), + ...(input.ticket.projectName + ? { projectName: input.ticket.projectName } + : {}), + }, + title: input.ticket.title, + status: input.ticket.status, + createdAt: input.ticket.createdAt, + updatedAt: input.ticket.updatedAt, + dependsOn: [...input.ticket.dependsOn], + blocks: [...input.ticket.blocks], + ...(input.ticket.assignee ? { assignee: input.ticket.assignee } : {}), + tags: [...input.ticket.tags], + provenance: { + origin: { + owner: input.ticket.owner, + source: input.ticket.source, + system: sourceSystem, + }, + remotes, + }, + documents, + fieldStates: buildFieldStates({ remotes, conflicts }), + sync: { + checkpoints: [...checkpoints], + conflicts: [...conflicts], + }, + }; +} + +export function projectNormalizedTicketSummary(input: { + readonly ticket: NormalizedTicket; +}): TicketSummaryCompatibility { + const description = input.ticket.documents.find( + (document) => document.role === "description" + ); + const primaryRemote = input.ticket.provenance.remotes[0]; + + return { + ticketId: input.ticket.identity.ticketId, + title: input.ticket.title, + ...(description ? { body: description.content } : {}), + status: input.ticket.status, + createdAt: input.ticket.createdAt, + updatedAt: input.ticket.updatedAt, + dependsOn: [...input.ticket.dependsOn], + blocks: [...input.ticket.blocks], + owner: input.ticket.provenance.origin.owner, + source: input.ticket.provenance.origin.source, + ...(input.ticket.assignee ? { assignee: input.ticket.assignee } : {}), + tags: [...input.ticket.tags], + ...(primaryRemote + ? { + externalSystem: primaryRemote.provider, + ...(primaryRemote.remoteId + ? { externalId: primaryRemote.remoteId } + : {}), + ...(primaryRemote.remoteKey + ? { externalKey: primaryRemote.remoteKey } + : {}), + ...(primaryRemote.remoteUrl + ? { externalUrl: primaryRemote.remoteUrl } + : {}), + ...(primaryRemote.projectId + ? { externalProjectId: primaryRemote.projectId } + : {}), + ...(primaryRemote.projectName + ? { externalProjectName: primaryRemote.projectName } + : {}), + ...(primaryRemote.teamId + ? { externalTeamId: primaryRemote.teamId } + : {}), + } + : {}), + ...(input.ticket.identity.projectId + ? { projectId: input.ticket.identity.projectId } + : {}), + ...(input.ticket.identity.projectName + ? { projectName: input.ticket.identity.projectName } + : {}), + }; +} + +function inferSourceSystem(input: { + readonly ticket: TicketSummaryCompatibility; +}): string { + if (input.ticket.externalSystem) { + return input.ticket.externalSystem; + } + if (input.ticket.source !== "hack") { + return input.ticket.source; + } + if (input.ticket.owner !== "hack") { + return input.ticket.owner; + } + return "hack"; +} + +function buildDocuments(input: { + readonly ticket: TicketSummaryCompatibility; +}): TicketDocument[] { + if (!input.ticket.body) { + return []; + } + + const contentSha256 = sha256Hex({ value: input.ticket.body }); + return [ + { + documentId: `${input.ticket.ticketId}:description:${contentSha256.slice(0, 12)}`, + ticketId: input.ticket.ticketId, + kind: "description", + role: "description", + content: input.ticket.body, + contentSha256, + createdAt: input.ticket.createdAt, + updatedAt: input.ticket.updatedAt, + }, + ]; +} + +function buildRemoteLinks(input: { + readonly ticket: TicketSummaryCompatibility; + readonly sourceSystem: string; +}): TicketRemoteLink[] { + if ( + !( + input.ticket.externalSystem || + input.ticket.externalId || + input.ticket.externalKey || + input.ticket.externalUrl || + input.ticket.externalProjectId || + input.ticket.externalProjectName || + input.ticket.externalTeamId + ) + ) { + return []; + } + + return [ + { + provider: input.ticket.externalSystem ?? input.sourceSystem, + ...(input.ticket.externalId ? { remoteId: input.ticket.externalId } : {}), + ...(input.ticket.externalKey + ? { remoteKey: input.ticket.externalKey } + : {}), + ...(input.ticket.externalUrl + ? { remoteUrl: input.ticket.externalUrl } + : {}), + ...(input.ticket.externalProjectId + ? { projectId: input.ticket.externalProjectId } + : {}), + ...(input.ticket.externalProjectName + ? { projectName: input.ticket.externalProjectName } + : {}), + ...(input.ticket.externalTeamId + ? { teamId: input.ticket.externalTeamId } + : {}), + }, + ]; +} + +function buildFieldStates(input: { + readonly remotes: readonly TicketRemoteLink[]; + readonly conflicts: readonly TicketSyncConflictCompatibility[]; +}): TicketFieldState[] { + const defaultAuthority = input.remotes.length > 0 ? "remote" : "local"; + const byField = new Map(); + + for (const field of ["title", "status", "assignee", "description"]) { + byField.set(field, { + field, + authority: defaultAuthority, + conflictIds: [], + }); + } + + for (const conflict of input.conflicts) { + const current = byField.get(conflict.field); + const conflictIds = [...(current?.conflictIds ?? []), conflict.conflictId]; + byField.set(conflict.field, { + field: conflict.field, + authority: conflict.authority ?? current?.authority ?? defaultAuthority, + conflictIds, + }); + } + + return [...byField.values()]; +} diff --git a/src/control-plane/extensions/tickets/store.ts b/src/control-plane/extensions/tickets/store.ts index cdf69ba2c..aa855a129 100644 --- a/src/control-plane/extensions/tickets/store.ts +++ b/src/control-plane/extensions/tickets/store.ts @@ -5,6 +5,10 @@ import { resolve } from "node:path"; import { isRecord } from "../../../lib/guards.ts"; import type { ControlPlaneConfig } from "../../sdk/config.ts"; +import { + createNormalizedTicket, + projectNormalizedTicketSummary, +} from "./domain.ts"; import { createGitTicketsChannel } from "./tickets-git-channel.ts"; import { formatTicketId, @@ -106,9 +110,18 @@ export type TicketSyncConflict = { export type TicketEvent = { readonly eventId: string; + readonly schemaVersion: number; readonly ts: number; readonly tsIso: string; + readonly eventType: string; + readonly occurredAt: string; + readonly recordedAt: string; readonly actor: string; + readonly sourceSystem: string; + readonly sourceOperation: string; + readonly idempotencyKey: string; + readonly causationId?: string; + readonly correlationId?: string; readonly orderKey?: string; readonly projectId?: string; readonly projectName?: string; @@ -157,6 +170,16 @@ type MaterializedTicketState = { readonly conflictsByTicket: Map; }; +function normalizeTicketSummaryCompatibility(input: { + readonly ticket: TicketSummary; +}): TicketSummary { + return projectNormalizedTicketSummary({ + ticket: createNormalizedTicket({ + ticket: input.ticket, + }), + }); +} + export function createTicketsStore(opts: { readonly projectRoot: string; readonly projectId?: string; @@ -316,15 +339,32 @@ export function createTicketsStore(opts: { readonly type: string; readonly payload: Record; readonly actor?: string; + readonly occurredAt?: string; + readonly sourceSystem?: string; + readonly sourceOperation?: string; + readonly idempotencyKey?: string; + readonly causationId?: string; + readonly correlationId?: string; }): TicketEvent => { const ts = unixSeconds(); + const recordedAt = new Date(ts * 1000).toISOString(); + const eventId = randomUUID(); const orderKey = `${Date.now()}-${String(eventSequence).padStart(6, "0")}`; eventSequence += 1; return { - eventId: randomUUID(), + eventId, + schemaVersion: 1, ts, - tsIso: new Date(ts * 1000).toISOString(), + tsIso: recordedAt, + eventType: input.type, + occurredAt: input.occurredAt ?? recordedAt, + recordedAt, actor: resolveActor(input.actor), + sourceSystem: input.sourceSystem ?? "hack", + sourceOperation: input.sourceOperation ?? "local_command", + idempotencyKey: input.idempotencyKey ?? eventId, + ...(input.causationId ? { causationId: input.causationId } : {}), + ...(input.correlationId ? { correlationId: input.correlationId } : {}), orderKey, ...(opts.projectId ? { projectId: opts.projectId } : {}), ...(opts.projectName ? { projectName: opts.projectName } : {}), @@ -346,6 +386,8 @@ export function createTicketsStore(opts: { } const events: TicketEvent[] = []; + const seenEventIds = new Set(); + const seenIdempotencyKeys = new Set(); for (const filename of entries.sort()) { const path = resolve(eventsDir, filename); const text = await Bun.file(path) @@ -358,13 +400,27 @@ export function createTicketsStore(opts: { } const parsed = safeJsonParse(trimmed); const event = parseEvent(parsed); - if (event) { - events.push(event); + if (!event || seenEventIds.has(event.eventId)) { + continue; } + if (seenIdempotencyKeys.has(event.idempotencyKey)) { + continue; + } + seenEventIds.add(event.eventId); + seenIdempotencyKeys.add(event.idempotencyKey); + events.push(event); } } events.sort((a, b) => { + if (a.ticketId === b.ticketId && a.type !== b.type) { + if (a.type === "ticket.created") { + return -1; + } + if (b.type === "ticket.created") { + return 1; + } + } if (a.ts !== b.ts) { return a.ts - b.ts; } @@ -539,29 +595,31 @@ export function createTicketsStore(opts: { return { ok: true, - ticket: { - ticketId, - title: input.title, - ...(input.body ? { body: input.body } : {}), - status: "open", - createdAt: event.tsIso, - updatedAt: event.tsIso, - dependsOn, - blocks, - owner, - source, - ...(assignee ? { assignee } : {}), - tags, - ...(externalSystem ? { externalSystem } : {}), - ...(externalId ? { externalId } : {}), - ...(externalKey ? { externalKey } : {}), - ...(externalUrl ? { externalUrl } : {}), - ...(externalProjectId ? { externalProjectId } : {}), - ...(externalProjectName ? { externalProjectName } : {}), - ...(externalTeamId ? { externalTeamId } : {}), - ...(opts.projectId ? { projectId: opts.projectId } : {}), - ...(opts.projectName ? { projectName: opts.projectName } : {}), - }, + ticket: normalizeTicketSummaryCompatibility({ + ticket: { + ticketId, + title: input.title, + ...(input.body ? { body: input.body } : {}), + status: "open", + createdAt: event.tsIso, + updatedAt: event.tsIso, + dependsOn, + blocks, + owner, + source, + ...(assignee ? { assignee } : {}), + tags, + ...(externalSystem ? { externalSystem } : {}), + ...(externalId ? { externalId } : {}), + ...(externalKey ? { externalKey } : {}), + ...(externalUrl ? { externalUrl } : {}), + ...(externalProjectId ? { externalProjectId } : {}), + ...(externalProjectName ? { externalProjectName } : {}), + ...(externalTeamId ? { externalTeamId } : {}), + ...(opts.projectId ? { projectId: opts.projectId } : {}), + ...(opts.projectName ? { projectName: opts.projectName } : {}), + }, + }), }; }, @@ -1145,31 +1203,36 @@ function applyTicketCreatedEvent(input: { value: input.event.payload.externalTeamId, }); - input.tickets.set(input.event.ticketId, { - ticketId: input.event.ticketId, - title, - body, - status: "open", - createdAt: input.event.tsIso, - updatedAt: input.event.tsIso, - dependsOn, - blocks, - owner, - source, - ...(assignee ? { assignee } : {}), - tags, - ...(externalSystem ? { externalSystem } : {}), - ...(externalId ? { externalId } : {}), - ...(externalKey ? { externalKey } : {}), - ...(externalUrl ? { externalUrl } : {}), - ...(externalProjectId ? { externalProjectId } : {}), - ...(externalProjectName ? { externalProjectName } : {}), - ...(externalTeamId ? { externalTeamId } : {}), - ...(input.event.projectId ? { projectId: input.event.projectId } : {}), - ...(input.event.projectName - ? { projectName: input.event.projectName } - : {}), - }); + input.tickets.set( + input.event.ticketId, + normalizeTicketSummaryCompatibility({ + ticket: { + ticketId: input.event.ticketId, + title, + body, + status: "open", + createdAt: input.event.tsIso, + updatedAt: input.event.tsIso, + dependsOn, + blocks, + owner, + source, + ...(assignee ? { assignee } : {}), + tags, + ...(externalSystem ? { externalSystem } : {}), + ...(externalId ? { externalId } : {}), + ...(externalKey ? { externalKey } : {}), + ...(externalUrl ? { externalUrl } : {}), + ...(externalProjectId ? { externalProjectId } : {}), + ...(externalProjectName ? { externalProjectName } : {}), + ...(externalTeamId ? { externalTeamId } : {}), + ...(input.event.projectId ? { projectId: input.event.projectId } : {}), + ...(input.event.projectName + ? { projectName: input.event.projectName } + : {}), + }, + }) + ); } function applyTicketStatusChangedEvent(input: { @@ -1186,11 +1249,16 @@ function applyTicketStatusChangedEvent(input: { return; } - input.tickets.set(input.event.ticketId, { - ...current, - status, - updatedAt: input.event.tsIso, - }); + input.tickets.set( + input.event.ticketId, + normalizeTicketSummaryCompatibility({ + ticket: { + ...current, + status, + updatedAt: input.event.tsIso, + }, + }) + ); } function applyTicketUpdatedEvent(input: { @@ -1263,47 +1331,56 @@ function applyTicketUpdatedEvent(input: { key: "externalTeamId", }); - input.tickets.set(input.event.ticketId, { - ...current, - ...(title ? { title } : {}), - ...(body !== undefined ? { body } : {}), - ...(dependsOn !== null ? { dependsOn } : {}), - ...(blocks !== null ? { blocks } : {}), - ...(owner !== null - ? { - owner: normalizeOwnerOrSource({ - value: owner, - fallback: current.owner, - }), - } - : {}), - ...(source !== null - ? { - source: normalizeOwnerOrSource({ - value: source, - fallback: current.source, - }), - } - : {}), - ...(assignee !== null ? { assignee: assignee || undefined } : {}), - ...(tags !== null ? { tags } : {}), - ...(externalSystem !== null - ? { externalSystem: externalSystem || undefined } - : {}), - ...(externalId !== null ? { externalId: externalId || undefined } : {}), - ...(externalKey !== null ? { externalKey: externalKey || undefined } : {}), - ...(externalUrl !== null ? { externalUrl: externalUrl || undefined } : {}), - ...(externalProjectId !== null - ? { externalProjectId: externalProjectId || undefined } - : {}), - ...(externalProjectName !== null - ? { externalProjectName: externalProjectName || undefined } - : {}), - ...(externalTeamId !== null - ? { externalTeamId: externalTeamId || undefined } - : {}), - updatedAt: input.event.tsIso, - }); + input.tickets.set( + input.event.ticketId, + normalizeTicketSummaryCompatibility({ + ticket: { + ...current, + ...(title ? { title } : {}), + ...(body !== undefined ? { body } : {}), + ...(dependsOn !== null ? { dependsOn } : {}), + ...(blocks !== null ? { blocks } : {}), + ...(owner !== null + ? { + owner: normalizeOwnerOrSource({ + value: owner, + fallback: current.owner, + }), + } + : {}), + ...(source !== null + ? { + source: normalizeOwnerOrSource({ + value: source, + fallback: current.source, + }), + } + : {}), + ...(assignee !== null ? { assignee: assignee || undefined } : {}), + ...(tags !== null ? { tags } : {}), + ...(externalSystem !== null + ? { externalSystem: externalSystem || undefined } + : {}), + ...(externalId !== null ? { externalId: externalId || undefined } : {}), + ...(externalKey !== null + ? { externalKey: externalKey || undefined } + : {}), + ...(externalUrl !== null + ? { externalUrl: externalUrl || undefined } + : {}), + ...(externalProjectId !== null + ? { externalProjectId: externalProjectId || undefined } + : {}), + ...(externalProjectName !== null + ? { externalProjectName: externalProjectName || undefined } + : {}), + ...(externalTeamId !== null + ? { externalTeamId: externalTeamId || undefined } + : {}), + updatedAt: input.event.tsIso, + }, + }) + ); } function applyTicketCommentAppendedEvent(input: { @@ -1782,12 +1859,35 @@ function parseEvent(value: unknown): TicketEvent | null { return null; } const eventId = typeof value.eventId === "string" ? value.eventId : ""; + const schemaVersion = + typeof value.schemaVersion === "number" ? value.schemaVersion : 0; const ts = typeof value.ts === "number" ? value.ts : Number.NaN; + const tsIso = new Date(ts * 1000).toISOString(); const actor = typeof value.actor === "string" ? value.actor : ""; + const eventTypeCandidate = + typeof value.eventType === "string" ? value.eventType : undefined; + const typeCandidate = typeof value.type === "string" ? value.type : undefined; + const eventType = eventTypeCandidate ?? typeCandidate ?? ""; + const occurredAt = + typeof value.occurredAt === "string" ? value.occurredAt : tsIso; + const recordedAt = + typeof value.recordedAt === "string" ? value.recordedAt : tsIso; const orderKey = typeof value.orderKey === "string" ? value.orderKey : undefined; + const sourceSystem = + typeof value.sourceSystem === "string" ? value.sourceSystem : "hack"; + const sourceOperation = + typeof value.sourceOperation === "string" + ? value.sourceOperation + : eventType; + const idempotencyKey = + typeof value.idempotencyKey === "string" ? value.idempotencyKey : eventId; + const causationId = + typeof value.causationId === "string" ? value.causationId : undefined; + const correlationId = + typeof value.correlationId === "string" ? value.correlationId : undefined; const ticketId = typeof value.ticketId === "string" ? value.ticketId : ""; - const type = typeof value.type === "string" ? value.type : ""; + const type = typeCandidate ?? eventType; const payload = isRecord(value.payload) ? (value.payload as Record) : null; @@ -1805,9 +1905,18 @@ function parseEvent(value: unknown): TicketEvent | null { return { eventId, + schemaVersion, ts, - tsIso: new Date(ts * 1000).toISOString(), + tsIso, + eventType, + occurredAt, + recordedAt, actor, + sourceSystem, + sourceOperation, + idempotencyKey, + ...(causationId ? { causationId } : {}), + ...(correlationId ? { correlationId } : {}), ...(orderKey ? { orderKey } : {}), ...(projectId ? { projectId } : {}), ...(projectName ? { projectName } : {}), diff --git a/src/control-plane/extensions/tickets/tickets-git-channel.ts b/src/control-plane/extensions/tickets/tickets-git-channel.ts index 7517170e2..cff02ff1b 100644 --- a/src/control-plane/extensions/tickets/tickets-git-channel.ts +++ b/src/control-plane/extensions/tickets/tickets-git-channel.ts @@ -631,6 +631,7 @@ export function createGitTicketsChannel(opts: { .text() .catch(() => ""); const existingIds = new Set(); + const existingIdempotencyKeys = new Set(); for (const line of existing.split("\n")) { const trimmed = line.trim(); if (!trimmed) { @@ -639,15 +640,29 @@ export function createGitTicketsChannel(opts: { const parsed = safeJsonParse(trimmed); if (isRecord(parsed) && typeof parsed.eventId === "string") { existingIds.add(parsed.eventId); + const idempotencyKey = + typeof parsed.idempotencyKey === "string" + ? parsed.idempotencyKey + : parsed.eventId; + existingIdempotencyKeys.add(idempotencyKey); } } const lines: string[] = []; + const pendingIdempotencyKeys = new Set(); for (const ev of events) { const id = typeof ev.eventId === "string" ? ev.eventId : ""; - if (!id || existingIds.has(id)) { + const idempotencyKey = + typeof ev.idempotencyKey === "string" ? ev.idempotencyKey : id; + if ( + !id || + existingIds.has(id) || + existingIdempotencyKeys.has(idempotencyKey) || + pendingIdempotencyKeys.has(idempotencyKey) + ) { continue; } + pendingIdempotencyKeys.add(idempotencyKey); lines.push(stableStringify(ev)); } @@ -684,6 +699,7 @@ export function createGitTicketsChannel(opts: { .catch(() => ""); const parsed: Record[] = []; const seen = new Set(); + const seenIdempotencyKeys = new Set(); for (const line of text.split("\n")) { const trimmed = line.trim(); @@ -695,14 +711,19 @@ export function createGitTicketsChannel(opts: { continue; } const eventId = typeof value.eventId === "string" ? value.eventId : ""; + const idempotencyKey = + typeof value.idempotencyKey === "string" + ? value.idempotencyKey + : eventId; const ts = typeof value.ts === "number" ? value.ts : Number.NaN; if (!(eventId && Number.isFinite(ts))) { continue; } - if (seen.has(eventId)) { + if (seen.has(eventId) || seenIdempotencyKeys.has(idempotencyKey)) { continue; } seen.add(eventId); + seenIdempotencyKeys.add(idempotencyKey); parsed.push(value); } @@ -1178,6 +1199,7 @@ function mergeTicketEventLogs(input: { }): string { const parsed: Record[] = []; const seen = new Set(); + const seenIdempotencyKeys = new Set(); for (const text of [input.existing, input.incoming]) { for (const line of text.split("\n")) { @@ -1190,14 +1212,19 @@ function mergeTicketEventLogs(input: { continue; } const eventId = typeof value.eventId === "string" ? value.eventId : ""; + const idempotencyKey = + typeof value.idempotencyKey === "string" + ? value.idempotencyKey + : eventId; const ts = typeof value.ts === "number" ? value.ts : Number.NaN; if (!(eventId && Number.isFinite(ts))) { continue; } - if (seen.has(eventId)) { + if (seen.has(eventId) || seenIdempotencyKeys.has(idempotencyKey)) { continue; } seen.add(eventId); + seenIdempotencyKeys.add(idempotencyKey); parsed.push(value); } } diff --git a/src/control-plane/extensions/tickets/util.ts b/src/control-plane/extensions/tickets/util.ts index 1e4e3e981..baadb87f2 100644 --- a/src/control-plane/extensions/tickets/util.ts +++ b/src/control-plane/extensions/tickets/util.ts @@ -1,3 +1,5 @@ +import { createHash } from "node:crypto"; + const DIGITS_ONLY_PATTERN = /^\d+$/; export function unixSeconds(): number { @@ -75,6 +77,10 @@ export function stableStringify(value: unknown): string { return JSON.stringify(stableSort(value)); } +export function sha256Hex(input: { readonly value: string }): string { + return createHash("sha256").update(input.value).digest("hex"); +} + function stableSort(value: unknown): unknown { if (Array.isArray(value)) { return value.map(stableSort); diff --git a/tests/tickets-git-channel.test.ts b/tests/tickets-git-channel.test.ts index c93f8419f..9b184843b 100644 --- a/tests/tickets-git-channel.test.ts +++ b/tests/tickets-git-channel.test.ts @@ -7,30 +7,62 @@ test("mergeTicketEventLogs dedupes by event id and preserves chronological order existing: [ JSON.stringify({ eventId: "event-2", + schemaVersion: 1, ts: 2, + occurredAt: "2026-03-13T00:00:02.000Z", + recordedAt: "2026-03-13T00:00:02.000Z", + sourceSystem: "hack", + sourceOperation: "ticket.created", + idempotencyKey: "event-2", ticketId: "T-00002", + eventType: "ticket.created", type: "ticket.created", + payload: {}, }), JSON.stringify({ eventId: "event-3", + schemaVersion: 1, ts: 3, + occurredAt: "2026-03-13T00:00:03.000Z", + recordedAt: "2026-03-13T00:00:03.000Z", + sourceSystem: "hack", + sourceOperation: "ticket.created", + idempotencyKey: "event-3", ticketId: "T-00003", + eventType: "ticket.created", type: "ticket.created", + payload: {}, }), "", ].join("\n"), incoming: [ JSON.stringify({ eventId: "event-1", + schemaVersion: 1, ts: 1, + occurredAt: "2026-03-13T00:00:01.000Z", + recordedAt: "2026-03-13T00:00:01.000Z", + sourceSystem: "linear", + sourceOperation: "issue.import", + idempotencyKey: "linear:issue:1", ticketId: "T-00001", + eventType: "ticket.created", type: "ticket.created", + payload: {}, }), JSON.stringify({ eventId: "event-2", + schemaVersion: 1, ts: 2, + occurredAt: "2026-03-13T00:00:02.000Z", + recordedAt: "2026-03-13T00:00:02.000Z", + sourceSystem: "hack", + sourceOperation: "ticket.created", + idempotencyKey: "event-2", ticketId: "T-00002", + eventType: "ticket.created", type: "ticket.created", + payload: {}, }), "", ].join("\n"), @@ -41,7 +73,15 @@ test("mergeTicketEventLogs dedupes by event id and preserves chronological order .split("\n") .map( (line) => - JSON.parse(line) as { readonly eventId: string; readonly ts: number } + JSON.parse(line) as { + readonly eventId: string; + readonly ts: number; + readonly schemaVersion: number; + readonly sourceSystem: string; + readonly sourceOperation: string; + readonly idempotencyKey: string; + readonly eventType: string; + } ); expect(lines.map((line) => line.eventId)).toEqual([ @@ -50,6 +90,70 @@ test("mergeTicketEventLogs dedupes by event id and preserves chronological order "event-3", ]); expect(lines.map((line) => line.ts)).toEqual([1, 2, 3]); + expect(lines[0]).toMatchObject({ + schemaVersion: 1, + sourceSystem: "linear", + sourceOperation: "issue.import", + idempotencyKey: "linear:issue:1", + eventType: "ticket.created", + }); +}); + +test("mergeTicketEventLogs preserves normalized journal envelope fields", () => { + const merged = __testOnly.mergeTicketEventLogs({ + existing: [ + JSON.stringify({ + eventId: "event-2", + schemaVersion: 1, + ts: 2, + occurredAt: "2026-03-13T10:00:02.000Z", + recordedAt: "2026-03-13T10:00:03.000Z", + ticketId: "T-00002", + type: "ticket.created", + sourceSystem: "linear", + sourceOperation: "webhook_pull", + idempotencyKey: "linear:event-2", + }), + "", + ].join("\n"), + incoming: [ + JSON.stringify({ + eventId: "event-1", + schemaVersion: 1, + ts: 1, + occurredAt: "2026-03-13T10:00:01.000Z", + recordedAt: "2026-03-13T10:00:01.500Z", + ticketId: "T-00001", + type: "ticket.created", + sourceSystem: "hack", + sourceOperation: "local_command", + idempotencyKey: "hack:event-1", + }), + "", + ].join("\n"), + }); + + const lines = merged + .trim() + .split("\n") + .map((line) => JSON.parse(line) as Record); + + expect(lines).toEqual([ + expect.objectContaining({ + eventId: "event-1", + schemaVersion: 1, + sourceSystem: "hack", + sourceOperation: "local_command", + idempotencyKey: "hack:event-1", + }), + expect.objectContaining({ + eventId: "event-2", + schemaVersion: 1, + sourceSystem: "linear", + sourceOperation: "webhook_pull", + idempotencyKey: "linear:event-2", + }), + ]); }); test("resolvePushRefForCheckoutRef prefers legacy branch when checkout came from legacy tracking ref", () => { diff --git a/tests/tickets-store.test.ts b/tests/tickets-store.test.ts index 099e38e01..b5f94d7a1 100644 --- a/tests/tickets-store.test.ts +++ b/tests/tickets-store.test.ts @@ -9,6 +9,10 @@ import { } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; +import { + createNormalizedTicket, + projectNormalizedTicketSummary, +} from "../src/control-plane/extensions/tickets/domain.ts"; import { createTicketsStore } from "../src/control-plane/extensions/tickets/store.ts"; import { readControlPlaneConfig } from "../src/control-plane/sdk/config.ts"; @@ -218,6 +222,15 @@ test("tickets store materializes assignee, review notes, comments, checkpoints, "ticket.sync_conflict_recorded", "ticket.sync_conflict_resolved", ]); + expect(events[0]).toMatchObject({ + schemaVersion: 1, + eventType: "ticket.created", + occurredAt: events[0]?.tsIso, + recordedAt: events[0]?.tsIso, + sourceSystem: "hack", + sourceOperation: "local_command", + }); + expect(events[0]?.idempotencyKey).toBe(events[0]?.eventId); }, 20_000); test("tickets show json includes materialized sync metadata", async () => { @@ -340,6 +353,199 @@ test("tickets store recovers from a stale tickets bare repo index.lock", async ( expect(tickets.map((ticket) => ticket.title)).toContain("Stale lock ticket"); }, 20_000); +test("tickets store writes normalized journal envelope metadata and ignores duplicate idempotency keys", async () => { + const projectRoot = await createTempGitProject({ + prefix: "hack-cli-tickets-envelope-", + }); + const store = await createStore({ projectRoot }); + + const created = await store.createTicket({ + title: "Envelope ticket", + owner: "hack", + source: "hack", + actor: "creator@hack", + }); + expect(created.ok).toBe(true); + if (!created.ok) { + throw new Error(created.error); + } + + const eventsDir = join( + projectRoot, + ".hack/tickets/git/worktree/.hack/tickets/events" + ); + const [eventsFile] = (await readdir(eventsDir)).filter((entry) => + entry.endsWith(".jsonl") + ); + expect(eventsFile).toBeString(); + if (!eventsFile) { + throw new Error("Missing tickets event log"); + } + + const eventsPath = join(eventsDir, eventsFile); + const rawLines = (await readFile(eventsPath, "utf8")) + .trim() + .split("\n") + .map((line) => JSON.parse(line) as Record); + + expect(rawLines[0]).toMatchObject({ + schemaVersion: 1, + ticketId: created.ticket.ticketId, + occurredAt: created.ticket.createdAt, + recordedAt: created.ticket.createdAt, + sourceSystem: "hack", + sourceOperation: "local_command", + }); + expect(rawLines[0]?.idempotencyKey).toBe(rawLines[0]?.eventId); + const baseTs = Number(rawLines[0]?.ts ?? 0) + 1; + + const duplicateBaseEvent = { + schemaVersion: 1, + ticketId: created.ticket.ticketId, + type: "ticket.comment_appended", + payload: { + commentId: "comment-1", + body: "Imported from Linear.", + source: "linear", + }, + actor: "linear@app", + sourceSystem: "linear", + sourceOperation: "webhook_pull", + idempotencyKey: "linear:comment:1", + occurredAt: "2026-03-13T11:00:00.000Z", + recordedAt: "2026-03-13T11:00:01.000Z", + }; + + await writeFile( + eventsPath, + [ + await readFile(eventsPath, "utf8"), + JSON.stringify({ + ...duplicateBaseEvent, + eventId: "event-comment-1", + ts: baseTs, + orderKey: `${baseTs}-000000`, + }), + JSON.stringify({ + ...duplicateBaseEvent, + eventId: "event-comment-2", + ts: baseTs + 1, + orderKey: `${baseTs + 1}-000000`, + }), + "", + ].join("\n") + ); + + const snapshot = await store.readSnapshot(); + const comments = snapshot.commentsByTicket.get(created.ticket.ticketId) ?? []; + expect(comments).toHaveLength(1); + expect(comments[0]).toMatchObject({ + commentId: "comment-1", + body: "Imported from Linear.", + source: "linear", + }); +}, 20_000); + +test("normalized ticket adapter preserves compatibility while exposing provenance and documents", () => { + const summary = { + ticketId: "T-00042", + title: "Normalize ticket metadata", + body: "## Context\nDocument-backed description", + status: "in_progress" as const, + createdAt: "2026-03-13T10:00:00.000Z", + updatedAt: "2026-03-13T11:00:00.000Z", + dependsOn: ["T-00001"], + blocks: ["T-00009"], + owner: "hack", + source: "linear", + assignee: "alice@hack", + tags: ["core", "tickets"], + externalSystem: "linear", + externalId: "lin_123", + externalKey: "HACK-431", + externalUrl: "https://linear.app/hack/issue/HACK-431", + externalProjectId: "project-1", + externalProjectName: "Hack App", + externalTeamId: "team-1", + projectId: "hack-cli", + projectName: "hack-cli", + }; + + const normalized = createNormalizedTicket({ + ticket: summary, + syncCheckpoints: [ + { + checkpointId: "checkpoint-1", + ticketId: summary.ticketId, + provider: "linear", + profileId: "default", + direction: "pull", + remoteCursor: "issue/lin_123#v2", + remoteUpdatedAt: "2026-03-13T10:59:00.000Z", + localUpdatedAt: "2026-03-13T11:00:00.000Z", + actor: "sync@app", + createdAt: "2026-03-13T11:00:00.000Z", + }, + ], + conflicts: [ + { + conflictId: "conflict-1", + ticketId: summary.ticketId, + provider: "linear", + field: "title", + status: "open", + authority: "review_required", + summary: "Local title drifted from Linear.", + localValue: summary.title, + remoteValue: "Normalize work model", + createdAt: "2026-03-13T11:00:00.000Z", + updatedAt: "2026-03-13T11:00:00.000Z", + }, + ], + }); + + expect(normalized.identity).toEqual({ + ticketId: "T-00042", + projectId: "hack-cli", + projectName: "hack-cli", + }); + expect(normalized.provenance.origin).toEqual({ + owner: "hack", + source: "linear", + system: "linear", + }); + expect(normalized.provenance.remotes).toEqual([ + expect.objectContaining({ + provider: "linear", + remoteId: "lin_123", + remoteKey: "HACK-431", + remoteUrl: "https://linear.app/hack/issue/HACK-431", + projectId: "project-1", + projectName: "Hack App", + teamId: "team-1", + }), + ]); + expect(normalized.documents).toEqual([ + expect.objectContaining({ + kind: "description", + role: "description", + content: "## Context\nDocument-backed description", + }), + ]); + expect(normalized.fieldStates).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + field: "title", + authority: "review_required", + conflictIds: ["conflict-1"], + }), + ]) + ); + expect(projectNormalizedTicketSummary({ ticket: normalized })).toEqual( + summary + ); +}); + async function createStore(opts: { readonly projectRoot: string }) { const configResult = await readControlPlaneConfig({ projectDir: join(opts.projectRoot, ".hack"), From de42cea3705fb33182d878eb1a88841dba91b597 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 22:39:32 +0000 Subject: [PATCH 006/134] feat(tickets): define normalized ticket entity model --- .../guides/linear-integration-architecture.md | 4 + docs/guides/tickets-normalization.md | 172 ++++ docs/guides/tickets.md | 1 + .../extensions/tickets/normalized-model.ts | 886 ++++++++++++++++++ tests/tickets-normalized-model.test.ts | 444 +++++++++ 5 files changed, 1507 insertions(+) create mode 100644 docs/guides/tickets-normalization.md create mode 100644 src/control-plane/extensions/tickets/normalized-model.ts create mode 100644 tests/tickets-normalized-model.test.ts diff --git a/docs/guides/linear-integration-architecture.md b/docs/guides/linear-integration-architecture.md index 02f3db9bd..07c6d11a7 100644 --- a/docs/guides/linear-integration-architecture.md +++ b/docs/guides/linear-integration-architecture.md @@ -35,6 +35,10 @@ Synced tickets always carry explicit ownership and lineage metadata: This guarantees filtering by local-only, Linear-only, or mixed sets without ambiguous state. +The normalized replacement for these flat fields is defined in `docs/guides/tickets-normalization.md` +and `src/control-plane/extensions/tickets/normalized-model.ts`. New sync/storage work should prefer +that entity boundary over adding more `external*` fields to `TicketSummary`. + ## Sync Mapping 1. Status: diff --git a/docs/guides/tickets-normalization.md b/docs/guides/tickets-normalization.md new file mode 100644 index 000000000..b15ff83f3 --- /dev/null +++ b/docs/guides/tickets-normalization.md @@ -0,0 +1,172 @@ +# Tickets Normalization Model + +The normalized tickets model lives in `src/control-plane/extensions/tickets/normalized-model.ts`. +It defines the canonical entity shape that storage and adapter sync should converge on before the +legacy flat `TicketSummary` projection is retired. + +## Goals + +- Keep one canonical ticket entity for local-only and dual-homed work. +- Make ticket origin, external links, and field authority explicit. +- Keep provider-specific data out of canonical fields. +- Stay generic enough for Linear, GitHub, and Asana. + +## Entity Shape + +```ts +type NormalizedTicketEntity = { + schemaVersion: 1 + kind: "ticket" + id: string + canonical: { + title: string + body?: string + status: "open" | "in_progress" | "blocked" | "done" + assignee?: string + project?: { id?: string; name?: string } + tags: string[] + relationships: { + dependsOn: string[] + blocks: string[] + } + timestamps: { + createdAt: string + updatedAt: string + } + } + provenance: { + origin: + | { kind: "local"; system: "hack" } + | { kind: "external"; system: "linear" | "github" | "asana"; linkId: string } + links: NormalizedTicketLink[] + authority: NormalizedTicketAuthority + } +} +``` + +## Provenance + +`origin` answers where the ticket started: + +- `local` means the ticket originated in Hack and may or may not have synced links yet. +- `external` means the ticket was imported from a provider and points at the authoritative link. + +`origin` is derived from source lineage, not current ownership. A ticket can stay `origin.local` +while an external system is the current owner or sync target. + +`links` tracks each external relationship explicitly: + +```ts +type NormalizedTicketLink = { + linkId: string + system: "linear" | "github" | "asana" + role: "origin" | "synced" | "reference" + syncDirection: "pull" | "push" | "bidirectional" + connection?: { + profileId?: string + accountId?: string + workspaceId?: string + workspaceName?: string + } + remote: { + id: string + key?: string + url?: string + containers: { + kind: "project" | "team" | "repo" | "workspace" | "board" | "list" + id: string + name?: string + }[] + } + adapterMetadata?: Record +} +``` + +Why generic containers matter: + +- Linear uses `project` and `team` +- GitHub uses `repo` +- Asana uses `workspace`, `project`, `board`, or `list` + +Those stay normalized instead of introducing more `externalProject*`-style top-level fields. + +## Authority + +Authority is modeled separately from origin because a linked ticket can still remain Hack-owned. + +```ts +type NormalizedTicketAuthority = { + defaultRule: { + policy: "replace" | "append" | "set_union" + winner?: { kind: "local" } | { kind: "origin" } | { kind: "link"; linkId: string } + } + fieldRules: Partial< + Record< + | "title" + | "body" + | "status" + | "assignee" + | "project" + | "tags" + | "dependsOn" + | "blocks", + { + policy: "replace" | "append" | "set_union" + winner?: { kind: "local" } | { kind: "origin" } | { kind: "link"; linkId: string } + } + > + > +} +``` + +Current defaults: + +- Local-origin tickets: `defaultRule = replace/local` +- External-origin tickets: `defaultRule = replace/origin` +- `tags`, `dependsOn`, `blocks`: `set_union` + +This matches current Linear sync behavior while leaving room for GitHub or Asana-specific field +ownership later. + +## Schema Invariants + +The schema is intentionally stricter than a plain JSON shape. A valid normalized ticket must also +satisfy these cross-record rules: + +- `provenance.links[*].linkId` values are unique within a ticket. +- Local-origin tickets cannot carry an `origin`-role external link. +- External-origin tickets must point at a matching `origin`-role link with the same `system` and + `linkId`. +- Any authority winner with `{ kind: "link", linkId }` must reference a link that exists on the + same ticket. +- `buildNormalizedTicketLink()` validates remote identifiers and URLs up front, so adapter code + cannot emit blank link records accidentally. +- Legacy summaries with `source` set to an external provider must include matching + `externalSystem` and `externalId`; the bridge rejects incomplete external lineage instead of + rewriting it as a local ticket. + +Those invariants are what make the model precise enough for storage migration, conflict resolution, +and multi-provider sync instead of just being a documentation-only shape. + +## Adapter Metadata Boundary + +Canonical fields stop at `canonical`, `provenance.origin`, `provenance.links[*].remote`, and +`provenance.authority`. + +Everything provider-specific goes in `adapterMetadata`, for example: + +- Linear state ids or cycle ids +- GitHub label payloads, milestone ids, or issue type metadata +- Asana custom fields or section ids + +That boundary lets storage index the normalized fields without understanding every provider payload. + +## Migration Boundary + +The current git-backed tickets store still materializes `TicketSummary` with flat `external*` +fields. `normalizeLegacyTicketSummary()` is the bridge layer that maps the legacy projection into +the normalized entity until storage is migrated. + +Comments are intentionally out of scope for this core ticket entity. They remain separate ticket +records in the current store and should become their own normalized sync surface rather than a +field on `NormalizedTicketEntity`. diff --git a/docs/guides/tickets.md b/docs/guides/tickets.md index 279f60a02..a729262b2 100644 --- a/docs/guides/tickets.md +++ b/docs/guides/tickets.md @@ -129,6 +129,7 @@ Tip: use `--body-stdin` for multi-line markdown. - The extension reads events, materializes tickets in-memory, and renders `list/show` outputs. - Ticket writes automatically commit and push to the tickets ref when git sync is enabled and a remote exists. - `sync` normalizes the event logs, commits, and pushes the tickets ref. +- The next-step normalized entity model is documented in `docs/guides/tickets-normalization.md`. ### Storage layout diff --git a/src/control-plane/extensions/tickets/normalized-model.ts b/src/control-plane/extensions/tickets/normalized-model.ts new file mode 100644 index 000000000..8e447f92c --- /dev/null +++ b/src/control-plane/extensions/tickets/normalized-model.ts @@ -0,0 +1,886 @@ +import type { TicketSummary } from "./store.ts"; + +export const NORMALIZED_TICKET_SCHEMA_VERSION = 1 as const; + +export const TICKET_STATUSES = [ + "open", + "in_progress", + "blocked", + "done", +] as const; + +export const EXTERNAL_TICKET_SYSTEMS = ["linear", "github", "asana"] as const; + +export const TICKET_LINK_CONTAINER_KINDS = [ + "project", + "team", + "repo", + "workspace", + "board", + "list", +] as const; + +export const TICKET_AUTHORITY_FIELDS = [ + "title", + "body", + "status", + "assignee", + "project", + "tags", + "dependsOn", + "blocks", +] as const; + +export type TicketStatus = (typeof TICKET_STATUSES)[number]; +export type ExternalTicketSystem = (typeof EXTERNAL_TICKET_SYSTEMS)[number]; +export type TicketLinkContainerKind = + (typeof TICKET_LINK_CONTAINER_KINDS)[number]; +export type TicketAuthorityField = (typeof TICKET_AUTHORITY_FIELDS)[number]; + +export type TicketSchemaValue = + | string + | number + | boolean + | null + | readonly TicketSchemaValue[] + | { readonly [key: string]: TicketSchemaValue }; + +export type LegacyTicketSummary = TicketSummary; + +export type TicketLinkContainer = { + readonly kind: TicketLinkContainerKind; + readonly id: string; + readonly name?: string; +}; + +export type TicketLinkConnection = { + readonly profileId?: string; + readonly accountId?: string; + readonly workspaceId?: string; + readonly workspaceName?: string; +}; + +export type NormalizedTicketLink = { + readonly linkId: string; + readonly system: ExternalTicketSystem; + readonly role: "origin" | "synced" | "reference"; + readonly syncDirection: "pull" | "push" | "bidirectional"; + readonly connection?: TicketLinkConnection; + readonly remote: { + readonly id: string; + readonly key?: string; + readonly url?: string; + readonly containers: readonly TicketLinkContainer[]; + }; + readonly adapterMetadata?: { readonly [key: string]: TicketSchemaValue }; +}; + +export type TicketAuthorityWinner = + | { readonly kind: "local" } + | { readonly kind: "origin" } + | { readonly kind: "link"; readonly linkId: string }; + +export type TicketAuthorityRule = { + readonly policy: "replace" | "append" | "set_union"; + readonly winner?: TicketAuthorityWinner; +}; + +export type NormalizedTicketAuthority = { + readonly defaultRule: TicketAuthorityRule; + readonly fieldRules: Partial< + Record + >; +}; + +export type NormalizedTicketOrigin = + | { + readonly kind: "local"; + readonly system: "hack"; + } + | { + readonly kind: "external"; + readonly system: ExternalTicketSystem; + readonly linkId: string; + }; + +export type NormalizedTicketEntity = { + readonly schemaVersion: typeof NORMALIZED_TICKET_SCHEMA_VERSION; + readonly kind: "ticket"; + readonly id: string; + readonly canonical: { + readonly title: string; + readonly body?: string; + readonly status: TicketStatus; + readonly assignee?: string; + readonly project?: { + readonly id?: string; + readonly name?: string; + }; + readonly tags: readonly string[]; + readonly relationships: { + readonly dependsOn: readonly string[]; + readonly blocks: readonly string[]; + }; + readonly timestamps: { + readonly createdAt: string; + readonly updatedAt: string; + }; + }; + readonly provenance: { + readonly origin: NormalizedTicketOrigin; + readonly links: readonly NormalizedTicketLink[]; + readonly authority: NormalizedTicketAuthority; + }; +}; + +export type NormalizedTicketLinkInput = { + readonly system: ExternalTicketSystem; + readonly remoteId: string; + readonly remoteKey?: string; + readonly remoteUrl?: string; + readonly containers?: readonly TicketLinkContainer[]; + readonly role?: NormalizedTicketLink["role"]; + readonly syncDirection?: NormalizedTicketLink["syncDirection"]; + readonly connection?: TicketLinkConnection; + readonly adapterMetadata?: { readonly [key: string]: TicketSchemaValue }; +}; + +export type NormalizeLegacyTicketSummaryInput = { + readonly ticket: LegacyTicketSummary; + readonly linkContext?: TicketLinkConnection & { + readonly adapterMetadata?: { + readonly [key: string]: TicketSchemaValue; + }; + }; +}; + +type Schema = { + readonly parse: (input: unknown) => T; +}; + +export const normalizedTicketAuthoritySchema: Schema = + { + parse: (input) => parseNormalizedTicketAuthority(input), + }; + +export const normalizedTicketEntitySchema: Schema = { + parse: (input) => parseNormalizedTicketEntity(input), +}; + +/** + * Builds a provider-agnostic external link record that keeps provider-specific + * data inside `adapterMetadata` rather than the canonical ticket fields. + */ +export const buildNormalizedTicketLink = ( + input: NormalizedTicketLinkInput +): NormalizedTicketLink => { + const remoteId = input.remoteId.trim(); + const remoteKey = normalizeOptionalString(input.remoteKey); + const remoteUrl = normalizeOptionalString(input.remoteUrl); + const containers = normalizeContainers(input.containers ?? []); + const connection = normalizeConnection(input.connection); + const adapterMetadata = normalizeAdapterMetadata(input.adapterMetadata); + + const normalizedLink: NormalizedTicketLink = { + linkId: `${input.system}:${remoteId}`, + system: input.system, + role: input.role ?? "synced", + syncDirection: input.syncDirection ?? "bidirectional", + ...(connection ? { connection } : {}), + remote: { + id: remoteId, + ...(remoteKey ? { key: remoteKey } : {}), + ...(remoteUrl ? { url: remoteUrl } : {}), + containers, + }, + ...(adapterMetadata ? { adapterMetadata } : {}), + }; + + return parseNormalizedTicketLink(normalizedLink); +}; + +/** + * Projects the current flat ticket summary into the normalized entity shape so + * storage and sync adapters can converge on one schema before the event store + * is migrated. + */ +export const normalizeLegacyTicketSummary = ( + input: NormalizeLegacyTicketSummaryInput +): NormalizedTicketEntity => { + assertLegacySummaryRemoteIdentity(input.ticket); + const link = buildLinkFromLegacySummary({ + ticket: input.ticket, + linkContext: input.linkContext, + }); + const origin = deriveOrigin({ + ticket: input.ticket, + link, + }); + const authority = deriveAuthority({ + origin, + }); + const project = normalizeProject({ + id: input.ticket.projectId, + name: input.ticket.projectName, + }); + + const normalized: NormalizedTicketEntity = { + schemaVersion: NORMALIZED_TICKET_SCHEMA_VERSION, + kind: "ticket", + id: input.ticket.ticketId, + canonical: { + title: input.ticket.title.trim(), + ...(input.ticket.body !== undefined ? { body: input.ticket.body } : {}), + status: input.ticket.status, + ...(normalizeOptionalString(input.ticket.assignee) + ? { assignee: input.ticket.assignee.trim() } + : {}), + ...(project ? { project } : {}), + tags: normalizeStringList(input.ticket.tags), + relationships: { + dependsOn: normalizeStringList(input.ticket.dependsOn), + blocks: normalizeStringList(input.ticket.blocks), + }, + timestamps: { + createdAt: input.ticket.createdAt, + updatedAt: input.ticket.updatedAt, + }, + }, + provenance: { + origin, + links: link ? [link] : [], + authority, + }, + }; + + return normalizedTicketEntitySchema.parse(normalized); +}; + +const buildLinkFromLegacySummary = (input: { + readonly ticket: LegacyTicketSummary; + readonly linkContext?: NormalizeLegacyTicketSummaryInput["linkContext"]; +}): NormalizedTicketLink | undefined => { + const system = normalizeExternalSystem(input.ticket.externalSystem); + const remoteId = normalizeOptionalString(input.ticket.externalId); + if (!(system && remoteId)) { + return undefined; + } + + const containers: TicketLinkContainer[] = []; + const projectId = normalizeOptionalString(input.ticket.externalProjectId); + const projectName = normalizeOptionalString(input.ticket.externalProjectName); + if (projectId) { + containers.push({ + kind: "project", + id: projectId, + ...(projectName ? { name: projectName } : {}), + }); + } + + const teamId = normalizeOptionalString(input.ticket.externalTeamId); + if (teamId) { + containers.push({ + kind: "team", + id: teamId, + }); + } + + return buildNormalizedTicketLink({ + system, + remoteId, + remoteKey: input.ticket.externalKey, + remoteUrl: input.ticket.externalUrl, + containers, + role: isExternalOrigin({ ticket: input.ticket, system }) + ? "origin" + : "synced", + syncDirection: "bidirectional", + connection: input.linkContext, + adapterMetadata: input.linkContext?.adapterMetadata, + }); +}; + +const deriveOrigin = (input: { + readonly ticket: LegacyTicketSummary; + readonly link?: NormalizedTicketLink; +}): NormalizedTicketOrigin => { + if ( + input.link && + input.link.role === "origin" && + normalizeOwnerLikeValue(input.ticket.source) === input.link.system + ) { + return { + kind: "external", + system: input.link.system, + linkId: input.link.linkId, + }; + } + + return { + kind: "local", + system: "hack", + }; +}; + +const deriveAuthority = (input: { + readonly origin: NormalizedTicketOrigin; +}): NormalizedTicketAuthority => { + const defaultRule: TicketAuthorityRule = + input.origin.kind === "external" + ? { + policy: "replace", + winner: { kind: "origin" }, + } + : { + policy: "replace", + winner: { kind: "local" }, + }; + + return { + defaultRule, + fieldRules: { + tags: { + policy: "set_union", + }, + dependsOn: { + policy: "set_union", + }, + blocks: { + policy: "set_union", + }, + }, + }; +}; + +const assertLegacySummaryRemoteIdentity = ( + ticket: LegacyTicketSummary +): void => { + const sourceSystem = normalizeExternalSystem(ticket.source); + if (!sourceSystem) { + return; + } + + const externalSystem = normalizeExternalSystem(ticket.externalSystem); + const remoteId = normalizeOptionalString(ticket.externalId); + assertValue( + externalSystem === sourceSystem && remoteId !== undefined, + "externally sourced legacy ticket requires externalSystem and externalId" + ); +}; + +const isExternalOrigin = (input: { + readonly ticket: LegacyTicketSummary; + readonly system: ExternalTicketSystem; +}): boolean => normalizeOwnerLikeValue(input.ticket.source) === input.system; + +const normalizeProject = (input: { + readonly id?: string; + readonly name?: string; +}): + | { + readonly id?: string; + readonly name?: string; + } + | undefined => { + const id = normalizeOptionalString(input.id); + const name = normalizeOptionalString(input.name); + if (!(id || name)) { + return undefined; + } + return { + ...(id ? { id } : {}), + ...(name ? { name } : {}), + }; +}; + +const normalizeConnection = ( + input: TicketLinkConnection | undefined +): TicketLinkConnection | undefined => { + if (!input) { + return undefined; + } + + const profileId = normalizeOptionalString(input.profileId); + const accountId = normalizeOptionalString(input.accountId); + const workspaceId = normalizeOptionalString(input.workspaceId); + const workspaceName = normalizeOptionalString(input.workspaceName); + + if (!(profileId || accountId || workspaceId || workspaceName)) { + return undefined; + } + + return { + ...(profileId ? { profileId } : {}), + ...(accountId ? { accountId } : {}), + ...(workspaceId ? { workspaceId } : {}), + ...(workspaceName ? { workspaceName } : {}), + }; +}; + +const normalizeContainers = ( + input: readonly TicketLinkContainer[] +): TicketLinkContainer[] => { + const normalized: TicketLinkContainer[] = []; + const seen = new Set(); + + for (const container of input) { + const id = normalizeOptionalString(container.id); + if (!id) { + continue; + } + const name = normalizeOptionalString(container.name); + const key = `${container.kind}:${id}`; + if (seen.has(key)) { + continue; + } + seen.add(key); + normalized.push({ + kind: container.kind, + id, + ...(name ? { name } : {}), + }); + } + + return normalized; +}; + +const normalizeStringList = (input: readonly string[]): string[] => { + const normalized: string[] = []; + const seen = new Set(); + + for (const item of input) { + const value = normalizeOptionalString(item); + if (!(value && !seen.has(value))) { + continue; + } + seen.add(value); + normalized.push(value); + } + + return normalized; +}; + +const normalizeAdapterMetadata = ( + input: { readonly [key: string]: TicketSchemaValue } | undefined +): { readonly [key: string]: TicketSchemaValue } | undefined => { + if (!input) { + return undefined; + } + + const entries = Object.entries(input) + .map(([key, value]) => [key.trim(), value] as const) + .filter(([key]) => key.length > 0); + + if (entries.length === 0) { + return undefined; + } + + return Object.fromEntries(entries); +}; + +const normalizeOptionalString = ( + input: string | undefined +): string | undefined => { + const value = input?.trim(); + return value ? value : undefined; +}; + +const normalizeOwnerLikeValue = ( + input: string | undefined +): string | undefined => normalizeOptionalString(input)?.toLowerCase(); + +const normalizeExternalSystem = ( + input: string | undefined +): ExternalTicketSystem | undefined => { + const value = normalizeOwnerLikeValue(input); + if (value === "linear" || value === "github" || value === "asana") { + return value; + } + return undefined; +}; + +const parseNormalizedTicketEntity = ( + input: unknown +): NormalizedTicketEntity => { + const record = asRecord(input, "normalized ticket entity"); + assertValue( + record.schemaVersion === NORMALIZED_TICKET_SCHEMA_VERSION, + "normalized ticket entity schemaVersion must be 1" + ); + assertValue( + record.kind === "ticket", + "normalized ticket entity kind must be ticket" + ); + + const canonicalRecord = asRecord( + record.canonical, + "normalized ticket entity canonical" + ); + const provenanceRecord = asRecord( + record.provenance, + "normalized ticket entity provenance" + ); + + assertNonEmptyString(record.id, "normalized ticket entity id"); + assertNonEmptyString( + canonicalRecord.title, + "normalized ticket canonical title" + ); + if (canonicalRecord.body !== undefined) { + assertString(canonicalRecord.body, "normalized ticket canonical body"); + } + assertTicketStatus(canonicalRecord.status); + if (canonicalRecord.assignee !== undefined) { + assertNonEmptyString( + canonicalRecord.assignee, + "normalized ticket canonical assignee" + ); + } + + if (canonicalRecord.project !== undefined) { + const projectRecord = asRecord( + canonicalRecord.project, + "normalized ticket canonical project" + ); + assertValue( + typeof projectRecord.id === "string" || + typeof projectRecord.name === "string", + "normalized ticket canonical project must include an id or name" + ); + if (projectRecord.id !== undefined) { + assertNonEmptyString( + projectRecord.id, + "normalized ticket canonical project id" + ); + } + if (projectRecord.name !== undefined) { + assertNonEmptyString( + projectRecord.name, + "normalized ticket canonical project name" + ); + } + } + + assertStringList(canonicalRecord.tags, "normalized ticket canonical tags"); + + const relationshipsRecord = asRecord( + canonicalRecord.relationships, + "normalized ticket canonical relationships" + ); + assertStringList( + relationshipsRecord.dependsOn, + "normalized ticket canonical dependsOn" + ); + assertStringList( + relationshipsRecord.blocks, + "normalized ticket canonical blocks" + ); + + const timestampsRecord = asRecord( + canonicalRecord.timestamps, + "normalized ticket canonical timestamps" + ); + assertIsoTimestamp(timestampsRecord.createdAt, "normalized ticket createdAt"); + assertIsoTimestamp(timestampsRecord.updatedAt, "normalized ticket updatedAt"); + + const origin = parseNormalizedTicketOrigin(provenanceRecord.origin); + assertValue( + Array.isArray(provenanceRecord.links), + "normalized ticket links must be an array" + ); + const links = provenanceRecord.links.map((link) => + parseNormalizedTicketLink(link) + ); + const linkIds = new Set(); + const originLinks: NormalizedTicketLink[] = []; + for (const link of links) { + assertValue( + !linkIds.has(link.linkId), + `normalized ticket link ${link.linkId} must be unique` + ); + linkIds.add(link.linkId); + if (link.role === "origin") { + originLinks.push(link); + } + } + + if (origin.kind === "local") { + assertValue( + originLinks.length === 0, + "normalized ticket local origin cannot include origin links" + ); + } + + if (origin.kind === "external") { + const matchingOriginLink = originLinks.find( + (link) => link.linkId === origin.linkId && link.system === origin.system + ); + assertValue( + matchingOriginLink !== undefined, + "normalized ticket external origin must reference a matching origin link" + ); + } + + parseNormalizedTicketAuthority(provenanceRecord.authority, linkIds); + + return record as NormalizedTicketEntity; +}; + +const parseNormalizedTicketAuthority = ( + input: unknown, + linkIds?: ReadonlySet +): NormalizedTicketAuthority => { + const record = asRecord(input, "normalized ticket authority"); + parseTicketAuthorityRule( + record.defaultRule, + "normalized ticket authority defaultRule", + linkIds + ); + const fieldRules = asRecord( + record.fieldRules, + "normalized ticket authority fieldRules" + ); + for (const [key, value] of Object.entries(fieldRules)) { + assertValue( + (TICKET_AUTHORITY_FIELDS as readonly string[]).includes(key), + `normalized ticket authority field ${key} is not supported` + ); + parseTicketAuthorityRule( + value, + `normalized ticket authority fieldRules.${key}`, + linkIds + ); + } + return record as NormalizedTicketAuthority; +}; + +const parseNormalizedTicketOrigin = ( + input: unknown +): NormalizedTicketOrigin => { + const record = asRecord(input, "normalized ticket origin"); + assertValue( + record.kind === "local" || record.kind === "external", + "normalized ticket origin kind must be local or external" + ); + + if (record.kind === "local") { + assertValue( + record.system === "hack", + "local ticket origin system must be hack" + ); + return record as NormalizedTicketOrigin; + } + + assertExternalTicketSystem(record.system); + assertNonEmptyString(record.linkId, "external ticket origin linkId"); + return record as NormalizedTicketOrigin; +}; + +const parseNormalizedTicketLink = (input: unknown): NormalizedTicketLink => { + const record = asRecord(input, "normalized ticket link"); + assertNonEmptyString(record.linkId, "normalized ticket linkId"); + assertExternalTicketSystem(record.system); + assertValue( + record.role === "origin" || + record.role === "synced" || + record.role === "reference", + "normalized ticket link role must be origin, synced, or reference" + ); + assertValue( + record.syncDirection === "pull" || + record.syncDirection === "push" || + record.syncDirection === "bidirectional", + "normalized ticket link syncDirection must be pull, push, or bidirectional" + ); + + if (record.connection !== undefined) { + const connectionRecord = asRecord( + record.connection, + "normalized ticket connection" + ); + const values = [ + connectionRecord.profileId, + connectionRecord.accountId, + connectionRecord.workspaceId, + connectionRecord.workspaceName, + ].filter((value) => typeof value === "string"); + assertValue( + values.length > 0, + "normalized ticket connection must include at least one identifier" + ); + for (const [key, value] of Object.entries(connectionRecord)) { + assertNonEmptyString(value, `normalized ticket connection ${key}`); + } + } + + const remoteRecord = asRecord(record.remote, "normalized ticket remote"); + assertNonEmptyString(remoteRecord.id, "normalized ticket remote id"); + if (remoteRecord.key !== undefined) { + assertNonEmptyString(remoteRecord.key, "normalized ticket remote key"); + } + if (remoteRecord.url !== undefined) { + assertUrl(remoteRecord.url, "normalized ticket remote url"); + } + assertValue( + Array.isArray(remoteRecord.containers), + "normalized ticket remote containers must be an array" + ); + for (const container of remoteRecord.containers) { + const containerRecord = asRecord(container, "normalized ticket container"); + assertValue( + (TICKET_LINK_CONTAINER_KINDS as readonly string[]).includes( + String(containerRecord.kind) + ), + "normalized ticket container kind is invalid" + ); + assertNonEmptyString(containerRecord.id, "normalized ticket container id"); + if (containerRecord.name !== undefined) { + assertNonEmptyString( + containerRecord.name, + "normalized ticket container name" + ); + } + } + + if (record.adapterMetadata !== undefined) { + const metadataRecord = asRecord( + record.adapterMetadata, + "normalized ticket adapter metadata" + ); + for (const [key, value] of Object.entries(metadataRecord)) { + assertNonEmptyString(key, "normalized ticket adapter metadata key"); + assertTicketSchemaValue( + value, + `normalized ticket adapter metadata ${key}` + ); + } + } + + return record as NormalizedTicketLink; +}; + +const parseTicketAuthorityRule = ( + input: unknown, + label: string, + linkIds?: ReadonlySet +): TicketAuthorityRule => { + const record = asRecord(input, label); + assertValue( + record.policy === "replace" || + record.policy === "append" || + record.policy === "set_union", + `${label} policy must be replace, append, or set_union` + ); + + if (record.winner !== undefined) { + const winnerRecord = asRecord(record.winner, `${label} winner`); + assertValue( + winnerRecord.kind === "local" || + winnerRecord.kind === "origin" || + winnerRecord.kind === "link", + `${label} winner kind must be local, origin, or link` + ); + if (winnerRecord.kind === "link") { + assertNonEmptyString(winnerRecord.linkId, `${label} winner linkId`); + if (linkIds) { + assertValue( + linkIds.has(winnerRecord.linkId), + `${label} winner linkId must reference an existing link` + ); + } + } + } + + return record as TicketAuthorityRule; +}; + +const assertTicketSchemaValue = (input: unknown, label: string): void => { + if ( + input === null || + typeof input === "string" || + typeof input === "number" || + typeof input === "boolean" + ) { + return; + } + + if (Array.isArray(input)) { + for (const [index, value] of input.entries()) { + assertTicketSchemaValue(value, `${label}[${index}]`); + } + return; + } + + if (typeof input === "object") { + for (const [key, value] of Object.entries(asRecord(input, label))) { + assertNonEmptyString(key, `${label} key`); + assertTicketSchemaValue(value, `${label}.${key}`); + } + return; + } + + throw new Error(`${label} must be a JSON-like metadata value`); +}; + +const assertStringList = (input: unknown, label: string): void => { + assertValue(Array.isArray(input), `${label} must be an array`); + for (const [index, value] of input.entries()) { + assertNonEmptyString(value, `${label}[${index}]`); + } +}; + +const assertTicketStatus = (input: unknown): asserts input is TicketStatus => { + assertValue( + (TICKET_STATUSES as readonly string[]).includes(String(input)), + "ticket status must be open, in_progress, blocked, or done" + ); +}; + +const assertExternalTicketSystem = ( + input: unknown +): asserts input is ExternalTicketSystem => { + assertValue( + (EXTERNAL_TICKET_SYSTEMS as readonly string[]).includes(String(input)), + "external ticket system must be linear, github, or asana" + ); +}; + +const assertIsoTimestamp = (input: unknown, label: string): void => { + assertNonEmptyString(input, label); + assertValue( + !Number.isNaN(Date.parse(input)), + `${label} must be an ISO timestamp` + ); +}; + +const assertUrl = (input: unknown, label: string): void => { + assertNonEmptyString(input, label); + try { + new URL(input); + } catch { + throw new Error(`${label} must be a valid URL`); + } +}; + +const assertString = (input: unknown, label: string): void => { + assertValue(typeof input === "string", `${label} must be a string`); +}; + +const assertNonEmptyString = (input: unknown, label: string): void => { + assertString(input, label); + assertValue(input.trim().length > 0, `${label} must be a non-empty string`); +}; + +const asRecord = (input: unknown, label: string): Record => { + assertValue( + Boolean(input) && typeof input === "object" && !Array.isArray(input), + `${label} must be an object` + ); + return input as Record; +}; + +const assertValue = (condition: boolean, message: string): void => { + if (!condition) { + throw new Error(message); + } +}; diff --git a/tests/tickets-normalized-model.test.ts b/tests/tickets-normalized-model.test.ts new file mode 100644 index 000000000..a9fec0f98 --- /dev/null +++ b/tests/tickets-normalized-model.test.ts @@ -0,0 +1,444 @@ +import { expect, test } from "bun:test"; + +import { + buildNormalizedTicketLink, + normalizedTicketAuthoritySchema, + normalizedTicketEntitySchema, + normalizeLegacyTicketSummary, + TICKET_AUTHORITY_FIELDS, +} from "../src/control-plane/extensions/tickets/normalized-model.ts"; + +const createMinimalNormalizedTicket = () => + ({ + schemaVersion: 1, + kind: "ticket", + id: "T-00099", + canonical: { + title: "Normalized ticket", + status: "open", + tags: [], + relationships: { + dependsOn: [], + blocks: [], + }, + timestamps: { + createdAt: "2026-03-13T18:00:00.000Z", + updatedAt: "2026-03-13T18:05:00.000Z", + }, + }, + provenance: { + origin: { + kind: "local", + system: "hack", + }, + links: [], + authority: { + defaultRule: { + policy: "replace", + winner: { kind: "local" }, + }, + fieldRules: {}, + }, + }, + }) as const; + +test("normalizes a local-only ticket with local authority defaults", () => { + const normalized = normalizeLegacyTicketSummary({ + ticket: { + ticketId: "T-00042", + title: "Keep local-only work local", + body: "No remote linkage yet.", + status: "open", + createdAt: "2026-03-13T18:00:00.000Z", + updatedAt: "2026-03-13T18:05:00.000Z", + dependsOn: [], + blocks: [], + owner: "hack", + source: "hack", + tags: ["core"], + projectId: "proj_local", + projectName: "Hack App", + }, + }); + + expect(normalizedTicketEntitySchema.parse(normalized)).toEqual(normalized); + expect(normalized.provenance.origin).toEqual({ + kind: "local", + system: "hack", + }); + expect(normalized.provenance.links).toEqual([]); + expect(normalized.provenance.authority.defaultRule).toEqual({ + policy: "replace", + winner: { kind: "local" }, + }); + expect(normalized.canonical.project).toEqual({ + id: "proj_local", + name: "Hack App", + }); +}); + +test("normalizes a Linear-origin ticket into generic provenance, links, and authority", () => { + const normalized = normalizeLegacyTicketSummary({ + ticket: { + ticketId: "T-00043", + title: "Imported from Linear", + body: "Remote source stays authoritative.", + status: "in_progress", + createdAt: "2026-03-13T18:10:00.000Z", + updatedAt: "2026-03-13T18:15:00.000Z", + dependsOn: ["T-00042"], + blocks: [], + owner: "linear", + source: "linear", + assignee: "alice@hack", + tags: ["sync", "linear"], + externalSystem: "linear", + externalId: "lin_123", + externalKey: "HACK-449", + externalUrl: "https://linear.app/hackdance/issue/HACK-449", + externalProjectId: "linear_project_1", + externalProjectName: "Tickets Normalization Core", + externalTeamId: "linear_team_1", + projectId: "proj_linear", + projectName: "Hack App", + }, + linkContext: { + profileId: "default", + workspaceId: "linear_workspace_1", + workspaceName: "Hack Dance", + adapterMetadata: { + stateId: "started", + priority: 2, + }, + }, + }); + + expect(normalizedTicketEntitySchema.parse(normalized)).toEqual(normalized); + expect(normalized.provenance.origin).toEqual({ + kind: "external", + system: "linear", + linkId: "linear:lin_123", + }); + expect(normalized.provenance.authority.defaultRule).toEqual({ + policy: "replace", + winner: { kind: "origin" }, + }); + expect(normalized.provenance.links).toEqual([ + { + linkId: "linear:lin_123", + system: "linear", + role: "origin", + syncDirection: "bidirectional", + connection: { + profileId: "default", + workspaceId: "linear_workspace_1", + workspaceName: "Hack Dance", + }, + remote: { + id: "lin_123", + key: "HACK-449", + url: "https://linear.app/hackdance/issue/HACK-449", + containers: [ + { + kind: "project", + id: "linear_project_1", + name: "Tickets Normalization Core", + }, + { + kind: "team", + id: "linear_team_1", + }, + ], + }, + adapterMetadata: { + stateId: "started", + priority: 2, + }, + }, + ]); +}); + +test("builds generic external links that keep adapter metadata outside canonical fields", () => { + const link = buildNormalizedTicketLink({ + system: "github", + remoteId: "issue_987", + remoteKey: "hack-dance/hack-cli#987", + remoteUrl: "https://github.com/hack-dance/hack-cli/issues/987", + containers: [ + { + kind: "repo", + id: "hack-dance/hack-cli", + name: "hack-dance/hack-cli", + }, + ], + connection: { + profileId: "work", + accountId: "installation_1", + workspaceName: "hack-dance", + }, + adapterMetadata: { + labels: ["tickets", "sync"], + milestone: "Tickets Normalization Core", + }, + }); + + expect(link).toEqual({ + linkId: "github:issue_987", + system: "github", + role: "synced", + syncDirection: "bidirectional", + connection: { + profileId: "work", + accountId: "installation_1", + workspaceName: "hack-dance", + }, + remote: { + id: "issue_987", + key: "hack-dance/hack-cli#987", + url: "https://github.com/hack-dance/hack-cli/issues/987", + containers: [ + { + kind: "repo", + id: "hack-dance/hack-cli", + name: "hack-dance/hack-cli", + }, + ], + }, + adapterMetadata: { + labels: ["tickets", "sync"], + milestone: "Tickets Normalization Core", + }, + }); +}); + +test("keeps Hack-origin tickets local while adding synced external links", () => { + const normalized = normalizeLegacyTicketSummary({ + ticket: { + ticketId: "T-00044", + title: "Push to GitHub after local creation", + status: "open", + createdAt: "2026-03-13T18:20:00.000Z", + updatedAt: "2026-03-13T18:25:00.000Z", + dependsOn: [], + blocks: [], + owner: "hack", + source: "hack", + tags: ["github"], + externalSystem: "github", + externalId: "issue_44", + externalKey: "hack-dance/hack-cli#44", + externalUrl: "https://github.com/hack-dance/hack-cli/issues/44", + }, + }); + + expect(normalized.provenance.origin).toEqual({ + kind: "local", + system: "hack", + }); + expect(normalized.provenance.links[0]).toMatchObject({ + system: "github", + role: "synced", + linkId: "github:issue_44", + }); + expect(normalized.provenance.authority.defaultRule).toEqual({ + policy: "replace", + winner: { kind: "local" }, + }); +}); + +test("treats owner as separate from origin when a local ticket is synced externally", () => { + const normalized = normalizeLegacyTicketSummary({ + ticket: { + ticketId: "T-00045", + title: "Locally created but externally owned", + status: "in_progress", + createdAt: "2026-03-13T18:30:00.000Z", + updatedAt: "2026-03-13T18:35:00.000Z", + dependsOn: [], + blocks: [], + owner: "linear", + source: "hack", + tags: ["linear"], + externalSystem: "linear", + externalId: "lin_45", + externalKey: "HACK-45", + externalUrl: "https://linear.app/hackdance/issue/HACK-45", + }, + }); + + expect(normalized.provenance.origin).toEqual({ + kind: "local", + system: "hack", + }); + expect(normalized.provenance.links).toEqual([ + expect.objectContaining({ + linkId: "linear:lin_45", + system: "linear", + role: "synced", + }), + ]); +}); + +test("covers the canonical authority fields required by sync adapters", () => { + expect(TICKET_AUTHORITY_FIELDS).toEqual([ + "title", + "body", + "status", + "assignee", + "project", + "tags", + "dependsOn", + "blocks", + ]); +}); + +test("rejects externally sourced legacy tickets that do not include remote identity", () => { + expect(() => + normalizeLegacyTicketSummary({ + ticket: { + ticketId: "T-00046", + title: "Imported but missing remote identity", + status: "open", + createdAt: "2026-03-13T18:40:00.000Z", + updatedAt: "2026-03-13T18:45:00.000Z", + dependsOn: [], + blocks: [], + owner: "linear", + source: "linear", + tags: ["linear"], + externalSystem: "linear", + }, + }) + ).toThrow( + "externally sourced legacy ticket requires externalSystem and externalId" + ); +}); + +test("does not fabricate container ids from legacy project names", () => { + const normalized = normalizeLegacyTicketSummary({ + ticket: { + ticketId: "T-00047", + title: "Imported with a project name only", + status: "open", + createdAt: "2026-03-13T18:50:00.000Z", + updatedAt: "2026-03-13T18:55:00.000Z", + dependsOn: [], + blocks: [], + owner: "linear", + source: "linear", + tags: ["linear"], + externalSystem: "linear", + externalId: "lin_47", + externalProjectName: "Name Without Stable Id", + externalTeamId: "team_47", + }, + }); + + expect(normalized.provenance.links).toEqual([ + { + linkId: "linear:lin_47", + system: "linear", + role: "origin", + syncDirection: "bidirectional", + remote: { + id: "lin_47", + containers: [ + { + kind: "team", + id: "team_47", + }, + ], + }, + }, + ]); +}); + +test("rejects external origin records that do not point at a matching origin link", () => { + const normalized = createMinimalNormalizedTicket(); + + expect(() => + normalizedTicketEntitySchema.parse({ + ...normalized, + provenance: { + ...normalized.provenance, + origin: { + kind: "external", + system: "linear", + linkId: "linear:lin_missing", + }, + links: [ + buildNormalizedTicketLink({ + system: "linear", + remoteId: "lin_123", + role: "origin", + }), + ], + }, + }) + ).toThrow( + "normalized ticket external origin must reference a matching origin link" + ); +}); + +test("rejects authority winners that reference unknown links", () => { + const normalized = createMinimalNormalizedTicket(); + + expect(() => + normalizedTicketEntitySchema.parse({ + ...normalized, + provenance: { + ...normalized.provenance, + links: [ + buildNormalizedTicketLink({ + system: "github", + remoteId: "issue_42", + }), + ], + authority: { + defaultRule: { + policy: "replace", + winner: { kind: "link", linkId: "github:issue_missing" }, + }, + fieldRules: {}, + }, + }, + }) + ).toThrow( + "normalized ticket authority defaultRule winner linkId must reference an existing link" + ); +}); + +test("parses standalone authority rules with link winners without entity context", () => { + expect( + normalizedTicketAuthoritySchema.parse({ + defaultRule: { + policy: "replace", + winner: { kind: "link", linkId: "github:issue_42" }, + }, + fieldRules: { + tags: { + policy: "set_union", + }, + }, + }) + ).toEqual({ + defaultRule: { + policy: "replace", + winner: { kind: "link", linkId: "github:issue_42" }, + }, + fieldRules: { + tags: { + policy: "set_union", + }, + }, + }); +}); + +test("rejects blank remote identifiers when building normalized links", () => { + expect(() => + buildNormalizedTicketLink({ + system: "asana", + remoteId: " ", + }) + ).toThrow("normalized ticket remote id must be a non-empty string"); +}); From 57c479e1db1b47fdb2822faaea791fe349fca7d8 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 22:51:02 +0000 Subject: [PATCH 007/134] feat(tickets): persist sqlite projection cache --- .../extensions/tickets/sqlite-projection.ts | 402 ++++++++++++++++++ src/control-plane/extensions/tickets/store.ts | 207 +++++++-- tests/tickets-extension.test.ts | 52 +++ tests/tickets-store.test.ts | 42 ++ 4 files changed, 656 insertions(+), 47 deletions(-) create mode 100644 src/control-plane/extensions/tickets/sqlite-projection.ts diff --git a/src/control-plane/extensions/tickets/sqlite-projection.ts b/src/control-plane/extensions/tickets/sqlite-projection.ts new file mode 100644 index 000000000..141299da2 --- /dev/null +++ b/src/control-plane/extensions/tickets/sqlite-projection.ts @@ -0,0 +1,402 @@ +import { Database } from "bun:sqlite"; +import { mkdir, readdir } from "node:fs/promises"; +import { dirname, resolve } from "node:path"; + +import type { + TicketComment, + TicketEvent, + TicketReviewNote, + TicketStoreSnapshot, + TicketSummary, + TicketSyncCheckpoint, + TicketSyncConflict, +} from "./store.ts"; +import { sha256Hex, stableStringify } from "./util.ts"; + +const PROJECTION_SCHEMA_VERSION = 1; + +type ProjectionMetaRow = { + readonly key: string; + readonly value: string; +}; + +type ProjectionJsonRow = { + readonly ticket_id: string; + readonly json_value: string; +}; + +type ProjectionEventRow = { + readonly ticket_id: string; + readonly json_value: string; + readonly ts: number; + readonly order_key: string | null; +}; + +export function createTicketsSqliteProjection(opts: { + readonly projectRoot: string; +}): { + readonly computeJournalSignature: (input: { + readonly ticketsRoot: string; + }) => Promise; + readonly path: string; + readonly readSnapshot: (input: { + readonly journalSignature: string; + }) => Promise; + readonly replaceSnapshot: (input: { + readonly journalSignature: string; + readonly snapshot: TicketStoreSnapshot; + }) => Promise; +} { + const path = resolve(opts.projectRoot, ".hack/tickets/projection.sqlite"); + + const computeJournalSignature = async (input: { + readonly ticketsRoot: string; + }): Promise => { + const eventsDir = resolve(input.ticketsRoot, ".hack/tickets/events"); + let entries: string[] = []; + try { + entries = (await readdir(eventsDir)) + .filter((entry) => entry.endsWith(".jsonl")) + .sort(); + } catch { + return sha256Hex({ value: "" }); + } + + const files = await Promise.all( + entries.map(async (entry) => { + const text = await Bun.file(resolve(eventsDir, entry)) + .text() + .catch(() => ""); + return { + entry, + text, + }; + }) + ); + + return sha256Hex({ + value: stableStringify( + files.map((file) => ({ + entry: file.entry, + text: file.text, + })) + ), + }); + }; + + const replaceSnapshot = async (input: { + readonly journalSignature: string; + readonly snapshot: TicketStoreSnapshot; + }): Promise => { + await mkdir(dirname(path), { recursive: true }); + const db = new Database(path); + try { + initializeProjectionSchema({ db }); + + const insertMeta = db.query( + "INSERT INTO projection_meta (key, value) VALUES (?, ?)" + ); + const insertTicket = db.query( + "INSERT INTO tickets (ticket_id, title, updated_at, json_value) VALUES (?, ?, ?, ?)" + ); + const insertEvent = db.query( + "INSERT INTO journal_events (event_id, ticket_id, ts, order_key, event_type, idempotency_key, json_value) VALUES (?, ?, ?, ?, ?, ?, ?)" + ); + const insertComment = db.query( + "INSERT INTO ticket_comments (comment_id, ticket_id, json_value) VALUES (?, ?, ?)" + ); + const insertReviewNote = db.query( + "INSERT INTO ticket_review_notes (note_id, ticket_id, json_value) VALUES (?, ?, ?)" + ); + const insertCheckpoint = db.query( + "INSERT INTO ticket_sync_checkpoints (checkpoint_id, ticket_id, json_value) VALUES (?, ?, ?)" + ); + const insertConflict = db.query( + "INSERT INTO ticket_sync_conflicts (conflict_id, ticket_id, json_value) VALUES (?, ?, ?)" + ); + + const events = [...input.snapshot.eventsByTicket.values()].flat(); + + const transaction = db.transaction(() => { + db.exec(` + DELETE FROM projection_meta; + DELETE FROM tickets; + DELETE FROM journal_events; + DELETE FROM ticket_comments; + DELETE FROM ticket_review_notes; + DELETE FROM ticket_sync_checkpoints; + DELETE FROM ticket_sync_conflicts; + `); + + insertMeta.run("schema_version", String(PROJECTION_SCHEMA_VERSION)); + insertMeta.run("journal_signature", input.journalSignature); + insertMeta.run("rebuilt_at", new Date().toISOString()); + + for (const ticket of input.snapshot.tickets) { + insertTicket.run( + ticket.ticketId, + ticket.title, + ticket.updatedAt, + JSON.stringify(ticket) + ); + } + + for (const event of events) { + insertEvent.run( + event.eventId, + event.ticketId, + event.ts, + event.orderKey ?? null, + event.eventType, + event.idempotencyKey, + JSON.stringify(event) + ); + } + + writeMapEntries({ + map: input.snapshot.commentsByTicket, + write: (comment: TicketComment) => { + insertComment.run( + comment.commentId, + comment.ticketId, + JSON.stringify(comment) + ); + }, + }); + + writeMapEntries({ + map: input.snapshot.reviewNotesByTicket, + write: (reviewNote: TicketReviewNote) => { + insertReviewNote.run( + reviewNote.noteId, + reviewNote.ticketId, + JSON.stringify(reviewNote) + ); + }, + }); + + writeMapEntries({ + map: input.snapshot.syncCheckpointsByTicket, + write: (checkpoint: TicketSyncCheckpoint) => { + insertCheckpoint.run( + checkpoint.checkpointId, + checkpoint.ticketId, + JSON.stringify(checkpoint) + ); + }, + }); + + writeMapEntries({ + map: input.snapshot.conflictsByTicket, + write: (conflict: TicketSyncConflict) => { + insertConflict.run( + conflict.conflictId, + conflict.ticketId, + JSON.stringify(conflict) + ); + }, + }); + }); + + transaction(); + } finally { + db.close(); + } + }; + + const readSnapshot = async (input: { + readonly journalSignature: string; + }): Promise => { + if (!(await Bun.file(path).exists())) { + return null; + } + + const db = new Database(path); + try { + initializeProjectionSchema({ db }); + const metaRows = db + .query("SELECT key, value FROM projection_meta") + .all() as ProjectionMetaRow[]; + const meta = new Map( + metaRows.map((row) => [row.key, row.value] as const) + ); + if ( + meta.get("schema_version") !== String(PROJECTION_SCHEMA_VERSION) || + meta.get("journal_signature") !== input.journalSignature + ) { + return null; + } + + const tickets = ( + db + .query( + "SELECT ticket_id, json_value FROM tickets ORDER BY CAST(SUBSTR(ticket_id, 3) AS INTEGER), ticket_id" + ) + .all() as ProjectionJsonRow[] + ) + .map((row) => + parseProjectionJson({ json: row.json_value }) + ) + .filter((ticket): ticket is TicketSummary => ticket !== null); + + const eventsByTicket = readEventRows({ db }); + const commentsByTicket = readRowsByTicket({ + db, + table: "ticket_comments", + }); + const reviewNotesByTicket = readRowsByTicket({ + db, + table: "ticket_review_notes", + }); + const syncCheckpointsByTicket = readRowsByTicket({ + db, + table: "ticket_sync_checkpoints", + }); + const conflictsByTicket = readRowsByTicket({ + db, + table: "ticket_sync_conflicts", + }); + + return { + tickets, + eventsByTicket, + commentsByTicket, + reviewNotesByTicket, + syncCheckpointsByTicket, + conflictsByTicket, + }; + } finally { + db.close(); + } + }; + + return { + computeJournalSignature, + path, + readSnapshot, + replaceSnapshot, + }; +} + +function initializeProjectionSchema(input: { readonly db: Database }): void { + input.db.exec(` + PRAGMA journal_mode = WAL; + CREATE TABLE IF NOT EXISTS projection_meta ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS tickets ( + ticket_id TEXT PRIMARY KEY, + title TEXT NOT NULL, + updated_at TEXT NOT NULL, + json_value TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS journal_events ( + event_id TEXT PRIMARY KEY, + ticket_id TEXT NOT NULL, + ts INTEGER NOT NULL, + order_key TEXT, + event_type TEXT NOT NULL, + idempotency_key TEXT NOT NULL, + json_value TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS ticket_comments ( + comment_id TEXT PRIMARY KEY, + ticket_id TEXT NOT NULL, + json_value TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS ticket_review_notes ( + note_id TEXT PRIMARY KEY, + ticket_id TEXT NOT NULL, + json_value TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS ticket_sync_checkpoints ( + checkpoint_id TEXT PRIMARY KEY, + ticket_id TEXT NOT NULL, + json_value TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS ticket_sync_conflicts ( + conflict_id TEXT PRIMARY KEY, + ticket_id TEXT NOT NULL, + json_value TEXT NOT NULL + ); + CREATE INDEX IF NOT EXISTS journal_events_ticket_id_idx + ON journal_events (ticket_id, ts, order_key); + CREATE INDEX IF NOT EXISTS ticket_comments_ticket_id_idx + ON ticket_comments (ticket_id); + CREATE INDEX IF NOT EXISTS ticket_review_notes_ticket_id_idx + ON ticket_review_notes (ticket_id); + CREATE INDEX IF NOT EXISTS ticket_sync_checkpoints_ticket_id_idx + ON ticket_sync_checkpoints (ticket_id); + CREATE INDEX IF NOT EXISTS ticket_sync_conflicts_ticket_id_idx + ON ticket_sync_conflicts (ticket_id); + `); +} + +function parseProjectionJson(input: { readonly json: string }): T | null { + try { + return JSON.parse(input.json) as T; + } catch { + return null; + } +} + +function readEventRows(input: { + readonly db: Database; +}): Map { + const rows = input.db + .query( + "SELECT ticket_id, json_value, ts, order_key FROM journal_events ORDER BY ts, order_key, event_id" + ) + .all() as ProjectionEventRow[]; + + const grouped = new Map(); + for (const row of rows) { + const event = parseProjectionJson({ json: row.json_value }); + if (!event) { + continue; + } + const list = grouped.get(row.ticket_id) ?? []; + list.push(event); + grouped.set(row.ticket_id, list); + } + return grouped; +} + +function readRowsByTicket(input: { + readonly db: Database; + readonly table: + | "ticket_comments" + | "ticket_review_notes" + | "ticket_sync_checkpoints" + | "ticket_sync_conflicts"; +}): Map { + const rows = input.db + .query( + `SELECT ticket_id, json_value FROM ${input.table} ORDER BY ticket_id, rowid` + ) + .all() as ProjectionJsonRow[]; + + const grouped = new Map(); + for (const row of rows) { + const value = parseProjectionJson({ json: row.json_value }); + if (!value) { + continue; + } + const list = grouped.get(row.ticket_id) ?? []; + list.push(value); + grouped.set(row.ticket_id, list); + } + return grouped; +} + +function writeMapEntries(input: { + readonly map: ReadonlyMap; + readonly write: (value: T) => void; +}): void { + for (const values of input.map.values()) { + for (const value of values) { + input.write(value); + } + } +} diff --git a/src/control-plane/extensions/tickets/store.ts b/src/control-plane/extensions/tickets/store.ts index aa855a129..ef2851212 100644 --- a/src/control-plane/extensions/tickets/store.ts +++ b/src/control-plane/extensions/tickets/store.ts @@ -9,6 +9,7 @@ import { createNormalizedTicket, projectNormalizedTicketSummary, } from "./domain.ts"; +import { createTicketsSqliteProjection } from "./sqlite-projection.ts"; import { createGitTicketsChannel } from "./tickets-git-channel.ts"; import { formatTicketId, @@ -170,6 +171,11 @@ type MaterializedTicketState = { readonly conflictsByTicket: Map; }; +type TicketStoreContext = { + readonly events: readonly TicketEvent[]; + readonly snapshot: TicketStoreSnapshot; +}; + function normalizeTicketSummaryCompatibility(input: { readonly ticket: TicketSummary; }): TicketSummary { @@ -323,6 +329,9 @@ export function createTicketsStore(opts: { config: opts.controlPlaneConfig.tickets.git, logger: opts.logger, }); + const projection = createTicketsSqliteProjection({ + projectRoot: opts.projectRoot, + }); let eventSequence = 0; const resolveActor = (override?: string): string => { @@ -374,9 +383,10 @@ export function createTicketsStore(opts: { }; }; - const readAllEvents = async (): Promise => { - const root = await git.ensureCheckedOut(); - const eventsDir = resolve(root, ".hack/tickets/events"); + const readAllEventsFromRoot = async (input: { + readonly root: string; + }): Promise => { + const eventsDir = resolve(input.root, ".hack/tickets/events"); let entries: string[] = []; try { @@ -432,11 +442,6 @@ export function createTicketsStore(opts: { return events; }; - const materializeTickets = async (): Promise> => { - const events = await readAllEvents(); - return materializeSnapshotFromEvents({ events }).tickets; - }; - const materializeSnapshotFromEvents = (input: { readonly events: readonly TicketEvent[]; }): MaterializedTicketState => { @@ -490,12 +495,101 @@ export function createTicketsStore(opts: { return out; }; + const buildStoreSnapshot = (input: { + readonly events: readonly TicketEvent[]; + readonly materialized: MaterializedTicketState; + }): TicketStoreSnapshot => { + return { + tickets: sortTickets({ tickets: input.materialized.tickets.values() }), + eventsByTicket: groupEventsByTicket({ events: input.events }), + commentsByTicket: input.materialized.commentsByTicket, + reviewNotesByTicket: input.materialized.reviewNotesByTicket, + syncCheckpointsByTicket: input.materialized.syncCheckpointsByTicket, + conflictsByTicket: input.materialized.conflictsByTicket, + }; + }; + + const rebuildProjectionFromJournal = async (): Promise< + | { readonly ok: true; readonly context: TicketStoreContext } + | { + readonly ok: false; + readonly error: string; + } + > => { + try { + const root = await git.ensureCheckedOut(); + const events = await readAllEventsFromRoot({ root }); + const materialized = materializeSnapshotFromEvents({ events }); + const snapshot = buildStoreSnapshot({ events, materialized }); + const journalSignature = await projection.computeJournalSignature({ + ticketsRoot: root, + }); + await projection.replaceSnapshot({ + journalSignature, + snapshot, + }); + return { + ok: true, + context: { + events, + snapshot, + }, + }; + } catch (error: unknown) { + const message = + error instanceof Error + ? error.message + : "Failed to rebuild tickets sqlite projection"; + return { + ok: false, + error: message, + }; + } + }; + + const loadStoreContext = async (): Promise => { + const root = await git.ensureCheckedOut(); + const journalSignature = await projection.computeJournalSignature({ + ticketsRoot: root, + }); + const persisted = await projection.readSnapshot({ + journalSignature, + }); + if (persisted) { + const events = [...persisted.eventsByTicket.values()].flat(); + return { + events, + snapshot: persisted, + }; + } + + const rebuilt = await rebuildProjectionFromJournal(); + if (rebuilt.ok) { + return rebuilt.context; + } + + const events = await readAllEventsFromRoot({ root }); + const materialized = materializeSnapshotFromEvents({ events }); + return { + events, + snapshot: buildStoreSnapshot({ events, materialized }), + }; + }; + + const materializeTickets = async (): Promise> => { + const context = await loadStoreContext(); + return new Map( + context.snapshot.tickets.map( + (ticket) => [ticket.ticketId, ticket] as const + ) + ); + }; + const computeNextTicketId = async (): Promise => { - const events = await readAllEvents(); - const snapshot = materializeSnapshotFromEvents({ events }); + const { snapshot } = await loadStoreContext(); let max = 0; - for (const ticketId of snapshot.tickets.keys()) { - const n = parseTicketNumber(ticketId); + for (const ticket of snapshot.tickets) { + const n = parseTicketNumber(ticket.ticketId); if (n !== null && n > max) { max = n; } @@ -523,7 +617,35 @@ export function createTicketsStore(opts: { actor: input.actor, }); - return await git.appendEvents({ events: [event] }); + const wrote = await git.appendEvents({ events: [event] }); + if (!wrote.ok) { + return wrote; + } + + const rebuilt = await rebuildProjectionFromJournal(); + if (!rebuilt.ok) { + return rebuilt; + } + + return { ok: true }; + }; + + const appendEventsAndRefresh = async (input: { + readonly events: readonly TicketEvent[]; + }): Promise< + { readonly ok: true } | { readonly ok: false; readonly error: string } + > => { + const wrote = await git.appendEvents({ events: input.events }); + if (!wrote.ok) { + return wrote; + } + + const rebuilt = await rebuildProjectionFromJournal(); + if (!rebuilt.ok) { + return rebuilt; + } + + return { ok: true }; }; return { @@ -588,7 +710,7 @@ export function createTicketsStore(opts: { actor: input.actor, }); - const wrote = await git.appendEvents({ events: [event] }); + const wrote = await appendEventsAndRefresh({ events: [event] }); if (!wrote.ok) { return wrote; } @@ -715,32 +837,33 @@ export function createTicketsStore(opts: { actor: input.actor, }); - return await git.appendEvents({ events: [event] }); + return await appendEventsAndRefresh({ events: [event] }); }, listTickets: async () => { - const events = await readAllEvents(); - const snapshot = materializeSnapshotFromEvents({ events }); - return sortTickets({ tickets: snapshot.tickets.values() }); + const { snapshot } = await loadStoreContext(); + return snapshot.tickets; }, getTicket: async ({ ticketId }) => { - const events = await readAllEvents(); - const snapshot = materializeSnapshotFromEvents({ events }); - return snapshot.tickets.get(ticketId) ?? null; + const { snapshot } = await loadStoreContext(); + return ( + snapshot.tickets.find((ticket) => ticket.ticketId === ticketId) ?? null + ); }, listEvents: async ({ ticketId }) => { - const events = await readAllEvents(); - return events.filter((event) => event.ticketId === ticketId); + const { snapshot } = await loadStoreContext(); + return snapshot.eventsByTicket.get(ticketId) ?? []; }, getTicketDetail: async ({ ticketId }) => { - const events = await readAllEvents(); - const snapshot = materializeSnapshotFromEvents({ events }); + const { snapshot } = await loadStoreContext(); return { - ticket: snapshot.tickets.get(ticketId) ?? null, - events: events.filter((event) => event.ticketId === ticketId), + ticket: + snapshot.tickets.find((ticket) => ticket.ticketId === ticketId) ?? + null, + events: snapshot.eventsByTicket.get(ticketId) ?? [], comments: snapshot.commentsByTicket.get(ticketId) ?? [], reviewNotes: snapshot.reviewNotesByTicket.get(ticketId) ?? [], syncCheckpoints: snapshot.syncCheckpointsByTicket.get(ticketId) ?? [], @@ -784,7 +907,7 @@ export function createTicketsStore(opts: { actor: input.actor, }); - const wrote = await git.appendEvents({ events: [event] }); + const wrote = await appendEventsAndRefresh({ events: [event] }); if (!wrote.ok) { return wrote; } @@ -830,7 +953,7 @@ export function createTicketsStore(opts: { actor: input.actor, }); - const wrote = await git.appendEvents({ events: [event] }); + const wrote = await appendEventsAndRefresh({ events: [event] }); if (!wrote.ok) { return wrote; } @@ -849,8 +972,7 @@ export function createTicketsStore(opts: { }, linkCommentExternalId: async (input) => { - const detail = await readAllEvents(); - const snapshot = materializeSnapshotFromEvents({ events: detail }); + const { snapshot } = await loadStoreContext(); const comments = snapshot.commentsByTicket.get(input.ticketId) ?? []; const current = comments.find( (comment) => comment.commentId === input.commentId @@ -879,7 +1001,7 @@ export function createTicketsStore(opts: { actor: input.actor, }); - return await git.appendEvents({ events: [event] }); + return await appendEventsAndRefresh({ events: [event] }); }, recordSyncCheckpoint: async (input) => { @@ -926,7 +1048,7 @@ export function createTicketsStore(opts: { actor: input.actor, }); - const wrote = await git.appendEvents({ events: [event] }); + const wrote = await appendEventsAndRefresh({ events: [event] }); if (!wrote.ok) { return wrote; } @@ -993,7 +1115,7 @@ export function createTicketsStore(opts: { actor: input.actor, }); - const wrote = await git.appendEvents({ events: [event] }); + const wrote = await appendEventsAndRefresh({ events: [event] }); if (!wrote.ok) { return wrote; } @@ -1021,8 +1143,7 @@ export function createTicketsStore(opts: { }, resolveSyncConflict: async (input) => { - const detail = await readAllEvents(); - const snapshot = materializeSnapshotFromEvents({ events: detail }); + const { snapshot } = await loadStoreContext(); const conflicts = snapshot.conflictsByTicket.get(input.ticketId) ?? []; const current = conflicts.find( (conflict) => conflict.conflictId === input.conflictId @@ -1045,20 +1166,12 @@ export function createTicketsStore(opts: { actor: input.actor, }); - return await git.appendEvents({ events: [event] }); + return await appendEventsAndRefresh({ events: [event] }); }, readSnapshot: async () => { - const events = await readAllEvents(); - const snapshot = materializeSnapshotFromEvents({ events }); - return { - tickets: sortTickets({ tickets: snapshot.tickets.values() }), - eventsByTicket: groupEventsByTicket({ events }), - commentsByTicket: snapshot.commentsByTicket, - reviewNotesByTicket: snapshot.reviewNotesByTicket, - syncCheckpointsByTicket: snapshot.syncCheckpointsByTicket, - conflictsByTicket: snapshot.conflictsByTicket, - }; + const { snapshot } = await loadStoreContext(); + return snapshot; }, sync: async () => { diff --git a/tests/tickets-extension.test.ts b/tests/tickets-extension.test.ts index dc336da44..e85e863cf 100644 --- a/tests/tickets-extension.test.ts +++ b/tests/tickets-extension.test.ts @@ -260,6 +260,58 @@ testIntegration( } ); +testIntegration( + "tickets extension: cli rebuilds sqlite projection after local deletion", + { timeout: 60_000 }, + async () => { + const root = await mkdirTempDir({ + prefix: "hack-cli-tickets-projection-e2e-", + }); + const projectDir = join(root, "project"); + + await mkdir(projectDir, { recursive: true }); + await copyDir({ + from: resolve(import.meta.dir, "../examples/tickets"), + to: projectDir, + }); + + await run({ cwd: projectDir, cmd: ["git", "init"] }); + await run({ + cwd: projectDir, + cmd: ["git", "config", "user.email", "tests@hack"], + }); + await run({ + cwd: projectDir, + cmd: ["git", "config", "user.name", "hack-cli-tests"], + }); + await run({ cwd: projectDir, cmd: ["git", "add", "-A"] }); + await run({ cwd: projectDir, cmd: ["git", "commit", "-m", "init"] }); + + const created = await runHack({ + cwd: projectDir, + args: ["tickets", "create", "--title", "Projection lifecycle", "--json"], + }); + const createdJson = JSON.parse(created.stdout) as { + ticket: { ticketId: string }; + }; + + const projectionPath = join(projectDir, ".hack/tickets/projection.sqlite"); + expect(await Bun.file(projectionPath).exists()).toBe(true); + + await rm(projectionPath, { force: true }); + expect(await Bun.file(projectionPath).exists()).toBe(false); + + const shown = await runHack({ + cwd: projectDir, + args: ["tickets", "show", createdJson.ticket.ticketId, "--json"], + }); + expect(shown.exitCode).toBe(0); + expect(await Bun.file(projectionPath).exists()).toBe(true); + + await rm(root, { recursive: true, force: true }); + } +); + interface RunResult { readonly stdout: string; readonly stderr: string; diff --git a/tests/tickets-store.test.ts b/tests/tickets-store.test.ts index b5f94d7a1..5250b78c0 100644 --- a/tests/tickets-store.test.ts +++ b/tests/tickets-store.test.ts @@ -446,6 +446,48 @@ test("tickets store writes normalized journal envelope metadata and ignores dupl }); }, 20_000); +test("tickets store persists a sqlite projection and rebuilds it when deleted", async () => { + const projectRoot = await createTempGitProject({ + prefix: "hack-cli-tickets-projection-", + }); + const projectionPath = join(projectRoot, ".hack/tickets/projection.sqlite"); + + const firstStore = await createStore({ projectRoot }); + const created = await firstStore.createTicket({ + title: "Projection ticket", + body: "Persisted through sqlite projection.", + owner: "hack", + source: "hack", + actor: "creator@hack", + }); + expect(created.ok).toBe(true); + if (!created.ok) { + throw new Error(created.error); + } + + const initialTickets = await firstStore.listTickets(); + expect(initialTickets.map((ticket) => ticket.title)).toContain( + "Projection ticket" + ); + expect(await Bun.file(projectionPath).exists()).toBe(true); + + const secondStore = await createStore({ projectRoot }); + const persistedTickets = await secondStore.listTickets(); + expect(persistedTickets.map((ticket) => ticket.title)).toContain( + "Projection ticket" + ); + + await rm(projectionPath, { force: true }); + expect(await Bun.file(projectionPath).exists()).toBe(false); + + const rebuiltStore = await createStore({ projectRoot }); + const rebuiltTickets = await rebuiltStore.listTickets(); + expect(rebuiltTickets.map((ticket) => ticket.title)).toContain( + "Projection ticket" + ); + expect(await Bun.file(projectionPath).exists()).toBe(true); +}, 20_000); + test("normalized ticket adapter preserves compatibility while exposing provenance and documents", () => { const summary = { ticketId: "T-00042", From e6c8cb8389e9da23b289af7e366f778cc4fa3f7f Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 22:52:04 +0000 Subject: [PATCH 008/134] docs(tickets): document sqlite projection cache --- docs/guides/tickets.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/guides/tickets.md b/docs/guides/tickets.md index 279f60a02..c70da16d6 100644 --- a/docs/guides/tickets.md +++ b/docs/guides/tickets.md @@ -126,7 +126,9 @@ Tip: use `--body-stdin` for multi-line markdown. ## How it works - Ticket history is an append-only event log (`ticket.created`, etc.) stored as monthly JSONL files. -- The extension reads events, materializes tickets in-memory, and renders `list/show` outputs. +- Each event carries a normalized journal envelope (`eventId`, schema version, occurrence/recording times, source metadata, and idempotency key). +- The journal is the portable source of truth for tickets. +- The extension projects journal state into a local SQLite cache for durable reads and rebuilds that cache automatically when it is missing or stale. - Ticket writes automatically commit and push to the tickets ref when git sync is enabled and a remote exists. - `sync` normalizes the event logs, commits, and pushes the tickets ref. @@ -135,9 +137,16 @@ Tip: use `--body-stdin` for multi-line markdown. In your project repo: - `.hack/tickets/events/events-YYYY-MM.jsonl` — event log segments (UTC month) +- `.hack/tickets/projection.sqlite` — local SQLite projection cache rebuilt from the journal - `.hack/tickets/git/bare.git` — a bare repo used to manage the tickets ref - `.hack/tickets/git/worktree` — a worktree used for reading/writing ticket data +Portability rules: + +- Only the journal under `.hack/tickets/events/` is portable ticket state. +- The SQLite projection is local-only and can be deleted safely. +- After sync or clone, peers rebuild `.hack/tickets/projection.sqlite` from the journal on first read. + ## Configuration Tickets git configuration lives under `controlPlane.tickets.git`. From 77e2c91d99ef29a7fcc6ad1cdc1dcc25070116c7 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 23:03:40 +0000 Subject: [PATCH 009/134] feat(tickets): normalize provenance helpers --- .../extensions/linear/commands.ts | 114 +++-- .../extensions/tickets/domain.ts | 170 +++---- .../extensions/tickets/provenance.ts | 457 ++++++++++++++++++ tests/linear-commands.test.ts | 147 ++++++ tests/tickets-store.test.ts | 203 +++++++- 5 files changed, 926 insertions(+), 165 deletions(-) create mode 100644 src/control-plane/extensions/tickets/provenance.ts diff --git a/src/control-plane/extensions/linear/commands.ts b/src/control-plane/extensions/linear/commands.ts index ebaf9b342..26bd0c3b6 100644 --- a/src/control-plane/extensions/linear/commands.ts +++ b/src/control-plane/extensions/linear/commands.ts @@ -11,6 +11,10 @@ import { import { isRecord } from "../../../lib/guards.ts"; import { openUrl } from "../../../lib/os.ts"; import { display } from "../../../ui/display.ts"; +import { + findTicketRemoteLink, + projectRemoteLinkToCompatibilityFields, +} from "../tickets/provenance.ts"; import { createTicketsStore, type TicketComment, @@ -3780,13 +3784,17 @@ async function syncTicketToLinearIssue(input: { const updated = await input.runtime.tickets.updateTicket({ ticketId: ticket.ticketId, - externalSystem: "linear", - externalId: syncedIssue.data.id, - externalKey: syncedIssue.data.identifier, - externalUrl: syncedIssue.data.url, - externalProjectId: syncedIssue.data.projectId, - externalProjectName: syncedIssue.data.projectName, - externalTeamId: syncedIssue.data.teamId, + ...projectRemoteLinkToCompatibilityFields({ + remote: { + provider: "linear", + remoteId: syncedIssue.data.id, + remoteKey: syncedIssue.data.identifier, + remoteUrl: syncedIssue.data.url, + projectId: syncedIssue.data.projectId, + projectName: syncedIssue.data.projectName, + teamId: syncedIssue.data.teamId, + }, + }), }); if (!updated.ok) { return { ok: false, error: updated.error }; @@ -3841,15 +3849,19 @@ async function resolveLinearTargetForTicketSync(input: { } | { readonly ok: false; readonly error: string } > { + const linearRemote = findTicketRemoteLink({ + ticket: input.ticket, + provider: "linear", + }); let projectId = input.explicitProjectId ?? input.existingIssue?.projectId ?? - input.ticket.externalProjectId ?? + linearRemote?.projectId ?? input.runtime.projectBinding.projectId; let teamId = input.explicitTeamId ?? input.existingIssue?.teamId ?? - input.ticket.externalTeamId ?? + linearRemote?.teamId ?? input.runtime.projectBinding.teamId; if (!(teamId || !projectId)) { @@ -4743,13 +4755,17 @@ async function applyLinearIssueToExistingTicket(input: { ...(input.projection.dependsOn !== undefined ? { dependsOn: input.projection.dependsOn } : {}), - externalSystem: "linear", - externalId: input.issue.id, - externalKey: input.issue.identifier, - externalUrl: input.issue.url, - externalProjectId: input.issue.projectId, - externalProjectName: input.issue.projectName, - externalTeamId: input.issue.teamId, + ...projectRemoteLinkToCompatibilityFields({ + remote: { + provider: "linear", + remoteId: input.issue.id, + remoteKey: input.issue.identifier, + remoteUrl: input.issue.url, + projectId: input.issue.projectId, + projectName: input.issue.projectName, + teamId: input.issue.teamId, + }, + }), }); if (!updated.ok) { return { ok: false, error: updated.error }; @@ -4817,13 +4833,17 @@ async function createTicketFromLinearIssueProjection(input: { ...(input.projection.dependsOn !== undefined ? { dependsOn: input.projection.dependsOn } : {}), - externalSystem: "linear", - externalId: input.issue.id, - externalKey: input.issue.identifier, - externalUrl: input.issue.url, - externalProjectId: input.issue.projectId, - externalProjectName: input.issue.projectName, - externalTeamId: input.issue.teamId, + ...projectRemoteLinkToCompatibilityFields({ + remote: { + provider: "linear", + remoteId: input.issue.id, + remoteKey: input.issue.identifier, + remoteUrl: input.issue.url, + projectId: input.issue.projectId, + projectName: input.issue.projectName, + teamId: input.issue.teamId, + }, + }), }); if (!created.ok) { return created; @@ -4880,14 +4900,18 @@ async function buildLinearDependencyIndex(input: { const byLinearId = new Map(); const byLinearIdentifier = new Map(); for (const ticket of tickets) { - if (ticket.externalSystem !== "linear") { + const linearRemote = findTicketRemoteLink({ + ticket, + provider: "linear", + }); + if (!linearRemote) { continue; } - if (ticket.externalId) { - byLinearId.set(ticket.externalId, ticket.ticketId); + if (linearRemote.remoteId) { + byLinearId.set(linearRemote.remoteId, ticket.ticketId); } - if (ticket.externalKey) { - byLinearIdentifier.set(ticket.externalKey, ticket.ticketId); + if (linearRemote.remoteKey) { + byLinearIdentifier.set(linearRemote.remoteKey, ticket.ticketId); } } return { @@ -4908,12 +4932,16 @@ async function resolveLinkedLinearIssue(input: { | { readonly ok: true; readonly issue: LinearIssue | null } | { readonly ok: false; readonly error: string } > { - if (input.ticket.externalSystem !== "linear") { + const linearRemote = findTicketRemoteLink({ + ticket: input.ticket, + provider: "linear", + }); + if (!linearRemote) { return { ok: true, issue: null }; } - if (input.ticket.externalId) { + if (linearRemote.remoteId) { const byId = await input.runtime.linear.getIssueById({ - issueId: input.ticket.externalId, + issueId: linearRemote.remoteId, }); if (!byId.ok) { return { ok: false, error: byId.error }; @@ -4922,9 +4950,9 @@ async function resolveLinkedLinearIssue(input: { return { ok: true, issue: byId.data }; } } - if (input.ticket.externalKey) { + if (linearRemote.remoteKey) { const byKey = await input.runtime.linear.getIssueByIdentifier({ - identifier: input.ticket.externalKey, + identifier: linearRemote.remoteKey, }); if (!byKey.ok) { return { ok: false, error: byKey.error }; @@ -5029,8 +5057,12 @@ async function resolveLinearParentIssueId(input: { if (!parentTicket) { return { ok: true }; } - if (parentTicket.externalSystem === "linear" && parentTicket.externalId) { - return { ok: true, value: parentTicket.externalId }; + const linearRemote = findTicketRemoteLink({ + ticket: parentTicket, + provider: "linear", + }); + if (linearRemote?.remoteId) { + return { ok: true, value: linearRemote.remoteId }; } return { ok: true }; } @@ -5058,15 +5090,19 @@ function isLinearLinkedTicket(input: { readonly ticket: TicketSummary; readonly issue: LinearIssue; }): boolean { - if (input.ticket.externalSystem !== "linear") { + const linearRemote = findTicketRemoteLink({ + ticket: input.ticket, + provider: "linear", + }); + if (!linearRemote) { return false; } - if (input.ticket.externalId && input.ticket.externalId === input.issue.id) { + if (linearRemote.remoteId && linearRemote.remoteId === input.issue.id) { return true; } if ( - input.ticket.externalKey && - input.ticket.externalKey === input.issue.identifier + linearRemote.remoteKey && + linearRemote.remoteKey === input.issue.identifier ) { return true; } diff --git a/src/control-plane/extensions/tickets/domain.ts b/src/control-plane/extensions/tickets/domain.ts index 4e11cd67e..953b9daee 100644 --- a/src/control-plane/extensions/tickets/domain.ts +++ b/src/control-plane/extensions/tickets/domain.ts @@ -1,3 +1,7 @@ +import { + buildTicketProvenance, + projectRemoteLinkToCompatibilityFields, +} from "./provenance.ts"; import { sha256Hex } from "./util.ts"; export type TicketStatus = "open" | "in_progress" | "blocked" | "done"; @@ -82,9 +86,32 @@ export type TicketRemoteLink = { readonly remoteId?: string; readonly remoteKey?: string; readonly remoteUrl?: string; + readonly profileId?: string; readonly projectId?: string; readonly projectName?: string; readonly teamId?: string; + readonly remoteCursor?: string; + readonly remoteUpdatedAt?: string; +}; + +export type TicketFieldAuthority = + | "local" + | "remote" + | "append_only" + | "derived" + | "review_required"; + +export type TicketFieldAuthorityEntry = { + readonly field: string; + readonly authority: TicketFieldAuthority; +}; + +export type TicketFieldVersion = { + readonly field: string; + readonly source: "local" | "remote"; + readonly provider?: string; + readonly recordedAt: string; + readonly value?: TicketMetadataValue; }; export type TicketDocumentKind = "description" | "spec" | "notes"; @@ -120,6 +147,8 @@ export type NormalizedTicket = { readonly provenance: { readonly origin: TicketOrigin; readonly remotes: readonly TicketRemoteLink[]; + readonly fieldAuthorities: readonly TicketFieldAuthorityEntry[]; + readonly fieldVersions: readonly TicketFieldVersion[]; }; readonly documents: readonly TicketDocument[]; readonly fieldStates: readonly TicketFieldState[]; @@ -134,14 +163,14 @@ export function createNormalizedTicket(input: { readonly syncCheckpoints?: readonly TicketSyncCheckpointCompatibility[]; readonly conflicts?: readonly TicketSyncConflictCompatibility[]; }): NormalizedTicket { - const sourceSystem = inferSourceSystem({ ticket: input.ticket }); const documents = buildDocuments({ ticket: input.ticket }); - const remotes = buildRemoteLinks({ - ticket: input.ticket, - sourceSystem, - }); const checkpoints = input.syncCheckpoints ?? []; const conflicts = input.conflicts ?? []; + const provenance = buildTicketProvenance({ + ticket: input.ticket, + syncCheckpoints: checkpoints, + conflicts, + }); return { identity: { @@ -159,16 +188,12 @@ export function createNormalizedTicket(input: { blocks: [...input.ticket.blocks], ...(input.ticket.assignee ? { assignee: input.ticket.assignee } : {}), tags: [...input.ticket.tags], - provenance: { - origin: { - owner: input.ticket.owner, - source: input.ticket.source, - system: sourceSystem, - }, - remotes, - }, + provenance, documents, - fieldStates: buildFieldStates({ remotes, conflicts }), + fieldStates: buildFieldStates({ + fieldAuthorities: provenance.fieldAuthorities, + conflicts, + }), sync: { checkpoints: [...checkpoints], conflicts: [...conflicts], @@ -198,27 +223,9 @@ export function projectNormalizedTicketSummary(input: { ...(input.ticket.assignee ? { assignee: input.ticket.assignee } : {}), tags: [...input.ticket.tags], ...(primaryRemote - ? { - externalSystem: primaryRemote.provider, - ...(primaryRemote.remoteId - ? { externalId: primaryRemote.remoteId } - : {}), - ...(primaryRemote.remoteKey - ? { externalKey: primaryRemote.remoteKey } - : {}), - ...(primaryRemote.remoteUrl - ? { externalUrl: primaryRemote.remoteUrl } - : {}), - ...(primaryRemote.projectId - ? { externalProjectId: primaryRemote.projectId } - : {}), - ...(primaryRemote.projectName - ? { externalProjectName: primaryRemote.projectName } - : {}), - ...(primaryRemote.teamId - ? { externalTeamId: primaryRemote.teamId } - : {}), - } + ? projectRemoteLinkToCompatibilityFields({ + remote: primaryRemote, + }) : {}), ...(input.ticket.identity.projectId ? { projectId: input.ticket.identity.projectId } @@ -229,21 +236,6 @@ export function projectNormalizedTicketSummary(input: { }; } -function inferSourceSystem(input: { - readonly ticket: TicketSummaryCompatibility; -}): string { - if (input.ticket.externalSystem) { - return input.ticket.externalSystem; - } - if (input.ticket.source !== "hack") { - return input.ticket.source; - } - if (input.ticket.owner !== "hack") { - return input.ticket.owner; - } - return "hack"; -} - function buildDocuments(input: { readonly ticket: TicketSummaryCompatibility; }): TicketDocument[] { @@ -266,71 +258,21 @@ function buildDocuments(input: { ]; } -function buildRemoteLinks(input: { - readonly ticket: TicketSummaryCompatibility; - readonly sourceSystem: string; -}): TicketRemoteLink[] { - if ( - !( - input.ticket.externalSystem || - input.ticket.externalId || - input.ticket.externalKey || - input.ticket.externalUrl || - input.ticket.externalProjectId || - input.ticket.externalProjectName || - input.ticket.externalTeamId - ) - ) { - return []; - } - - return [ - { - provider: input.ticket.externalSystem ?? input.sourceSystem, - ...(input.ticket.externalId ? { remoteId: input.ticket.externalId } : {}), - ...(input.ticket.externalKey - ? { remoteKey: input.ticket.externalKey } - : {}), - ...(input.ticket.externalUrl - ? { remoteUrl: input.ticket.externalUrl } - : {}), - ...(input.ticket.externalProjectId - ? { projectId: input.ticket.externalProjectId } - : {}), - ...(input.ticket.externalProjectName - ? { projectName: input.ticket.externalProjectName } - : {}), - ...(input.ticket.externalTeamId - ? { teamId: input.ticket.externalTeamId } - : {}), - }, - ]; -} - function buildFieldStates(input: { - readonly remotes: readonly TicketRemoteLink[]; + readonly fieldAuthorities: readonly TicketFieldAuthorityEntry[]; readonly conflicts: readonly TicketSyncConflictCompatibility[]; }): TicketFieldState[] { - const defaultAuthority = input.remotes.length > 0 ? "remote" : "local"; - const byField = new Map(); - - for (const field of ["title", "status", "assignee", "description"]) { - byField.set(field, { - field, - authority: defaultAuthority, - conflictIds: [], - }); - } - - for (const conflict of input.conflicts) { - const current = byField.get(conflict.field); - const conflictIds = [...(current?.conflictIds ?? []), conflict.conflictId]; - byField.set(conflict.field, { - field: conflict.field, - authority: conflict.authority ?? current?.authority ?? defaultAuthority, - conflictIds, - }); - } - - return [...byField.values()]; + return input.fieldAuthorities + .filter((fieldAuthority) => + ["title", "status", "assignee", "description"].includes( + fieldAuthority.field + ) + ) + .map((fieldAuthority) => ({ + field: fieldAuthority.field, + authority: fieldAuthority.authority, + conflictIds: input.conflicts + .filter((conflict) => conflict.field === fieldAuthority.field) + .map((conflict) => conflict.conflictId), + })); } diff --git a/src/control-plane/extensions/tickets/provenance.ts b/src/control-plane/extensions/tickets/provenance.ts new file mode 100644 index 000000000..2407ab100 --- /dev/null +++ b/src/control-plane/extensions/tickets/provenance.ts @@ -0,0 +1,457 @@ +export type TicketMetadataValue = + | string + | number + | boolean + | null + | readonly TicketMetadataValue[] + | { readonly [key: string]: TicketMetadataValue }; + +export type TicketSyncCheckpointCompatibility = { + readonly checkpointId: string; + readonly ticketId: string; + readonly provider: string; + readonly profileId?: string; + readonly direction?: string; + readonly remoteCursor?: string; + readonly remoteUpdatedAt?: string; + readonly localUpdatedAt?: string; + readonly actor: string; + readonly createdAt: string; +}; + +export type TicketSyncConflictCompatibility = { + readonly conflictId: string; + readonly ticketId: string; + readonly provider: string; + readonly field: string; + readonly status: "open" | "resolved"; + readonly authority?: string; + readonly summary?: string; + readonly localValue?: TicketMetadataValue; + readonly remoteValue?: TicketMetadataValue; + readonly createdAt: string; + readonly updatedAt: string; + readonly resolution?: "accept_local" | "accept_remote" | "merged" | "ignore"; + readonly resolutionSummary?: string; + readonly resolvedAt?: string; + readonly resolvedBy?: string; +}; + +export type TicketProvenanceCompatibility = { + readonly owner: string; + readonly source: string; + readonly updatedAt: string; + readonly title: string; + readonly status: string; + readonly body?: string; + readonly assignee?: string; + readonly externalSystem?: string; + readonly externalId?: string; + readonly externalKey?: string; + readonly externalUrl?: string; + readonly externalProjectId?: string; + readonly externalProjectName?: string; + readonly externalTeamId?: string; +}; + +export type TicketOrigin = { + readonly owner: string; + readonly source: string; + readonly system: string; +}; + +export type TicketRemoteLink = { + readonly provider: string; + readonly remoteId?: string; + readonly remoteKey?: string; + readonly remoteUrl?: string; + readonly profileId?: string; + readonly projectId?: string; + readonly projectName?: string; + readonly teamId?: string; + readonly remoteCursor?: string; + readonly remoteUpdatedAt?: string; +}; + +export type TicketFieldAuthority = + | "local" + | "remote" + | "append_only" + | "derived" + | "review_required"; + +export type TicketFieldAuthorityEntry = { + readonly field: string; + readonly authority: TicketFieldAuthority; +}; + +export type TicketFieldVersion = { + readonly field: string; + readonly source: "local" | "remote"; + readonly provider?: string; + readonly recordedAt: string; + readonly value?: TicketMetadataValue; +}; + +export function inferTicketSourceSystem(input: { + readonly ticket: Pick< + TicketProvenanceCompatibility, + "owner" | "source" | "externalSystem" + >; +}): string { + if (input.ticket.externalSystem) { + return input.ticket.externalSystem; + } + if (input.ticket.source !== "hack") { + return input.ticket.source; + } + if (input.ticket.owner !== "hack") { + return input.ticket.owner; + } + return "hack"; +} + +export function buildTicketRemoteLinks(input: { + readonly ticket: Pick< + TicketProvenanceCompatibility, + | "owner" + | "source" + | "externalSystem" + | "externalId" + | "externalKey" + | "externalUrl" + | "externalProjectId" + | "externalProjectName" + | "externalTeamId" + >; + readonly syncCheckpoints?: readonly TicketSyncCheckpointCompatibility[]; +}): TicketRemoteLink[] { + const sourceSystem = inferTicketSourceSystem({ + ticket: input.ticket, + }); + const remotes: TicketRemoteLink[] = []; + + if (ticketHasRemoteIdentity({ ticket: input.ticket })) { + remotes.push( + createPrimaryRemoteLink({ + ticket: input.ticket, + sourceSystem, + }) + ); + } + + for (const checkpoint of input.syncCheckpoints ?? []) { + const trackedIndex = remotes.findIndex((remote) => + remoteMatchesCheckpoint({ remote, checkpoint }) + ); + + if (trackedIndex >= 0) { + remotes[trackedIndex] = mergeCheckpointRemote({ + remote: remotes[trackedIndex], + checkpoint, + }); + continue; + } + + remotes.push(createCheckpointRemoteLink({ checkpoint })); + } + + return remotes; +} + +function ticketHasRemoteIdentity(input: { + readonly ticket: Pick< + TicketProvenanceCompatibility, + | "externalSystem" + | "externalId" + | "externalKey" + | "externalUrl" + | "externalProjectId" + | "externalProjectName" + | "externalTeamId" + >; +}): boolean { + return Boolean( + input.ticket.externalSystem || + input.ticket.externalId || + input.ticket.externalKey || + input.ticket.externalUrl || + input.ticket.externalProjectId || + input.ticket.externalProjectName || + input.ticket.externalTeamId + ); +} + +function createPrimaryRemoteLink(input: { + readonly ticket: Pick< + TicketProvenanceCompatibility, + | "externalSystem" + | "externalId" + | "externalKey" + | "externalUrl" + | "externalProjectId" + | "externalProjectName" + | "externalTeamId" + >; + readonly sourceSystem: string; +}): TicketRemoteLink { + return { + provider: input.ticket.externalSystem ?? input.sourceSystem, + ...(input.ticket.externalId ? { remoteId: input.ticket.externalId } : {}), + ...(input.ticket.externalKey + ? { remoteKey: input.ticket.externalKey } + : {}), + ...(input.ticket.externalUrl + ? { remoteUrl: input.ticket.externalUrl } + : {}), + ...(input.ticket.externalProjectId + ? { projectId: input.ticket.externalProjectId } + : {}), + ...(input.ticket.externalProjectName + ? { projectName: input.ticket.externalProjectName } + : {}), + ...(input.ticket.externalTeamId + ? { teamId: input.ticket.externalTeamId } + : {}), + }; +} + +function remoteMatchesCheckpoint(input: { + readonly remote: TicketRemoteLink; + readonly checkpoint: TicketSyncCheckpointCompatibility; +}): boolean { + return ( + input.remote.provider === input.checkpoint.provider && + (input.remote.profileId === input.checkpoint.profileId || + input.remote.profileId === undefined || + input.checkpoint.profileId === undefined) + ); +} + +function mergeCheckpointRemote(input: { + readonly remote: TicketRemoteLink; + readonly checkpoint: TicketSyncCheckpointCompatibility; +}): TicketRemoteLink { + return { + ...input.remote, + ...(input.remote.profileId === undefined && input.checkpoint.profileId + ? { profileId: input.checkpoint.profileId } + : {}), + ...(input.checkpoint.remoteCursor + ? { remoteCursor: input.checkpoint.remoteCursor } + : {}), + ...(input.checkpoint.remoteUpdatedAt + ? { remoteUpdatedAt: input.checkpoint.remoteUpdatedAt } + : {}), + }; +} + +function createCheckpointRemoteLink(input: { + readonly checkpoint: TicketSyncCheckpointCompatibility; +}): TicketRemoteLink { + return { + provider: input.checkpoint.provider, + ...(input.checkpoint.profileId + ? { profileId: input.checkpoint.profileId } + : {}), + ...(input.checkpoint.remoteCursor + ? { remoteCursor: input.checkpoint.remoteCursor } + : {}), + ...(input.checkpoint.remoteUpdatedAt + ? { remoteUpdatedAt: input.checkpoint.remoteUpdatedAt } + : {}), + }; +} + +export function buildTicketFieldAuthorities(input: { + readonly remotes: readonly TicketRemoteLink[]; + readonly conflicts?: readonly TicketSyncConflictCompatibility[]; +}): TicketFieldAuthorityEntry[] { + const defaultAuthority = input.remotes.length > 0 ? "remote" : "local"; + const byField = new Map(); + + for (const field of [ + "title", + "status", + "assignee", + "description", + "project", + ]) { + byField.set(field, { + field, + authority: defaultAuthority, + }); + } + byField.set("comment", { + field: "comment", + authority: "append_only", + }); + byField.set("review_note", { + field: "review_note", + authority: "local", + }); + byField.set("sync_checkpoint", { + field: "sync_checkpoint", + authority: "derived", + }); + byField.set("sync_conflict", { + field: "sync_conflict", + authority: "derived", + }); + + for (const conflict of input.conflicts ?? []) { + const current = byField.get(conflict.field); + byField.set(conflict.field, { + field: conflict.field, + authority: conflict.authority ?? current?.authority ?? defaultAuthority, + }); + } + + return [...byField.values()]; +} + +export function buildTicketFieldVersions(input: { + readonly ticket: TicketProvenanceCompatibility; + readonly conflicts?: readonly TicketSyncConflictCompatibility[]; +}): TicketFieldVersion[] { + const versions: TicketFieldVersion[] = [ + { + field: "title", + source: "local", + recordedAt: input.ticket.updatedAt, + value: input.ticket.title, + }, + { + field: "status", + source: "local", + recordedAt: input.ticket.updatedAt, + value: input.ticket.status, + }, + { + field: "description", + source: "local", + recordedAt: input.ticket.updatedAt, + ...(input.ticket.body ? { value: input.ticket.body } : {}), + }, + ]; + + if (input.ticket.assignee) { + versions.push({ + field: "assignee", + source: "local", + recordedAt: input.ticket.updatedAt, + value: input.ticket.assignee, + }); + } + + for (const conflict of input.conflicts ?? []) { + if (conflict.localValue !== undefined) { + versions.push({ + field: conflict.field, + source: "local", + recordedAt: conflict.updatedAt, + value: conflict.localValue, + }); + } + if (conflict.remoteValue !== undefined) { + versions.push({ + field: conflict.field, + source: "remote", + provider: conflict.provider, + recordedAt: conflict.updatedAt, + value: conflict.remoteValue, + }); + } + } + + return versions; +} + +export function buildTicketProvenance(input: { + readonly ticket: TicketProvenanceCompatibility; + readonly syncCheckpoints?: readonly TicketSyncCheckpointCompatibility[]; + readonly conflicts?: readonly TicketSyncConflictCompatibility[]; +}): { + readonly origin: TicketOrigin; + readonly remotes: readonly TicketRemoteLink[]; + readonly fieldAuthorities: readonly TicketFieldAuthorityEntry[]; + readonly fieldVersions: readonly TicketFieldVersion[]; +} { + const system = inferTicketSourceSystem({ + ticket: input.ticket, + }); + const remotes = buildTicketRemoteLinks({ + ticket: input.ticket, + syncCheckpoints: input.syncCheckpoints, + }); + + return { + origin: { + owner: input.ticket.owner, + source: input.ticket.source, + system, + }, + remotes, + fieldAuthorities: buildTicketFieldAuthorities({ + remotes, + conflicts: input.conflicts, + }), + fieldVersions: buildTicketFieldVersions({ + ticket: input.ticket, + conflicts: input.conflicts, + }), + }; +} + +export function findTicketRemoteLink(input: { + readonly ticket: Pick< + TicketProvenanceCompatibility, + | "owner" + | "source" + | "updatedAt" + | "title" + | "status" + | "externalSystem" + | "externalId" + | "externalKey" + | "externalUrl" + | "externalProjectId" + | "externalProjectName" + | "externalTeamId" + >; + readonly provider: string; + readonly syncCheckpoints?: readonly TicketSyncCheckpointCompatibility[]; +}): TicketRemoteLink | null { + return ( + buildTicketRemoteLinks({ + ticket: input.ticket, + syncCheckpoints: input.syncCheckpoints, + }).find((remote) => remote.provider === input.provider) ?? null + ); +} + +export function projectRemoteLinkToCompatibilityFields(input: { + readonly remote: TicketRemoteLink; +}): { + readonly externalSystem: string; + readonly externalId?: string; + readonly externalKey?: string; + readonly externalUrl?: string; + readonly externalProjectId?: string; + readonly externalProjectName?: string; + readonly externalTeamId?: string; +} { + return { + externalSystem: input.remote.provider, + ...(input.remote.remoteId ? { externalId: input.remote.remoteId } : {}), + ...(input.remote.remoteKey ? { externalKey: input.remote.remoteKey } : {}), + ...(input.remote.remoteUrl ? { externalUrl: input.remote.remoteUrl } : {}), + ...(input.remote.projectId + ? { externalProjectId: input.remote.projectId } + : {}), + ...(input.remote.projectName + ? { externalProjectName: input.remote.projectName } + : {}), + ...(input.remote.teamId ? { externalTeamId: input.remote.teamId } : {}), + }; +} diff --git a/tests/linear-commands.test.ts b/tests/linear-commands.test.ts index 75eee414b..d39152f28 100644 --- a/tests/linear-commands.test.ts +++ b/tests/linear-commands.test.ts @@ -1310,6 +1310,153 @@ test("syncTicketToLinearIssue pushes missing local comments and records a checkp expect(updatedIssues[0]?.assigneeId).toBe("user-1"); }); +test("syncTicketToLinearIssue reuses a linked Linear issue inferred from provenance metadata", async () => { + const updatedIssues: Record[] = []; + + const runtime = { + profileId: "default", + apiUrl: "https://api.linear.app/graphql", + projectBinding: { + teamId: "team-1", + additionalProjects: [], + }, + assigneeMappings: [], + tickets: { + getTicket: async () => ({ + ticketId: "T-00002", + title: "Synced from Linear", + body: "Keep using the existing Linear issue.", + status: "open" as const, + createdAt: "2026-03-05T10:00:00.000Z", + updatedAt: "2026-03-05T10:00:00.000Z", + dependsOn: [], + blocks: [], + owner: "hack", + source: "linear", + tags: [], + externalId: "issue-2", + externalKey: "ENG-222", + externalUrl: "https://linear.app/hack/issue/ENG-222", + externalProjectId: "proj-2", + externalTeamId: "team-1", + }), + updateTicket: async () => ({ ok: true as const }), + listTickets: async () => [], + getTicketDetail: async () => ({ + ticket: null, + events: [], + comments: [], + reviewNotes: [], + syncCheckpoints: [], + conflicts: [], + }), + linkCommentExternalId: async () => ({ ok: true as const }), + recordSyncCheckpoint: async () => ({ + ok: true as const, + checkpoint: { + checkpointId: "checkpoint-2", + ticketId: "T-00002", + provider: "linear", + direction: "hack_to_linear", + actor: "test", + createdAt: "2026-03-05T10:10:00.000Z", + }, + }), + recordSyncConflict: async () => ({ ok: true as const, conflict: null }), + }, + linear: { + getIssueById: async () => ({ + ok: true as const, + data: { + id: "issue-2", + identifier: "ENG-222", + title: "Synced from Linear", + description: "Keep using the existing Linear issue.", + state: { + id: "state-1", + name: "Todo", + type: "unstarted" as const, + }, + url: "https://linear.app/hack/issue/ENG-222", + teamId: "team-1", + projectId: "proj-2", + labels: [], + }, + }), + getIssueByIdentifier: async () => ({ ok: true as const, data: null }), + updateIssue: async (input: Record) => { + updatedIssues.push(input); + return { + ok: true as const, + data: { + id: "issue-2", + identifier: "ENG-222", + title: "Synced from Linear", + description: "Keep using the existing Linear issue.", + state: { + id: "state-1", + name: "Todo", + type: "unstarted" as const, + }, + url: "https://linear.app/hack/issue/ENG-222", + teamId: "team-1", + projectId: "proj-2", + labels: [], + }, + }; + }, + createIssue: async () => { + throw new Error("createIssue should not be called"); + }, + listTeamStates: async () => ({ + ok: true as const, + data: [ + { + id: "state-1", + name: "Todo", + type: "unstarted" as const, + }, + ], + }), + listTeamLabels: async () => ({ + ok: true as const, + data: [], + }), + listIssueComments: async () => ({ + ok: true as const, + data: [], + }), + createComment: async () => { + throw new Error("createComment should not be called"); + }, + listTeamUsers: async () => ({ + ok: true as const, + data: [], + }), + getProject: async () => ({ ok: true as const, data: null }), + }, + }; + + const result = await __testOnly.syncTicketToLinearIssue({ + runtime, + ticketId: "T-00002", + syncToggles: { + labels: false, + statuses: true, + dependencies: false, + projects: true, + }, + }); + + expect(result.ok).toBe(true); + if (!result.ok) { + return; + } + + expect(result.operation).toBe("updated"); + expect(updatedIssues).toHaveLength(1); +}); + test("runProjectLinearAutosync syncs issue and project deliveries, then applies them", async () => { const appliedDeliveryIds: string[] = []; const issueSyncCalls: Array<{ issueIdentifier?: string; issueId?: string }> = diff --git a/tests/tickets-store.test.ts b/tests/tickets-store.test.ts index 5250b78c0..4ef4a6a63 100644 --- a/tests/tickets-store.test.ts +++ b/tests/tickets-store.test.ts @@ -13,6 +13,10 @@ import { createNormalizedTicket, projectNormalizedTicketSummary, } from "../src/control-plane/extensions/tickets/domain.ts"; +import { + buildTicketProvenance, + findTicketRemoteLink, +} from "../src/control-plane/extensions/tickets/provenance.ts"; import { createTicketsStore } from "../src/control-plane/extensions/tickets/store.ts"; import { readControlPlaneConfig } from "../src/control-plane/sdk/config.ts"; @@ -231,7 +235,7 @@ test("tickets store materializes assignee, review notes, comments, checkpoints, sourceOperation: "local_command", }); expect(events[0]?.idempotencyKey).toBe(events[0]?.eventId); -}, 20_000); +}, 30_000); test("tickets show json includes materialized sync metadata", async () => { const projectRoot = await createTempGitProject({ @@ -556,17 +560,24 @@ test("normalized ticket adapter preserves compatibility while exposing provenanc source: "linear", system: "linear", }); - expect(normalized.provenance.remotes).toEqual([ - expect.objectContaining({ - provider: "linear", - remoteId: "lin_123", - remoteKey: "HACK-431", - remoteUrl: "https://linear.app/hack/issue/HACK-431", - projectId: "project-1", - projectName: "Hack App", - teamId: "team-1", - }), - ]); + expect(normalized.provenance.remotes).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + provider: "linear", + remoteId: "lin_123", + remoteKey: "HACK-431", + remoteUrl: "https://linear.app/hack/issue/HACK-431", + projectId: "project-1", + projectName: "Hack App", + teamId: "team-1", + }), + expect.objectContaining({ + provider: "linear", + profileId: "default", + remoteCursor: "issue/lin_123#v2", + }), + ]) + ); expect(normalized.documents).toEqual([ expect.objectContaining({ kind: "description", @@ -588,6 +599,174 @@ test("normalized ticket adapter preserves compatibility while exposing provenanc ); }); +test("normalized ticket provenance captures multiple remotes, field authority, and field versions", () => { + const normalized = createNormalizedTicket({ + ticket: { + ticketId: "T-00077", + title: "Normalize provenance", + body: "Track provenance explicitly.", + status: "open", + createdAt: "2026-03-13T09:00:00.000Z", + updatedAt: "2026-03-13T12:00:00.000Z", + dependsOn: [], + blocks: [], + owner: "hack", + source: "linear", + assignee: "alice@hack", + tags: ["normalization"], + externalSystem: "linear", + externalId: "lin-77", + externalKey: "HACK-77", + externalUrl: "https://linear.app/hack/issue/HACK-77", + externalProjectId: "proj-77", + externalProjectName: "Hack App", + externalTeamId: "team-77", + projectId: "hack-cli", + projectName: "hack-cli", + }, + syncCheckpoints: [ + { + checkpointId: "checkpoint-linear", + ticketId: "T-00077", + provider: "linear", + profileId: "default", + direction: "pull", + remoteCursor: "issue/lin-77#v3", + remoteUpdatedAt: "2026-03-13T11:59:00.000Z", + actor: "sync@app", + createdAt: "2026-03-13T12:00:00.000Z", + }, + { + checkpointId: "checkpoint-github", + ticketId: "T-00077", + provider: "github", + profileId: "mirror", + direction: "push", + remoteCursor: "issue/gh-77#v1", + remoteUpdatedAt: "2026-03-13T11:45:00.000Z", + actor: "sync@app", + createdAt: "2026-03-13T12:00:00.000Z", + }, + ], + conflicts: [ + { + conflictId: "conflict-title", + ticketId: "T-00077", + provider: "linear", + field: "title", + status: "open", + authority: "review_required", + summary: "Title changed in both places.", + localValue: "Normalize provenance", + remoteValue: "Normalize explicit provenance", + createdAt: "2026-03-13T12:00:00.000Z", + updatedAt: "2026-03-13T12:00:00.000Z", + }, + ], + }); + + expect(normalized.provenance.remotes).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + provider: "linear", + remoteId: "lin-77", + remoteKey: "HACK-77", + }), + expect.objectContaining({ + provider: "github", + profileId: "mirror", + remoteCursor: "issue/gh-77#v1", + }), + ]) + ); + expect(normalized.provenance.fieldAuthorities).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + field: "title", + authority: "review_required", + }), + expect.objectContaining({ + field: "comment", + authority: "append_only", + }), + ]) + ); + expect(normalized.provenance.fieldVersions).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + field: "title", + source: "local", + recordedAt: "2026-03-13T12:00:00.000Z", + }), + expect.objectContaining({ + field: "title", + source: "remote", + provider: "linear", + value: "Normalize explicit provenance", + }), + ]) + ); +}); + +test("ticket provenance infers a remote provider from source metadata when externalSystem is absent", () => { + const ticket = { + ticketId: "T-00088", + title: "Infer remote provider", + body: "Link a remote without an explicit external system field.", + status: "open" as const, + createdAt: "2026-03-13T09:00:00.000Z", + updatedAt: "2026-03-13T12:00:00.000Z", + dependsOn: [], + blocks: [], + owner: "hack", + source: "linear", + tags: ["normalization"], + externalId: "lin-88", + externalKey: "HACK-88", + externalUrl: "https://linear.app/hack/issue/HACK-88", + externalProjectId: "proj-88", + externalProjectName: "Hack App", + externalTeamId: "team-88", + }; + + const provenance = buildTicketProvenance({ + ticket, + syncCheckpoints: [ + { + checkpointId: "checkpoint-linear", + ticketId: "T-00088", + provider: "linear", + direction: "pull", + remoteCursor: "issue/lin-88#v1", + remoteUpdatedAt: "2026-03-13T11:59:00.000Z", + actor: "sync@app", + createdAt: "2026-03-13T12:00:00.000Z", + }, + ], + }); + + expect(findTicketRemoteLink({ ticket, provider: "linear" })).toEqual( + expect.objectContaining({ + provider: "linear", + remoteId: "lin-88", + remoteKey: "HACK-88", + remoteUrl: "https://linear.app/hack/issue/HACK-88", + }) + ); + expect(provenance.fieldAuthorities).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + field: "title", + authority: "remote", + }), + expect.objectContaining({ + field: "comment", + authority: "append_only", + }), + ]) + ); +}); + async function createStore(opts: { readonly projectRoot: string }) { const configResult = await readControlPlaneConfig({ projectDir: join(opts.projectRoot, ".hack"), From 517031d46cf223cba2d839dca7295437fb8c0576 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 23:11:36 +0000 Subject: [PATCH 010/134] feat(tickets): make sync writes idempotent --- .../extensions/linear/commands.ts | 56 ++++++ src/control-plane/extensions/tickets/store.ts | 25 ++- tests/linear-commands.test.ts | 164 ++++++++++++++++++ tests/tickets-store.test.ts | 50 ++++++ 4 files changed, 294 insertions(+), 1 deletion(-) diff --git a/src/control-plane/extensions/linear/commands.ts b/src/control-plane/extensions/linear/commands.ts index 26bd0c3b6..313811047 100644 --- a/src/control-plane/extensions/linear/commands.ts +++ b/src/control-plane/extensions/linear/commands.ts @@ -3186,6 +3186,52 @@ function buildConflictDedupKey(input: { ].join("|"); } +function buildLinearCheckpointIdempotencyKey(input: { + readonly ticketId: string; + readonly profileId: string; + readonly direction: string; + readonly issue: LinearIssue; +}): string { + const fingerprint = createHash("sha256") + .update( + JSON.stringify({ + issueId: input.issue.id, + identifier: input.issue.identifier, + title: input.issue.title, + description: input.issue.description ?? null, + stateId: input.issue.state.id, + teamId: input.issue.teamId, + projectId: input.issue.projectId ?? null, + assigneeId: input.issue.assigneeId ?? null, + labels: input.issue.labels.map((label) => label.id).sort(), + }) + ) + .digest("hex") + .slice(0, 16); + + return `linear:checkpoint:${input.ticketId}:${input.profileId}:${input.direction}:${fingerprint}`; +} + +function buildLinearConflictIdempotencyKey(input: { + readonly ticketId: string; + readonly conflict: RecordedSyncConflict; +}): string { + const fingerprint = createHash("sha256") + .update( + JSON.stringify({ + field: input.conflict.field, + authority: input.conflict.authority ?? null, + summary: input.conflict.summary ?? null, + localValue: input.conflict.localValue ?? null, + remoteValue: input.conflict.remoteValue ?? null, + }) + ) + .digest("hex") + .slice(0, 16); + + return `linear:conflict:${input.ticketId}:${fingerprint}`; +} + async function recordAuthoritativeConflicts(input: { readonly tickets: Pick< TicketSyncStore, @@ -3228,6 +3274,10 @@ async function recordAuthoritativeConflicts(input: { field: conflict.field, authority: conflict.authority, summary: conflict.summary, + idempotencyKey: buildLinearConflictIdempotencyKey({ + ticketId: input.ticketId, + conflict, + }), ...(conflict.localValue !== undefined ? { localValue: conflict.localValue } : {}), @@ -3349,6 +3399,12 @@ async function recordLinearSyncCheckpoint(input: { direction: input.direction, remoteCursor: input.issue.identifier, localUpdatedAt: new Date().toISOString(), + idempotencyKey: buildLinearCheckpointIdempotencyKey({ + ticketId: input.ticketId, + profileId: input.profileId, + direction: input.direction, + issue: input.issue, + }), }); if (!checkpoint.ok) { return { ok: false, error: checkpoint.error }; diff --git a/src/control-plane/extensions/tickets/store.ts b/src/control-plane/extensions/tickets/store.ts index ef2851212..e6afc3743 100644 --- a/src/control-plane/extensions/tickets/store.ts +++ b/src/control-plane/extensions/tickets/store.ts @@ -287,6 +287,7 @@ export function createTicketsStore(opts: { readonly remoteCursor?: string; readonly remoteUpdatedAt?: string; readonly localUpdatedAt?: string; + readonly idempotencyKey?: string; readonly actor?: string; }) => Promise< | { readonly ok: true; readonly checkpoint: TicketSyncCheckpoint } @@ -300,6 +301,7 @@ export function createTicketsStore(opts: { readonly summary?: string; readonly localValue?: TicketMetadataValue; readonly remoteValue?: TicketMetadataValue; + readonly idempotencyKey?: string; readonly actor?: string; }) => Promise< | { readonly ok: true; readonly conflict: TicketSyncConflict } @@ -635,7 +637,22 @@ export function createTicketsStore(opts: { }): Promise< { readonly ok: true } | { readonly ok: false; readonly error: string } > => { - const wrote = await git.appendEvents({ events: input.events }); + const context = await loadStoreContext(); + const seenIdempotencyKeys = new Set( + context.events.map((event) => event.idempotencyKey) + ); + const pendingEvents = input.events.filter((event) => { + if (seenIdempotencyKeys.has(event.idempotencyKey)) { + return false; + } + seenIdempotencyKeys.add(event.idempotencyKey); + return true; + }); + if (pendingEvents.length === 0) { + return { ok: true }; + } + + const wrote = await git.appendEvents({ events: pendingEvents }); if (!wrote.ok) { return wrote; } @@ -1045,6 +1062,9 @@ export function createTicketsStore(opts: { ...(remoteUpdatedAt ? { remoteUpdatedAt } : {}), ...(localUpdatedAt ? { localUpdatedAt } : {}), }, + ...(input.idempotencyKey + ? { idempotencyKey: input.idempotencyKey } + : {}), actor: input.actor, }); @@ -1112,6 +1132,9 @@ export function createTicketsStore(opts: { ? { remoteValue: input.remoteValue } : {}), }, + ...(input.idempotencyKey + ? { idempotencyKey: input.idempotencyKey } + : {}), actor: input.actor, }); diff --git a/tests/linear-commands.test.ts b/tests/linear-commands.test.ts index d39152f28..60cd4fab1 100644 --- a/tests/linear-commands.test.ts +++ b/tests/linear-commands.test.ts @@ -1457,6 +1457,170 @@ test("syncTicketToLinearIssue reuses a linked Linear issue inferred from provena expect(updatedIssues).toHaveLength(1); }); +test("syncTicketToLinearIssue records a deterministic checkpoint idempotency key", async () => { + const checkpointInputs: Record[] = []; + + const runtime = { + profileId: "default", + apiUrl: "https://api.linear.app/graphql", + projectBinding: { + teamId: "team-1", + additionalProjects: [], + }, + assigneeMappings: [], + tickets: { + getTicket: async () => ({ + ticketId: "T-00003", + title: "Deterministic checkpoint", + body: "Keep checkpoint writes idempotent.", + status: "open" as const, + createdAt: "2026-03-05T10:00:00.000Z", + updatedAt: "2026-03-05T10:00:00.000Z", + dependsOn: [], + blocks: [], + owner: "hack", + source: "hack", + assignee: "alice@example.com", + tags: [], + externalSystem: "linear", + externalId: "issue-3", + externalKey: "ENG-333", + externalTeamId: "team-1", + }), + updateTicket: async () => ({ ok: true as const }), + listTickets: async () => [], + getTicketDetail: async () => ({ + ticket: null, + events: [], + comments: [], + reviewNotes: [], + syncCheckpoints: [], + conflicts: [], + }), + linkCommentExternalId: async () => ({ ok: true as const }), + recordSyncCheckpoint: async (input: Record) => { + checkpointInputs.push(input); + return { + ok: true as const, + checkpoint: { + checkpointId: "checkpoint-3", + ticketId: "T-00003", + provider: "linear", + direction: input.direction, + actor: "test", + createdAt: "2026-03-05T10:10:00.000Z", + }, + }; + }, + recordSyncConflict: async () => ({ ok: true as const, conflict: null }), + }, + linear: { + getIssueById: async () => ({ + ok: true as const, + data: { + id: "issue-3", + identifier: "ENG-333", + title: "Deterministic checkpoint", + description: "Keep checkpoint writes idempotent.", + state: { + id: "state-1", + name: "Todo", + type: "unstarted" as const, + }, + teamId: "team-1", + assigneeId: "user-1", + labels: [], + }, + }), + updateIssue: async () => ({ + ok: true as const, + data: { + id: "issue-3", + identifier: "ENG-333", + title: "Deterministic checkpoint", + description: "Keep checkpoint writes idempotent.", + state: { + id: "state-1", + name: "Todo", + type: "unstarted" as const, + }, + teamId: "team-1", + assigneeId: "user-1", + labels: [], + }, + }), + createIssue: async () => { + throw new Error("createIssue should not be called"); + }, + listTeamStates: async () => ({ + ok: true as const, + data: [ + { + id: "state-1", + name: "Todo", + type: "unstarted" as const, + }, + ], + }), + listTeamLabels: async () => ({ + ok: true as const, + data: [], + }), + listIssueComments: async () => ({ + ok: true as const, + data: [], + }), + createComment: async () => { + throw new Error("createComment should not be called"); + }, + listTeamUsers: async () => ({ + ok: true as const, + data: [ + { + id: "user-1", + email: "alice@example.com", + displayName: "Alice", + }, + ], + }), + getIssueByIdentifier: async () => ({ ok: true as const, data: null }), + getProject: async () => ({ ok: true as const, data: null }), + }, + }; + + const first = await __testOnly.syncTicketToLinearIssue({ + runtime, + ticketId: "T-00003", + syncToggles: { + labels: false, + statuses: true, + dependencies: false, + projects: true, + }, + }); + expect(first.ok).toBe(true); + + const second = await __testOnly.syncTicketToLinearIssue({ + runtime, + ticketId: "T-00003", + syncToggles: { + labels: false, + statuses: true, + dependencies: false, + projects: true, + }, + }); + expect(second.ok).toBe(true); + + expect(checkpointInputs).toHaveLength(2); + expect(checkpointInputs[0]?.idempotencyKey).toBe( + checkpointInputs[1]?.idempotencyKey + ); + expect(checkpointInputs[0]?.idempotencyKey).toContain( + "linear:checkpoint:T-00003" + ); +}); + test("runProjectLinearAutosync syncs issue and project deliveries, then applies them", async () => { const appliedDeliveryIds: string[] = []; const issueSyncCalls: Array<{ issueIdentifier?: string; issueId?: string }> = diff --git a/tests/tickets-store.test.ts b/tests/tickets-store.test.ts index 4ef4a6a63..becf0d088 100644 --- a/tests/tickets-store.test.ts +++ b/tests/tickets-store.test.ts @@ -450,6 +450,56 @@ test("tickets store writes normalized journal envelope metadata and ignores dupl }); }, 20_000); +test("tickets store ignores duplicate sync checkpoints with the same idempotency key", async () => { + const projectRoot = await createTempGitProject({ + prefix: "hack-cli-tickets-checkpoint-idempotency-", + }); + const store = await createStore({ projectRoot }); + + const created = await store.createTicket({ + title: "Checkpoint idempotency", + owner: "hack", + source: "linear", + actor: "creator@hack", + }); + expect(created.ok).toBe(true); + if (!created.ok) { + throw new Error(created.error); + } + + const first = await store.recordSyncCheckpoint({ + ticketId: created.ticket.ticketId, + provider: "linear", + profileId: "default", + direction: "hack_to_linear", + remoteCursor: "ENG-321", + idempotencyKey: "linear:checkpoint:T-00001:ENG-321", + actor: "sync@app", + }); + expect(first.ok).toBe(true); + + const second = await store.recordSyncCheckpoint({ + ticketId: created.ticket.ticketId, + provider: "linear", + profileId: "default", + direction: "hack_to_linear", + remoteCursor: "ENG-321", + idempotencyKey: "linear:checkpoint:T-00001:ENG-321", + actor: "sync@app", + }); + expect(second.ok).toBe(true); + + const detail = await store.getTicketDetail({ + ticketId: created.ticket.ticketId, + }); + expect(detail.syncCheckpoints).toHaveLength(1); + expect( + detail.events.filter( + (event) => event.type === "ticket.sync_checkpoint_recorded" + ) + ).toHaveLength(1); +}); + test("tickets store persists a sqlite projection and rebuilds it when deleted", async () => { const projectRoot = await createTempGitProject({ prefix: "hack-cli-tickets-projection-", From d166be12e8c5994bdf13bd319160776cb23ceb39 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Fri, 13 Mar 2026 23:32:28 +0000 Subject: [PATCH 011/134] feat(tickets): add document-backed ticket content --- .../extensions/tickets/commands.ts | 168 ++++++++++++ .../extensions/tickets/documents.ts | 97 +++++++ .../extensions/tickets/domain.ts | 41 ++- .../extensions/tickets/sqlite-projection.ts | 29 ++ src/control-plane/extensions/tickets/store.ts | 230 ++++++++++++++-- tests/tickets-extension.test.ts | 252 ++++++++++++++++++ tests/tickets-store.test.ts | 67 +++++ 7 files changed, 845 insertions(+), 39 deletions(-) create mode 100644 src/control-plane/extensions/tickets/documents.ts diff --git a/src/control-plane/extensions/tickets/commands.ts b/src/control-plane/extensions/tickets/commands.ts index 673e0154c..82192f283 100644 --- a/src/control-plane/extensions/tickets/commands.ts +++ b/src/control-plane/extensions/tickets/commands.ts @@ -11,6 +11,12 @@ import { type TicketsAgentDocUpdateResult, upsertTicketsAgentDocs, } from "./agent-docs.ts"; +import { + isTicketDocumentKind, + isTicketDocumentRole, + type TicketDocumentKind, + type TicketDocumentRole, +} from "./documents.ts"; import { checkTicketsRepoState, ensureTicketsGitignore, @@ -686,6 +692,91 @@ export const TICKETS_COMMANDS: readonly ExtensionCommand[] = [ return 0; }, }, + { + name: "document", + summary: "Append an immutable ticket document", + scope: "project", + handler: async ({ ctx, args }) => { + if (!ctx.project) { + ctx.logger.error({ message: "No project found. Run inside a repo." }); + return 1; + } + + const parsed = parseTicketsArgs({ args }); + if (!parsed.ok) { + ctx.logger.error({ message: parsed.error }); + return 1; + } + + const ticketId = (parsed.value.rest[0] ?? "").trim(); + if (!ticketId) { + ctx.logger.error({ + message: + 'Usage: hack x tickets document --kind [--role ] [--body "..."] [--body-file ] [--body-stdin] [--json]', + }); + return 1; + } + if (!parsed.value.kind) { + ctx.logger.error({ + message: + "Document kind is required. Use --kind .", + }); + return 1; + } + + const body = await resolveTicketBody({ + body: parsed.value.body, + bodyFile: parsed.value.bodyFile, + bodyStdin: parsed.value.bodyStdin, + }); + if (!body) { + ctx.logger.error({ + message: + "Document body is required. Use --body, --body-file, or --body-stdin.", + }); + return 1; + } + + await maybeEnsureTicketsSetup({ ctx, json: parsed.value.json }); + + const store = createTicketsStore({ + projectRoot: ctx.project.projectRoot, + projectId: ctx.projectId, + projectName: ctx.projectName, + controlPlaneConfig: ctx.controlPlaneConfig, + logger: ctx.logger, + }); + + const appended = await store.appendDocument({ + ticketId, + kind: parsed.value.kind, + ...(parsed.value.role ? { role: parsed.value.role } : {}), + content: body, + actor: parsed.value.actor, + }); + if (!appended.ok) { + ctx.logger.error({ message: appended.error }); + return 1; + } + + if (parsed.value.json) { + process.stdout.write( + `${JSON.stringify({ document: appended.document }, null, 2)}\n` + ); + return 0; + } + + await display.panel({ + title: "Ticket document", + tone: "success", + lines: [ + `${ticketId} ${appended.document.role} document appended`, + appended.document.content, + ], + }); + return 0; + }, + }, { name: "list", summary: "List tickets", @@ -833,6 +924,7 @@ export const TICKETS_COMMANDS: readonly ExtensionCommand[] = [ `${JSON.stringify( { ticket, + documents: detail.documents, comments: detail.comments, reviewNotes: detail.reviewNotes, syncCheckpoints: detail.syncCheckpoints, @@ -879,6 +971,18 @@ export const TICKETS_COMMANDS: readonly ExtensionCommand[] = [ }); } + if (detail.documents.length > 0) { + await display.table({ + columns: ["document_id", "kind", "role", "updated_at"], + rows: detail.documents.map((document) => [ + document.documentId, + document.kind, + document.role, + document.updatedAt, + ]), + }); + } + if (detail.comments.length > 0) { await display.table({ columns: ["comment_id", "source", "actor", "created_at", "body"], @@ -1153,6 +1257,8 @@ type TicketsArgs = { readonly body?: string; readonly bodyFile?: string; readonly bodyStdin: boolean; + readonly kind?: TicketDocumentKind; + readonly role?: TicketDocumentRole; readonly dependsOn: readonly string[]; readonly blocks: readonly string[]; readonly clearDependsOn: boolean; @@ -1210,6 +1316,8 @@ function parseTicketsArgs(opts: { let body: string | undefined; let bodyFile: string | undefined; let bodyStdin = false; + let kind: TicketDocumentKind | undefined; + let role: TicketDocumentRole | undefined; const dependsOn: string[] = []; const blocks: string[] = []; let clearDependsOn = false; @@ -1303,6 +1411,64 @@ function parseTicketsArgs(opts: { continue; } + if (token.startsWith("--kind=")) { + const value = token.slice("--kind=".length); + if (!isTicketDocumentKind(value)) { + return { + ok: false, + error: "Invalid --kind value. Expected description|spec|notes.", + }; + } + kind = value; + continue; + } + + if (token === "--kind") { + const value = takeValue(token, opts.args[i + 1]); + if (!value) { + return { ok: false, error: "--kind requires a value." }; + } + if (!isTicketDocumentKind(value)) { + return { + ok: false, + error: "Invalid --kind value. Expected description|spec|notes.", + }; + } + kind = value; + i += 1; + continue; + } + + if (token.startsWith("--role=")) { + const value = token.slice("--role=".length); + if (!isTicketDocumentRole(value)) { + return { + ok: false, + error: + "Invalid --role value. Expected description|spec|notes|handoff.", + }; + } + role = value; + continue; + } + + if (token === "--role") { + const value = takeValue(token, opts.args[i + 1]); + if (!value) { + return { ok: false, error: "--role requires a value." }; + } + if (!isTicketDocumentRole(value)) { + return { + ok: false, + error: + "Invalid --role value. Expected description|spec|notes|handoff.", + }; + } + role = value; + i += 1; + continue; + } + if (token === "--clear-depends-on") { clearDependsOn = true; continue; @@ -1540,6 +1706,8 @@ function parseTicketsArgs(opts: { ...(body ? { body } : {}), ...(bodyFile ? { bodyFile } : {}), bodyStdin, + ...(kind ? { kind } : {}), + ...(role ? { role } : {}), dependsOn, blocks, clearDependsOn, diff --git a/src/control-plane/extensions/tickets/documents.ts b/src/control-plane/extensions/tickets/documents.ts new file mode 100644 index 000000000..f43ee8300 --- /dev/null +++ b/src/control-plane/extensions/tickets/documents.ts @@ -0,0 +1,97 @@ +import { sha256Hex } from "./util.ts"; + +export type TicketDocumentKind = "description" | "spec" | "notes"; +export type TicketDocumentRole = TicketDocumentKind | "handoff"; + +export type TicketDocument = { + readonly documentId: string; + readonly ticketId: string; + readonly kind: TicketDocumentKind; + readonly role: TicketDocumentRole; + readonly content: string; + readonly contentSha256: string; + readonly createdAt: string; + readonly updatedAt: string; +}; + +export function isTicketDocumentKind( + value: string +): value is TicketDocumentKind { + return ["description", "spec", "notes"].includes(value); +} + +export function isTicketDocumentRole( + value: string +): value is TicketDocumentRole { + return ["description", "spec", "notes", "handoff"].includes(value); +} + +export function resolveTicketDocumentRole(input: { + readonly kind: TicketDocumentKind; + readonly role?: TicketDocumentRole; +}): TicketDocumentRole { + return input.role ?? input.kind; +} + +export function buildTicketDocument(input: { + readonly documentId: string; + readonly ticketId: string; + readonly kind: TicketDocumentKind; + readonly role?: TicketDocumentRole; + readonly content: string; + readonly createdAt: string; + readonly updatedAt: string; +}): TicketDocument { + return { + documentId: input.documentId, + ticketId: input.ticketId, + kind: input.kind, + role: resolveTicketDocumentRole({ + kind: input.kind, + role: input.role, + }), + content: input.content, + contentSha256: sha256Hex({ value: input.content }), + createdAt: input.createdAt, + updatedAt: input.updatedAt, + }; +} + +export function buildLegacyDescriptionDocument(input: { + readonly eventId: string; + readonly ticketId: string; + readonly content: string; + readonly createdAt: string; + readonly updatedAt: string; +}): TicketDocument { + return buildTicketDocument({ + documentId: `${input.ticketId}:description:${input.eventId}`, + ticketId: input.ticketId, + kind: "description", + content: input.content, + createdAt: input.createdAt, + updatedAt: input.updatedAt, + }); +} + +export function getActiveTicketDescription(input: { + readonly documents: readonly TicketDocument[]; +}): TicketDocument | undefined { + for (let index = input.documents.length - 1; index >= 0; index -= 1) { + const document = input.documents[index]; + if (document?.role === "description") { + return document; + } + } + return undefined; +} + +export function projectTicketBodyFromDocuments(input: { + readonly documents: readonly TicketDocument[]; + readonly fallbackBody?: string; +}): string | undefined { + return ( + getActiveTicketDescription({ documents: input.documents })?.content ?? + input.fallbackBody + ); +} diff --git a/src/control-plane/extensions/tickets/domain.ts b/src/control-plane/extensions/tickets/domain.ts index 953b9daee..b2af16d47 100644 --- a/src/control-plane/extensions/tickets/domain.ts +++ b/src/control-plane/extensions/tickets/domain.ts @@ -1,8 +1,8 @@ +import { buildLegacyDescriptionDocument } from "./documents.ts"; import { buildTicketProvenance, projectRemoteLinkToCompatibilityFields, } from "./provenance.ts"; -import { sha256Hex } from "./util.ts"; export type TicketStatus = "open" | "in_progress" | "blocked" | "done"; @@ -114,19 +114,11 @@ export type TicketFieldVersion = { readonly value?: TicketMetadataValue; }; -export type TicketDocumentKind = "description" | "spec" | "notes"; -export type TicketDocumentRole = TicketDocumentKind | "handoff"; - -export type TicketDocument = { - readonly documentId: string; - readonly ticketId: string; - readonly kind: TicketDocumentKind; - readonly role: TicketDocumentRole; - readonly content: string; - readonly contentSha256: string; - readonly createdAt: string; - readonly updatedAt: string; -}; +export type { + TicketDocument, + TicketDocumentKind, + TicketDocumentRole, +} from "./documents.ts"; export type TicketFieldState = { readonly field: string; @@ -162,8 +154,12 @@ export function createNormalizedTicket(input: { readonly ticket: TicketSummaryCompatibility; readonly syncCheckpoints?: readonly TicketSyncCheckpointCompatibility[]; readonly conflicts?: readonly TicketSyncConflictCompatibility[]; + readonly documents?: readonly TicketDocument[]; }): NormalizedTicket { - const documents = buildDocuments({ ticket: input.ticket }); + const documents = buildDocuments({ + ticket: input.ticket, + documents: input.documents, + }); const checkpoints = input.syncCheckpoints ?? []; const conflicts = input.conflicts ?? []; const provenance = buildTicketProvenance({ @@ -238,23 +234,22 @@ export function projectNormalizedTicketSummary(input: { function buildDocuments(input: { readonly ticket: TicketSummaryCompatibility; + readonly documents?: readonly TicketDocument[]; }): TicketDocument[] { + if (input.documents && input.documents.length > 0) { + return [...input.documents]; + } if (!input.ticket.body) { return []; } - - const contentSha256 = sha256Hex({ value: input.ticket.body }); return [ - { - documentId: `${input.ticket.ticketId}:description:${contentSha256.slice(0, 12)}`, + buildLegacyDescriptionDocument({ + eventId: "compatibility-summary", ticketId: input.ticket.ticketId, - kind: "description", - role: "description", content: input.ticket.body, - contentSha256, createdAt: input.ticket.createdAt, updatedAt: input.ticket.updatedAt, - }, + }), ]; } diff --git a/src/control-plane/extensions/tickets/sqlite-projection.ts b/src/control-plane/extensions/tickets/sqlite-projection.ts index 141299da2..6e9ff6fa7 100644 --- a/src/control-plane/extensions/tickets/sqlite-projection.ts +++ b/src/control-plane/extensions/tickets/sqlite-projection.ts @@ -2,6 +2,7 @@ import { Database } from "bun:sqlite"; import { mkdir, readdir } from "node:fs/promises"; import { dirname, resolve } from "node:path"; +import type { TicketDocument } from "./documents.ts"; import type { TicketComment, TicketEvent, @@ -105,6 +106,9 @@ export function createTicketsSqliteProjection(opts: { const insertComment = db.query( "INSERT INTO ticket_comments (comment_id, ticket_id, json_value) VALUES (?, ?, ?)" ); + const insertDocument = db.query( + "INSERT INTO ticket_documents (document_id, ticket_id, json_value) VALUES (?, ?, ?)" + ); const insertReviewNote = db.query( "INSERT INTO ticket_review_notes (note_id, ticket_id, json_value) VALUES (?, ?, ?)" ); @@ -123,6 +127,7 @@ export function createTicketsSqliteProjection(opts: { DELETE FROM tickets; DELETE FROM journal_events; DELETE FROM ticket_comments; + DELETE FROM ticket_documents; DELETE FROM ticket_review_notes; DELETE FROM ticket_sync_checkpoints; DELETE FROM ticket_sync_conflicts; @@ -164,6 +169,17 @@ export function createTicketsSqliteProjection(opts: { }, }); + writeMapEntries({ + map: input.snapshot.documentsByTicket, + write: (document: TicketDocument) => { + insertDocument.run( + document.documentId, + document.ticketId, + JSON.stringify(document) + ); + }, + }); + writeMapEntries({ map: input.snapshot.reviewNotesByTicket, write: (reviewNote: TicketReviewNote) => { @@ -244,6 +260,10 @@ export function createTicketsSqliteProjection(opts: { db, table: "ticket_comments", }); + const documentsByTicket = readRowsByTicket({ + db, + table: "ticket_documents", + }); const reviewNotesByTicket = readRowsByTicket({ db, table: "ticket_review_notes", @@ -260,6 +280,7 @@ export function createTicketsSqliteProjection(opts: { return { tickets, eventsByTicket, + documentsByTicket, commentsByTicket, reviewNotesByTicket, syncCheckpointsByTicket, @@ -305,6 +326,11 @@ function initializeProjectionSchema(input: { readonly db: Database }): void { ticket_id TEXT NOT NULL, json_value TEXT NOT NULL ); + CREATE TABLE IF NOT EXISTS ticket_documents ( + document_id TEXT PRIMARY KEY, + ticket_id TEXT NOT NULL, + json_value TEXT NOT NULL + ); CREATE TABLE IF NOT EXISTS ticket_review_notes ( note_id TEXT PRIMARY KEY, ticket_id TEXT NOT NULL, @@ -324,6 +350,8 @@ function initializeProjectionSchema(input: { readonly db: Database }): void { ON journal_events (ticket_id, ts, order_key); CREATE INDEX IF NOT EXISTS ticket_comments_ticket_id_idx ON ticket_comments (ticket_id); + CREATE INDEX IF NOT EXISTS ticket_documents_ticket_id_idx + ON ticket_documents (ticket_id); CREATE INDEX IF NOT EXISTS ticket_review_notes_ticket_id_idx ON ticket_review_notes (ticket_id); CREATE INDEX IF NOT EXISTS ticket_sync_checkpoints_ticket_id_idx @@ -367,6 +395,7 @@ function readRowsByTicket(input: { readonly db: Database; readonly table: | "ticket_comments" + | "ticket_documents" | "ticket_review_notes" | "ticket_sync_checkpoints" | "ticket_sync_conflicts"; diff --git a/src/control-plane/extensions/tickets/store.ts b/src/control-plane/extensions/tickets/store.ts index e6afc3743..5df5be9f3 100644 --- a/src/control-plane/extensions/tickets/store.ts +++ b/src/control-plane/extensions/tickets/store.ts @@ -5,6 +5,16 @@ import { resolve } from "node:path"; import { isRecord } from "../../../lib/guards.ts"; import type { ControlPlaneConfig } from "../../sdk/config.ts"; +import { + buildLegacyDescriptionDocument, + buildTicketDocument, + getActiveTicketDescription, + isTicketDocumentKind, + isTicketDocumentRole, + type TicketDocument, + type TicketDocumentKind, + type TicketDocumentRole, +} from "./documents.ts"; import { createNormalizedTicket, projectNormalizedTicketSummary, @@ -134,6 +144,7 @@ export type TicketEvent = { export type TicketStoreSnapshot = { readonly tickets: readonly TicketSummary[]; readonly eventsByTicket: ReadonlyMap; + readonly documentsByTicket: ReadonlyMap; readonly commentsByTicket: ReadonlyMap; readonly reviewNotesByTicket: ReadonlyMap< string, @@ -165,6 +176,7 @@ type SyncResult = type MaterializedTicketState = { readonly tickets: Map; + readonly documentsByTicket: Map; readonly commentsByTicket: Map; readonly reviewNotesByTicket: Map; readonly syncCheckpointsByTicket: Map; @@ -245,6 +257,7 @@ export function createTicketsStore(opts: { readonly getTicketDetail: (input: { readonly ticketId: string }) => Promise<{ readonly ticket: TicketSummary | null; readonly events: readonly TicketEvent[]; + readonly documents: readonly TicketDocument[]; readonly comments: readonly TicketComment[]; readonly reviewNotes: readonly TicketReviewNote[]; readonly syncCheckpoints: readonly TicketSyncCheckpoint[]; @@ -270,6 +283,16 @@ export function createTicketsStore(opts: { | { readonly ok: true; readonly reviewNote: TicketReviewNote } | { readonly ok: false; readonly error: string } >; + readonly appendDocument: (input: { + readonly ticketId: string; + readonly kind: TicketDocumentKind; + readonly role?: TicketDocumentRole; + readonly content: string; + readonly actor?: string; + }) => Promise< + | { readonly ok: true; readonly document: TicketDocument } + | { readonly ok: false; readonly error: string } + >; readonly linkCommentExternalId: (input: { readonly ticketId: string; readonly commentId: string; @@ -448,6 +471,7 @@ export function createTicketsStore(opts: { readonly events: readonly TicketEvent[]; }): MaterializedTicketState => { const tickets = new Map(); + const documentsByTicket = new Map(); const commentsByTicket = new Map(); const reviewNotesByTicket = new Map(); const syncCheckpointsByTicket = new Map(); @@ -456,6 +480,7 @@ export function createTicketsStore(opts: { for (const event of input.events) { applyTicketEvent({ tickets, + documentsByTicket, commentsByTicket, reviewNotesByTicket, syncCheckpointsByTicket, @@ -466,6 +491,7 @@ export function createTicketsStore(opts: { return { tickets: applyDerivedBlocks(tickets), + documentsByTicket, commentsByTicket, reviewNotesByTicket, syncCheckpointsByTicket, @@ -504,6 +530,7 @@ export function createTicketsStore(opts: { return { tickets: sortTickets({ tickets: input.materialized.tickets.values() }), eventsByTicket: groupEventsByTicket({ events: input.events }), + documentsByTicket: input.materialized.documentsByTicket, commentsByTicket: input.materialized.commentsByTicket, reviewNotesByTicket: input.materialized.reviewNotesByTicket, syncCheckpointsByTicket: input.materialized.syncCheckpointsByTicket, @@ -764,12 +791,12 @@ export function createTicketsStore(opts: { updateTicket: async (input) => { const tickets = await materializeTickets(); - const current = tickets.get(input.ticketId); - if (!current) { + if (!tickets.has(input.ticketId)) { return { ok: false, error: `Ticket not found: ${input.ticketId}` }; } const payload: Record = {}; + const events: TicketEvent[] = []; if (input.title !== undefined) { const title = input.title.trim(); if (!title) { @@ -778,7 +805,22 @@ export function createTicketsStore(opts: { payload.title = title; } if (input.body !== undefined) { - payload.body = input.body; + const content = input.body.trimEnd(); + if (!content) { + return { ok: false, error: "Body cannot be empty." }; + } + events.push( + buildEvent({ + ticketId: input.ticketId, + type: "ticket.document_recorded", + payload: { + documentId: randomUUID(), + kind: "description", + content, + }, + actor: input.actor, + }) + ); } if (input.dependsOn !== undefined) { payload.dependsOn = normalizeTicketRefs(input.dependsOn); @@ -843,18 +885,22 @@ export function createTicketsStore(opts: { }); } - if (Object.keys(payload).length === 0) { - return { ok: false, error: "No updates provided." }; + if (Object.keys(payload).length > 0) { + events.push( + buildEvent({ + ticketId: input.ticketId, + type: "ticket.updated", + payload, + actor: input.actor, + }) + ); } - const event = buildEvent({ - ticketId: input.ticketId, - type: "ticket.updated", - payload, - actor: input.actor, - }); + if (events.length === 0) { + return { ok: false, error: "No updates provided." }; + } - return await appendEventsAndRefresh({ events: [event] }); + return await appendEventsAndRefresh({ events }); }, listTickets: async () => { @@ -881,6 +927,7 @@ export function createTicketsStore(opts: { snapshot.tickets.find((ticket) => ticket.ticketId === ticketId) ?? null, events: snapshot.eventsByTicket.get(ticketId) ?? [], + documents: snapshot.documentsByTicket.get(ticketId) ?? [], comments: snapshot.commentsByTicket.get(ticketId) ?? [], reviewNotes: snapshot.reviewNotesByTicket.get(ticketId) ?? [], syncCheckpoints: snapshot.syncCheckpointsByTicket.get(ticketId) ?? [], @@ -988,6 +1035,49 @@ export function createTicketsStore(opts: { }; }, + appendDocument: async (input) => { + const tickets = await materializeTickets(); + if (!tickets.has(input.ticketId)) { + return { ok: false, error: `Ticket not found: ${input.ticketId}` }; + } + + const content = input.content.trimEnd(); + if (!content.trim()) { + return { ok: false, error: "Document content cannot be empty." }; + } + + const documentId = randomUUID(); + const event = buildEvent({ + ticketId: input.ticketId, + type: "ticket.document_recorded", + payload: { + documentId, + kind: input.kind, + ...(input.role ? { role: input.role } : {}), + content, + }, + actor: input.actor, + }); + + const wrote = await appendEventsAndRefresh({ events: [event] }); + if (!wrote.ok) { + return wrote; + } + + return { + ok: true, + document: buildTicketDocument({ + documentId, + ticketId: input.ticketId, + kind: input.kind, + ...(input.role ? { role: input.role } : {}), + content, + createdAt: event.tsIso, + updatedAt: event.tsIso, + }), + }; + }, + linkCommentExternalId: async (input) => { const { snapshot } = await loadStoreContext(); const comments = snapshot.commentsByTicket.get(input.ticketId) ?? []; @@ -1207,6 +1297,7 @@ export function createTicketsStore(opts: { function applyTicketEvent(input: { readonly tickets: Map; + readonly documentsByTicket: Map; readonly commentsByTicket: Map; readonly reviewNotesByTicket: Map; readonly syncCheckpointsByTicket: Map; @@ -1217,6 +1308,7 @@ function applyTicketEvent(input: { case "ticket.created": { applyTicketCreatedEvent({ tickets: input.tickets, + documentsByTicket: input.documentsByTicket, event: input.event, }); break; @@ -1231,6 +1323,15 @@ function applyTicketEvent(input: { case "ticket.updated": { applyTicketUpdatedEvent({ tickets: input.tickets, + documentsByTicket: input.documentsByTicket, + event: input.event, + }); + break; + } + case "ticket.document_recorded": { + applyTicketDocumentRecordedEvent({ + tickets: input.tickets, + documentsByTicket: input.documentsByTicket, event: input.event, }); break; @@ -1289,6 +1390,7 @@ function applyTicketEvent(input: { function applyTicketCreatedEvent(input: { readonly tickets: Map; + readonly documentsByTicket: Map; readonly event: TicketEvent; }): void { const title = @@ -1369,6 +1471,20 @@ function applyTicketCreatedEvent(input: { }, }) ); + + if (body) { + appendMapValue({ + map: input.documentsByTicket, + key: input.event.ticketId, + value: buildLegacyDescriptionDocument({ + eventId: input.event.eventId, + ticketId: input.event.ticketId, + content: body, + createdAt: input.event.tsIso, + updatedAt: input.event.tsIso, + }), + }); + } } function applyTicketStatusChangedEvent(input: { @@ -1399,6 +1515,7 @@ function applyTicketStatusChangedEvent(input: { function applyTicketUpdatedEvent(input: { readonly tickets: Map; + readonly documentsByTicket: Map; readonly event: TicketEvent; }): void { const current = input.tickets.get(input.event.ticketId); @@ -1517,6 +1634,78 @@ function applyTicketUpdatedEvent(input: { }, }) ); + + if (body !== undefined) { + appendMapValue({ + map: input.documentsByTicket, + key: input.event.ticketId, + value: buildLegacyDescriptionDocument({ + eventId: input.event.eventId, + ticketId: input.event.ticketId, + content: body, + createdAt: input.event.tsIso, + updatedAt: input.event.tsIso, + }), + }); + } +} + +function applyTicketDocumentRecordedEvent(input: { + readonly tickets: Map; + readonly documentsByTicket: Map; + readonly event: TicketEvent; +}): void { + const current = input.tickets.get(input.event.ticketId); + if (!current) { + return; + } + + const kindValue = readOptionalStringPayload({ + value: input.event.payload.kind, + }); + const content = readOptionalTextPayload({ + value: input.event.payload.content, + }); + if (!(kindValue && content && isTicketDocumentKind(kindValue))) { + return; + } + + const roleValue = readOptionalStringPayload({ + value: input.event.payload.role, + }); + const role = + roleValue && isTicketDocumentRole(roleValue) ? roleValue : undefined; + + const document = buildTicketDocument({ + documentId: + readOptionalStringPayload({ value: input.event.payload.documentId }) ?? + input.event.eventId, + ticketId: input.event.ticketId, + kind: kindValue, + ...(role ? { role } : {}), + content, + createdAt: input.event.tsIso, + updatedAt: input.event.tsIso, + }); + + appendMapValue({ + map: input.documentsByTicket, + key: input.event.ticketId, + value: document, + }); + + const documents = input.documentsByTicket.get(input.event.ticketId) ?? []; + const activeDescription = getActiveTicketDescription({ documents }); + input.tickets.set( + input.event.ticketId, + normalizeTicketSummaryCompatibility({ + ticket: { + ...current, + ...(activeDescription ? { body: activeDescription.content } : {}), + updatedAt: input.event.tsIso, + }, + }) + ); } function applyTicketCommentAppendedEvent(input: { @@ -1529,10 +1718,10 @@ function applyTicketCommentAppendedEvent(input: { } const body = - readOptionalStringPayload({ + readOptionalTextPayload({ value: input.event.payload.body, }) ?? - readOptionalStringPayload({ + readOptionalTextPayload({ value: input.event.payload.markdown, }); if (!body) { @@ -1612,8 +1801,8 @@ function applyTicketReviewNoteAppendedEvent(input: { } const body = - readOptionalStringPayload({ value: input.event.payload.body }) ?? - readOptionalStringPayload({ value: input.event.payload.markdown }); + readOptionalTextPayload({ value: input.event.payload.body }) ?? + readOptionalTextPayload({ value: input.event.payload.markdown }); if (!body) { return; } @@ -1952,6 +2141,15 @@ function readOptionalStringPayload(input: { return trimmed.length > 0 ? trimmed : undefined; } +function readOptionalTextPayload(input: { + readonly value: unknown; +}): string | undefined { + if (typeof input.value !== "string") { + return undefined; + } + return input.value.trim().length > 0 ? input.value : undefined; +} + function normalizeOwnerOrSource(input: { readonly value: string | undefined; readonly fallback: string; diff --git a/tests/tickets-extension.test.ts b/tests/tickets-extension.test.ts index e85e863cf..4479920bf 100644 --- a/tests/tickets-extension.test.ts +++ b/tests/tickets-extension.test.ts @@ -312,6 +312,258 @@ testIntegration( } ); +testIntegration( + "tickets extension: document command appends spec docs and updates the active description", + { timeout: 60_000 }, + async () => { + const root = await mkdirTempDir({ prefix: "hack-cli-tickets-docs-e2e-" }); + const projectDir = join(root, "project"); + + await mkdir(projectDir, { recursive: true }); + await copyDir({ + from: resolve(import.meta.dir, "../examples/tickets"), + to: projectDir, + }); + + await run({ cwd: projectDir, cmd: ["git", "init"] }); + await run({ + cwd: projectDir, + cmd: ["git", "config", "user.email", "tests@hack"], + }); + await run({ + cwd: projectDir, + cmd: ["git", "config", "user.name", "hack-cli-tests"], + }); + await run({ cwd: projectDir, cmd: ["git", "add", "-A"] }); + await run({ cwd: projectDir, cmd: ["git", "commit", "-m", "init"] }); + + const created = await runHack({ + cwd: projectDir, + args: [ + "tickets", + "create", + "--title", + "Document lifecycle", + "--body", + "## Context\nInitial description", + "--json", + ], + }); + const createdJson = JSON.parse(created.stdout) as { + ticket: { ticketId: string }; + }; + const ticketId = createdJson.ticket.ticketId; + + const spec = await runHack({ + cwd: projectDir, + args: [ + "tickets", + "document", + ticketId, + "--kind", + "spec", + "--body", + "## Goals\n- Add spec support", + "--json", + ], + }); + const specJson = JSON.parse(spec.stdout) as { + document: { kind: string; role: string }; + }; + expect(specJson.document.kind).toBe("spec"); + expect(specJson.document.role).toBe("spec"); + + const description = await runHack({ + cwd: projectDir, + args: [ + "tickets", + "document", + ticketId, + "--kind", + "description", + "--body", + "## Context\nUpdated description", + "--json", + ], + }); + const descriptionJson = JSON.parse(description.stdout) as { + document: { kind: string; content: string }; + }; + expect(descriptionJson.document.kind).toBe("description"); + expect(descriptionJson.document.content).toBe( + "## Context\nUpdated description" + ); + + const shown = await runHack({ + cwd: projectDir, + args: ["tickets", "show", ticketId, "--json"], + }); + const showJson = JSON.parse(shown.stdout) as { + ticket: { body?: string }; + documents: Array<{ kind: string; role: string; content: string }>; + }; + expect(showJson.ticket.body).toBe("## Context\nUpdated description"); + expect(showJson.documents).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + kind: "description", + role: "description", + content: "## Context\nInitial description", + }), + expect.objectContaining({ + kind: "spec", + role: "spec", + content: "## Goals\n- Add spec support", + }), + expect.objectContaining({ + kind: "description", + role: "description", + content: "## Context\nUpdated description", + }), + ]) + ); + + await rm(root, { recursive: true, force: true }); + } +); + +testIntegration( + "tickets extension: hidden ref sync transports only the journal and peers rebuild projection state locally", + { timeout: 60_000 }, + async () => { + const root = await mkdirTempDir({ + prefix: "hack-cli-tickets-portability-e2e-", + }); + const authorDir = join(root, "author"); + const peerDir = join(root, "peer"); + const remoteDir = join(root, "remote.git"); + + await mkdir(authorDir, { recursive: true }); + await copyDir({ + from: resolve(import.meta.dir, "../examples/tickets"), + to: authorDir, + }); + + await run({ cwd: authorDir, cmd: ["git", "init"] }); + await run({ + cwd: authorDir, + cmd: ["git", "config", "user.email", "tests@hack"], + }); + await run({ + cwd: authorDir, + cmd: ["git", "config", "user.name", "hack-cli-tests"], + }); + await run({ cwd: authorDir, cmd: ["git", "add", "-A"] }); + await run({ cwd: authorDir, cmd: ["git", "commit", "-m", "init"] }); + + await run({ cwd: root, cmd: ["git", "init", "--bare", remoteDir] }); + await run({ + cwd: authorDir, + cmd: ["git", "remote", "add", "origin", remoteDir], + }); + await run({ + cwd: authorDir, + cmd: ["git", "push", "-u", "origin", "HEAD:main"], + }); + + const created = await runHack({ + cwd: authorDir, + args: ["tickets", "create", "--title", "Portable journal", "--json"], + }); + const createdJson = JSON.parse(created.stdout) as { + ticket: { ticketId: string }; + }; + + const remoteTree = await run({ + cwd: root, + cmd: [ + "git", + `--git-dir=${remoteDir}`, + "ls-tree", + "-r", + "--name-only", + "refs/hack/tickets", + ], + }); + const remotePaths = remoteTree.stdout + .split("\n") + .map((line) => line.trim()) + .filter((line) => line.length > 0); + expect( + remotePaths.some((path) => + path.startsWith(".hack/tickets/events/events-") + ) + ).toBe(true); + expect(remotePaths).not.toContain(".hack/tickets/projection.sqlite"); + + await mkdir(peerDir, { recursive: true }); + await copyDir({ + from: resolve(import.meta.dir, "../examples/tickets"), + to: peerDir, + }); + await run({ cwd: peerDir, cmd: ["git", "init"] }); + await run({ + cwd: peerDir, + cmd: ["git", "config", "user.email", "tests@hack"], + }); + await run({ + cwd: peerDir, + cmd: ["git", "config", "user.name", "hack-cli-tests"], + }); + await run({ cwd: peerDir, cmd: ["git", "add", "-A"] }); + await run({ cwd: peerDir, cmd: ["git", "commit", "-m", "init"] }); + await run({ + cwd: peerDir, + cmd: ["git", "remote", "add", "origin", remoteDir], + }); + + const synced = await runHack({ + cwd: peerDir, + args: ["tickets", "sync", "--json"], + }); + expect(synced.exitCode).toBe(0); + + const shown = await runHack({ + cwd: peerDir, + args: ["tickets", "show", createdJson.ticket.ticketId, "--json"], + }); + expect(shown.exitCode).toBe(0); + + const eventsDir = join( + peerDir, + ".hack/tickets/git/worktree/.hack/tickets/events" + ); + const eventFiles = (await readdir(eventsDir)) + .filter((entry) => entry.endsWith(".jsonl")) + .sort(); + expect(eventFiles.length).toBeGreaterThan(0); + const journalBeforeDeletion = await readFile( + join(eventsDir, eventFiles[0] ?? ""), + "utf8" + ); + + const projectionPath = join(peerDir, ".hack/tickets/projection.sqlite"); + expect(await Bun.file(projectionPath).exists()).toBe(true); + await rm(projectionPath, { force: true }); + expect(await Bun.file(projectionPath).exists()).toBe(false); + + const rebuilt = await runHack({ + cwd: peerDir, + args: ["tickets", "list", "--json"], + }); + expect(rebuilt.exitCode).toBe(0); + expect(await Bun.file(projectionPath).exists()).toBe(true); + + const journalAfterDeletion = await readFile( + join(eventsDir, eventFiles[0] ?? ""), + "utf8" + ); + expect(journalAfterDeletion).toBe(journalBeforeDeletion); + + await rm(root, { recursive: true, force: true }); + } +); + interface RunResult { readonly stdout: string; readonly stderr: string; diff --git a/tests/tickets-store.test.ts b/tests/tickets-store.test.ts index becf0d088..86c4baf20 100644 --- a/tests/tickets-store.test.ts +++ b/tests/tickets-store.test.ts @@ -500,6 +500,73 @@ test("tickets store ignores duplicate sync checkpoints with the same idempotency ).toHaveLength(1); }); +test("tickets store records immutable documents and projects body from the active description", async () => { + const projectRoot = await createTempGitProject({ + prefix: "hack-cli-tickets-documents-", + }); + const store = await createStore({ projectRoot }); + + const created = await store.createTicket({ + title: "Document-backed ticket", + body: "## Context\nInitial description", + owner: "hack", + source: "hack", + actor: "creator@hack", + }); + expect(created.ok).toBe(true); + if (!created.ok) { + throw new Error(created.error); + } + + const spec = await store.appendDocument({ + ticketId: created.ticket.ticketId, + kind: "spec", + content: "\n## Goals\n- Ship immutable ticket documents", + actor: "author@hack", + }); + expect(spec.ok).toBe(true); + if (!spec.ok) { + throw new Error(spec.error); + } + + const description = await store.appendDocument({ + ticketId: created.ticket.ticketId, + kind: "description", + content: "## Context\nUpdated description", + actor: "author@hack", + }); + expect(description.ok).toBe(true); + if (!description.ok) { + throw new Error(description.error); + } + + const detail = await store.getTicketDetail({ + ticketId: created.ticket.ticketId, + }); + + expect(detail.ticket?.body).toBe("## Context\nUpdated description"); + expect(detail.documents).toEqual([ + expect.objectContaining({ + kind: "description", + role: "description", + content: "## Context\nInitial description", + }), + expect.objectContaining({ + kind: "spec", + role: "spec", + content: "\n## Goals\n- Ship immutable ticket documents", + }), + expect.objectContaining({ + kind: "description", + role: "description", + content: "## Context\nUpdated description", + }), + ]); + expect( + detail.events.filter((event) => event.type === "ticket.document_recorded") + ).toHaveLength(2); +}, 20_000); + test("tickets store persists a sqlite projection and rebuilds it when deleted", async () => { const projectRoot = await createTempGitProject({ prefix: "hack-cli-tickets-projection-", From 362e2565c1b83f3ca6f090074426b4f8541b3ad2 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sat, 14 Mar 2026 04:00:55 +0000 Subject: [PATCH 012/134] docs: define prerequisite detection matrix --- docs/README.md | 1 + docs/guides/prerequisite-detection-matrix.md | 159 +++++ src/cli/prerequisites.ts | 628 +++++++++++++++++++ tests/prerequisites-matrix.test.ts | 209 ++++++ 4 files changed, 997 insertions(+) create mode 100644 docs/guides/prerequisite-detection-matrix.md create mode 100644 src/cli/prerequisites.ts create mode 100644 tests/prerequisites-matrix.test.ts diff --git a/docs/README.md b/docs/README.md index e493be242..973c47a74 100644 --- a/docs/README.md +++ b/docs/README.md @@ -34,6 +34,7 @@ Quick diagnostics: - Remote setup (one command): `hack remote setup` - [Initialize a project](guides/init-project.md) - [Global settings](guides/global-settings.md) +- [Prerequisite detection matrix](guides/prerequisite-detection-matrix.md) - [Tickets (git-backed)](guides/tickets.md) - [Expose the gateway over SSH](guides/remote-ssh.md) - [Expose the gateway with Cloudflare](guides/remote-cloudflare.md) diff --git a/docs/guides/prerequisite-detection-matrix.md b/docs/guides/prerequisite-detection-matrix.md new file mode 100644 index 000000000..10f72e2bd --- /dev/null +++ b/docs/guides/prerequisite-detection-matrix.md @@ -0,0 +1,159 @@ +# Prerequisite Detection Matrix + +This guide defines the prerequisite contract for guided interception in the CLI. +The implementation-facing source of truth lives in `src/cli/prerequisites.ts`. + +## Scope + +This matrix covers the prerequisite families called out in HACK-441: + +- Docker runtime +- Global install state +- tmux/zellij mux availability +- GitHub auth/config +- Linear auth/config + +The goal is not to make every command fail earlier. The goal is to decide, in a repeatable way, whether a missing prerequisite should: + +- block: stop and show the missing prerequisite inline +- guide: intercept into a repair/setup path +- warn: continue and surface degraded or diagnostic state + +## Check Catalog + +| Check ID | Domain | What it means | Primary guidance | +| --- | --- | --- | --- | +| `docker_cli` | Docker | `docker` is installed and on `PATH` | Install Docker Desktop or OrbStack | +| `docker_daemon` | Docker | `docker info` succeeds | Start Docker or route into `hack global install` | +| `global_bootstrap` | Global | `hack global install` has generated `~/.hack` artifacts | `hack global install` | +| `global_services` | Global | Caddy/CoreDNS/Loki are reachable for routed hosts, Loki logs, or Caddy-managed CA export | `hack global up` or `hack global install` | +| `mux_backend` | Mux | `sessions.mux` resolves to an installed tmux or zellij backend | Install tmux/zellij or change `sessions.mux` | +| `tmux_binary` | Mux | tmux exists for tmux-backed session commands | Install tmux directly, then optionally run `hack setup tmux` | +| `github_profile` | GitHub | A GitHub profile is resolvable from flags/config | `hack x github connect` or `hack x github use` | +| `github_token` | GitHub | The selected GitHub profile resolves a usable token | `hack x github connect` or `hack x github oauth-connect` | +| `github_gh_cli` | GitHub | `gh` exists for browser bootstrap | Install GitHub CLI | +| `linear_profile` | Linear | A Linear profile is resolvable from flags/config | `hack linear connect` or `hack linear use` | +| `linear_token` | Linear | The selected Linear profile resolves a usable token | `hack linear connect` or `hack linear oauth-connect` | +| `linear_broker_auth` | Linear | A Hack broker management token exists for broker-owned flows | `hack auth login` | +| `linear_oauth_client` | Linear | Local OAuth client credentials exist for direct OAuth fallback | Configure local Linear OAuth env/auth-ref values | + +## Command Matrix + +| Command(s) | Checks | Missing behavior | Notes | +| --- | --- | --- | --- | +| `hack global install` | `docker_cli`, `docker_daemon`, `mux_backend` | guide, guide, warn | Bootstrap should repair Docker issues inline, but only warn about missing mux tooling | +| `hack global up`, `hack global status`, `hack global logs`, `hack global logs-reset` | `global_bootstrap`, `docker_cli`, `docker_daemon` | guide, guide, guide | These commands operate on the generated global stack | +| `hack global down` | `docker_cli`, `docker_daemon` | guide, guide | Conditional: only when a managed global stack already exists; otherwise teardown should stay idempotent | +| `hack global ca`, `hack global trust` | `global_bootstrap`, `docker_cli`, `docker_daemon`, `global_services` | guide, guide, guide, guide | CA export/trust depends on an installed and running Caddy-managed global stack | +| `hack up`, `hack down`, `hack restart`, `hack ps`, `hack run`, `hack tui` | `docker_cli`, `docker_daemon` | guide, guide | Core Docker-backed runtime commands | +| `hack up`, `hack restart` | `mux_backend` | guide | Only when lifecycle host processes or mux-backed pre-start hooks are configured | +| `hack open` | `global_services` | guide | Routed `*.hack` access should repair global routing instead of failing opaquely | +| `hack logs` | `docker_cli`, `docker_daemon`, `global_services` | guide, guide, warn | Default logs may fall back from Loki to compose logs | +| `hack logs --loki`, `hack logs --query` | `global_services` | guide | Explicit Loki mode should not silently fall back | +| `hack session`, `hack session list`, `hack session start`, `hack session attach`, `hack session exec`, `hack session stop`, `hack ssh` | `tmux_binary` | guide | The current session and session-aware SSH implementation is tmux-backed even when mux config exists elsewhere | +| `hack session panes`, `hack session capture`, `hack session tail`, `hack setup tmux` | `tmux_binary` | guide | tmux-only surfaces | +| `hack x github status`, `hack x github profiles` | `github_profile`, `github_token` | warn, warn | Diagnostics should expose missing state instead of intercepting it | +| `hack x github use` | `github_profile` | guide | Profile selection should redirect into setup when there is no resolvable target | +| `hack x github oauth-connect` | `github_gh_cli` | guide | Browser bootstrap is `gh`-driven | +| `hack x github pr-upsert` | `github_profile`, `github_token` | guide, guide | Mutating GitHub API command | +| `hack linear status`, `hack linear profiles` | `linear_profile`, `linear_token` | warn, warn | Diagnostics should keep unresolved state visible | +| `hack linear use` | `linear_profile` | guide | Profile selection should redirect into setup when there is no resolvable target | +| `hack linear connect` | `linear_broker_auth`, `linear_oauth_client` | guide, guide | Conditional: broker auth only when broker OAuth is selected and local OAuth fallback is unavailable; direct OAuth config only when fallback is the chosen path | +| `hack linear oauth-connect` | `linear_broker_auth`, `linear_oauth_client` | guide, guide | Same split as `hack linear connect`, but on the explicit OAuth path | +| `hack linear connections`, `hack linear seed-local-access`, `hack linear deliveries`, `hack linear apply-delivery`, `hack linear subscriptions`, `hack linear set-subscription`, `hack linear remove-subscription` | `linear_broker_auth` | guide | These commands are broker-backed and should route into Hack auth first | +| `hack linear projects`, `hack linear sync-issue`, `hack linear sync-project` | `linear_profile`, `linear_token` | guide, guide | Linear API actions should repair auth/config before invoking the API | +| `hack linear project-bind`, `hack linear project-link` | `linear_profile`, `linear_token` | guide, guide | Conditional: only when the command must resolve project metadata remotely instead of using a fully specified local binding | +| `hack linear run-autosync` | `linear_profile`, `linear_token`, `linear_broker_auth` | guide, guide, guide | Autosync needs both Linear API access and broker-backed subscription/delivery access | + +## Decision Rules + +### Use `guide` + +Use guided interception when the command is operational and the missing prerequisite is something Hack can clearly repair or route the user toward: + +- Docker is missing or not running for Docker-backed workflows +- `hack global install` has never been run +- Routed global services are missing for `hack open` or explicit Loki commands +- tmux is missing for the current session and SSH implementation, or mux is missing for lifecycle-backed host-process flows +- GitHub or Linear action commands are missing resolvable profile/token state +- broker-owned Linear flows are missing Hack account auth + +### Use `warn` + +Use a warning when the command is itself diagnostic or the CLI can still continue in a degraded but useful mode: + +- `hack global install` with missing tmux/zellij +- `hack logs` when Loki is unavailable but compose fallback still works +- `hack x github status` and `hack x github profiles` +- `hack linear status` and `hack linear profiles` + +### Use `block` + +The current matrix leaves `block` for cases where guided repair is not appropriate, but the ticket scope does not require a first blocking case. The important contract is that operational commands guide, diagnostics warn, and only irreparable/unowned failures should block later. + +## Commands That Should Invoke Checks + +The explicit command set currently covered by the source-of-truth matrix is: + +```text +global install +global up +global status +global logs +global logs-reset +global down +global ca +global trust +up +down +restart +ps +run +tui +open +logs +logs --loki +logs --query +session +session list +session start +session attach +session exec +session stop +ssh +session panes +session capture +session tail +setup tmux +x github status +x github profiles +x github use +x github oauth-connect +x github pr-upsert +linear status +linear profiles +linear use +linear connect +linear oauth-connect +linear connections +linear seed-local-access +linear projects +linear project-bind +linear project-link +linear sync-issue +linear sync-project +linear run-autosync +linear deliveries +linear apply-delivery +linear subscriptions +linear set-subscription +linear remove-subscription +``` + +## Notes for Follow-on Implementation + +- Preserve the current distinction between diagnostics and actions. Status commands should not disappear behind setup wizards. +- Keep conditional checks conditional. `mux_backend` only matters for lifecycle-backed `hack up`/`hack restart` when lifecycle host processes are configured. +- Keep cleanup and local-only config commands out of shared interception. `x github disconnect`, `linear disconnect`, `linear assignee-mappings`, and `linear project-unlink` should use command-local validation instead of the prerequisite matrix. +- Docker detection should distinguish CLI missing from daemon unavailable because the repair path is different. +- GitHub and Linear setup flows already contain parts of this guidance. Interception should centralize the decision, not duplicate every downstream error string. diff --git a/src/cli/prerequisites.ts b/src/cli/prerequisites.ts new file mode 100644 index 000000000..331e39d22 --- /dev/null +++ b/src/cli/prerequisites.ts @@ -0,0 +1,628 @@ +export type PrerequisitePolicy = "block" | "guide" | "warn"; + +export type PrerequisiteCheckId = + | "docker_cli" + | "docker_daemon" + | "global_bootstrap" + | "global_services" + | "mux_backend" + | "tmux_binary" + | "github_profile" + | "github_token" + | "github_gh_cli" + | "linear_profile" + | "linear_token" + | "linear_broker_auth" + | "linear_oauth_client"; + +export type PrerequisiteDomain = + | "docker" + | "global" + | "mux" + | "github" + | "linear"; + +/** + * Stable metadata for a missing-prerequisite check. + */ +export type PrerequisiteCheckDefinition = { + readonly id: PrerequisiteCheckId; + readonly domain: PrerequisiteDomain; + readonly summary: string; + readonly detection: string; + readonly primaryGuidance: string; +}; + +/** + * A concrete prerequisite rule for a user-facing command. + */ +export type CommandPrerequisiteRule = { + readonly checkId: PrerequisiteCheckId; + readonly onMissing: PrerequisitePolicy; + readonly reason: string; + readonly when?: string; +}; + +/** + * A user-facing command or conditional command variant that should invoke checks. + */ +export type CommandPrerequisiteContract = { + readonly name: string; + readonly commands: readonly string[]; + readonly rules: readonly CommandPrerequisiteRule[]; +}; + +export const PREREQUISITE_CHECKS = [ + { + id: "docker_cli", + domain: "docker", + summary: "Docker CLI is installed and on PATH", + detection: + "Resolve `docker` from PATH before any Docker-backed command path.", + primaryGuidance: + "Install Docker Desktop or OrbStack before retrying the command.", + }, + { + id: "docker_daemon", + domain: "docker", + summary: "Docker engine is reachable", + detection: + "Run `docker info`; distinguish an installed CLI from a stopped or unreachable daemon.", + primaryGuidance: + "Offer to start the detected backend when possible, otherwise point to `hack global install` or the platform Docker app/service.", + }, + { + id: "global_bootstrap", + domain: "global", + summary: + "`hack global install` has generated the managed ~/.hack stack files", + detection: + "Require generated compose/config/schema artifacts under `~/.hack` before commands that operate on the global stack.", + primaryGuidance: "Launch `hack global install`.", + }, + { + id: "global_services", + domain: "global", + summary: + "Global routing, logging, and Caddy-managed services are reachable", + detection: + "Check Caddy/CoreDNS/Loki reachability when a command relies on `*.hack`, internal DNS, Loki-backed logs, or Caddy-managed CA export.", + primaryGuidance: + "Launch `hack global up` when bootstrap exists, otherwise fall back to `hack global install`.", + }, + { + id: "mux_backend", + domain: "mux", + summary: + "The configured `sessions.mux` backend resolves to an installed tmux or zellij binary", + detection: + "Read `sessions.mux`, then verify the selected backend in PATH; in `auto`, accept either tmux or zellij.", + primaryGuidance: + "Install tmux or zellij, or change `sessions.mux` to a valid value.", + }, + { + id: "tmux_binary", + domain: "mux", + summary: "tmux is available for tmux-only session features", + detection: + "Resolve `tmux` from PATH before tmux-specific commands like pane inspection or capture.", + primaryGuidance: + "Install tmux directly. After tmux is available, `hack setup tmux` can configure the picker keybinding.", + }, + { + id: "github_profile", + domain: "github", + summary: + "A GitHub profile is explicitly selected or resolvable from config", + detection: + "Resolve profile selection using command flag, project override, global default, then implicit default.", + primaryGuidance: + "Launch `hack x github connect --profile ` or choose a profile with `hack x github use --profile `.", + }, + { + id: "github_token", + domain: "github", + summary: "The selected GitHub profile can resolve a usable token", + detection: + "Resolve token from keychain, refresh flow, or env fallback before invoking GitHub API operations.", + primaryGuidance: + "Launch `hack x github connect --profile ` or rerun `hack x github oauth-connect --profile `.", + }, + { + id: "github_gh_cli", + domain: "github", + summary: "`gh` is installed for GitHub browser bootstrap", + detection: + "Resolve `gh` from PATH before `oauth-connect`; optionally preflight `gh auth status` once the CLI is present.", + primaryGuidance: + "Install GitHub CLI and rerun `hack x github oauth-connect`.", + }, + { + id: "linear_profile", + domain: "linear", + summary: + "A Linear profile is explicitly selected or resolvable from config", + detection: + "Resolve profile selection using command flag, project override, global default, then implicit default.", + primaryGuidance: + "Launch `hack linear connect --profile ` or choose a profile with `hack linear use --profile `.", + }, + { + id: "linear_token", + domain: "linear", + summary: "The selected Linear profile can resolve a usable access token", + detection: + "Resolve token from keychain, broker refresh, or env fallback before Linear API operations.", + primaryGuidance: + "Launch `hack linear connect --profile ` or rerun `hack linear oauth-connect --profile `.", + }, + { + id: "linear_broker_auth", + domain: "linear", + summary: + "A Hack broker management token is available for broker-owned Linear flows", + detection: + "Require the local Hack auth session before broker-owned OAuth bootstrap or token refresh.", + primaryGuidance: "Launch `hack auth login`.", + }, + { + id: "linear_oauth_client", + domain: "linear", + summary: + "Local Linear OAuth client credentials are configured for direct OAuth fallback", + detection: + "Require local Linear OAuth client id and secret when using the direct OAuth path instead of the broker.", + primaryGuidance: + "Configure the local Linear OAuth env/auth-ref values, then rerun `hack linear oauth-connect`.", + }, +] as const satisfies readonly PrerequisiteCheckDefinition[]; + +export const COMMAND_PREREQUISITE_CONTRACTS = [ + { + name: "Global bootstrap", + commands: ["global install"], + rules: [ + { + checkId: "docker_cli", + onMissing: "guide", + reason: + "Global bootstrap cannot create or start the managed stack without Docker.", + }, + { + checkId: "docker_daemon", + onMissing: "guide", + reason: + "This command should intercept into backend start guidance instead of failing on `docker info`.", + }, + { + checkId: "mux_backend", + onMissing: "warn", + reason: + "Global bootstrap should complete even if session management will be degraded afterward.", + }, + ], + }, + { + name: "Global stack lifecycle", + commands: [ + "global up", + "global status", + "global logs", + "global logs-reset", + ], + rules: [ + { + checkId: "global_bootstrap", + onMissing: "guide", + reason: + "These commands operate on the generated global stack and should redirect to install when the stack was never bootstrapped.", + }, + { + checkId: "docker_cli", + onMissing: "guide", + reason: "The global stack is Docker-managed.", + }, + { + checkId: "docker_daemon", + onMissing: "guide", + reason: + "These commands should guide users into starting Docker rather than surfacing raw engine errors.", + }, + ], + }, + { + name: "Global stack teardown", + commands: ["global down"], + rules: [ + { + checkId: "docker_cli", + onMissing: "guide", + reason: + "Tearing down an existing global stack still uses Docker-managed resources.", + when: "Only when the managed global stack has already been bootstrapped; otherwise the command should remain idempotent.", + }, + { + checkId: "docker_daemon", + onMissing: "guide", + reason: + "Stopping an existing global stack should guide users into starting Docker rather than surfacing raw engine errors.", + when: "Only when the managed global stack has already been bootstrapped; otherwise the command should remain idempotent.", + }, + ], + }, + { + name: "Global CA export and trust", + commands: ["global ca", "global trust"], + rules: [ + { + checkId: "global_bootstrap", + onMissing: "guide", + reason: + "CA export and trust flows depend on the managed global stack having been installed at least once.", + }, + { + checkId: "docker_cli", + onMissing: "guide", + reason: + "These commands extract the Caddy-managed certificate authority from Docker-managed global services.", + }, + { + checkId: "docker_daemon", + onMissing: "guide", + reason: + "CA export and trust should repair Docker availability before attempting to inspect or copy certificates.", + }, + { + checkId: "global_services", + onMissing: "guide", + reason: + "The Caddy-managed global services need to be running before Hack can export or trust the generated CA.", + }, + ], + }, + { + name: "Project runtime lifecycle", + commands: ["up", "down", "restart", "ps", "run", "tui"], + rules: [ + { + checkId: "docker_cli", + onMissing: "guide", + reason: + "These commands are fundamentally Docker-backed project operations.", + }, + { + checkId: "docker_daemon", + onMissing: "guide", + reason: + "The preferred first-run behavior is guided Docker startup rather than a hard failure.", + }, + ], + }, + { + name: "Project lifecycle host processes", + commands: ["up", "restart"], + rules: [ + { + checkId: "mux_backend", + onMissing: "guide", + reason: + "Lifecycle host processes use the configured mux backend and should fail fast with an actionable setup path.", + when: "Only when the project config defines lifecycle host processes or pre-start hooks that require a mux session.", + }, + ], + }, + { + name: "Project DNS and routed open", + commands: ["open"], + rules: [ + { + checkId: "global_services", + onMissing: "guide", + reason: + "Opening `https://.hack` without Caddy/CoreDNS reachability is usually a setup gap, not a user error.", + }, + ], + }, + { + name: "Project logs with fallback", + commands: ["logs"], + rules: [ + { + checkId: "docker_cli", + onMissing: "guide", + reason: + "The compose log backend remains the local fallback and still needs Docker available.", + }, + { + checkId: "docker_daemon", + onMissing: "guide", + reason: + "Without a reachable daemon there is no compose fallback path for logs.", + }, + { + checkId: "global_services", + onMissing: "warn", + reason: + "Default logs can degrade to compose output, so missing Loki/Caddy should warn instead of blocking.", + when: "Only for the default logs path when the command is allowed to fall back from Loki to compose logs.", + }, + ], + }, + { + name: "Project logs without fallback", + commands: ["logs --loki", "logs --query"], + rules: [ + { + checkId: "global_services", + onMissing: "guide", + reason: + "An explicit Loki request should not silently fall back; it should route into global logging setup.", + }, + ], + }, + { + name: "Session commands", + commands: [ + "session", + "session list", + "session start", + "session attach", + "session exec", + "session stop", + "ssh", + ], + rules: [ + { + checkId: "tmux_binary", + onMissing: "guide", + reason: + "The current session and SSH flows shell out to tmux directly, so they should repair tmux availability before continuing.", + }, + ], + }, + { + name: "tmux-only session commands", + commands: [ + "session panes", + "session capture", + "session tail", + "setup tmux", + ], + rules: [ + { + checkId: "tmux_binary", + onMissing: "guide", + reason: + "These commands are tmux-specific and should launch tmux setup guidance instead of failing later.", + }, + ], + }, + { + name: "GitHub diagnostics", + commands: ["x github status", "x github profiles"], + rules: [ + { + checkId: "github_profile", + onMissing: "warn", + reason: + "Status-style commands exist to expose missing GitHub configuration, so they should report the gap instead of intercepting it.", + }, + { + checkId: "github_token", + onMissing: "warn", + reason: + "Token resolution failures should remain visible in diagnostics output rather than being hidden behind setup interception.", + }, + ], + }, + { + name: "GitHub profile selection", + commands: ["x github use"], + rules: [ + { + checkId: "github_profile", + onMissing: "guide", + reason: + "Changing or removing a profile without a resolvable target should redirect into profile setup.", + }, + ], + }, + { + name: "GitHub OAuth bootstrap", + commands: ["x github oauth-connect"], + rules: [ + { + checkId: "github_gh_cli", + onMissing: "guide", + reason: + "GitHub browser bootstrap is built around the `gh` CLI and should intercept before spawning it.", + }, + ], + }, + { + name: "GitHub API actions", + commands: ["x github pr-upsert"], + rules: [ + { + checkId: "github_profile", + onMissing: "guide", + reason: + "PR automation needs a selected GitHub identity before it can talk to the API.", + }, + { + checkId: "github_token", + onMissing: "guide", + reason: + "This is a mutating GitHub API command and should route directly into token setup when auth is missing.", + }, + ], + }, + { + name: "Linear diagnostics", + commands: ["linear status", "linear profiles"], + rules: [ + { + checkId: "linear_profile", + onMissing: "warn", + reason: + "Status-style commands should show unresolved Linear profile state instead of intercepting it.", + }, + { + checkId: "linear_token", + onMissing: "warn", + reason: + "Users run these commands specifically to inspect missing token state, so guidance belongs in the output rather than a redirect.", + }, + ], + }, + { + name: "Linear profile selection", + commands: ["linear use"], + rules: [ + { + checkId: "linear_profile", + onMissing: "guide", + reason: + "Changing or removing a profile without a resolvable target should redirect into profile setup.", + }, + ], + }, + { + name: "Linear connect", + commands: ["linear connect"], + rules: [ + { + checkId: "linear_broker_auth", + onMissing: "guide", + reason: + "When connect falls into a broker-owned OAuth flow, it should send the user to Hack account login first.", + when: "Only when no direct token input is supplied, the command resolves to the broker-owned OAuth path, and local OAuth fallback is not configured.", + }, + { + checkId: "linear_oauth_client", + onMissing: "guide", + reason: + "Local direct OAuth should not start until the client id and secret are configured.", + when: "Only when no direct token input is supplied and the command resolves to the local OAuth fallback path after broker auth is unavailable or skipped.", + }, + ], + }, + { + name: "Linear OAuth bootstrap", + commands: ["linear oauth-connect"], + rules: [ + { + checkId: "linear_broker_auth", + onMissing: "guide", + reason: + "Broker-owned Linear connection flows should launch Hack account login instead of surfacing auth-broker errors.", + when: "Only when the broker-owned OAuth path is selected and local OAuth fallback is not configured.", + }, + { + checkId: "linear_oauth_client", + onMissing: "guide", + reason: + "Direct OAuth fallback should intercept before opening the browser if local credentials are missing.", + when: "Only when the local OAuth fallback path is selected after broker auth is unavailable or skipped.", + }, + ], + }, + { + name: "Linear broker-backed setup and delivery commands", + commands: [ + "linear connections", + "linear seed-local-access", + "linear deliveries", + "linear apply-delivery", + "linear subscriptions", + "linear set-subscription", + "linear remove-subscription", + ], + rules: [ + { + checkId: "linear_broker_auth", + onMissing: "guide", + reason: + "These commands are broker-backed and should route into Hack account auth before they attempt broker operations.", + }, + ], + }, + { + name: "Linear API actions", + commands: ["linear projects", "linear sync-issue", "linear sync-project"], + rules: [ + { + checkId: "linear_profile", + onMissing: "guide", + reason: + "These commands depend on a resolved Linear profile or project binding before they can talk to Linear.", + }, + { + checkId: "linear_token", + onMissing: "guide", + reason: + "These are action commands and should route directly into connection repair when access tokens are missing.", + }, + ], + }, + { + name: "Linear project binding with remote resolution", + commands: ["linear project-bind", "linear project-link"], + rules: [ + { + checkId: "linear_profile", + onMissing: "guide", + reason: + "Project binding needs a resolved Linear profile whenever it must look up project metadata remotely.", + when: "Only when the command must resolve project metadata from Linear instead of using a fully specified local binding.", + }, + { + checkId: "linear_token", + onMissing: "guide", + reason: + "Remote project lookup should route directly into token repair when Linear API access is missing.", + when: "Only when the command must resolve project metadata from Linear instead of using a fully specified local binding.", + }, + ], + }, + { + name: "Linear autosync runtime", + commands: ["linear run-autosync"], + rules: [ + { + checkId: "linear_profile", + onMissing: "guide", + reason: + "Autosync processing needs a resolved Linear profile once it begins working against a configured route.", + when: "Only when a target project route is being processed.", + }, + { + checkId: "linear_token", + onMissing: "guide", + reason: + "Autosync should repair missing Linear API access before it starts the sync runtime.", + when: "Only when the runtime needs to talk to the Linear API for the selected route.", + }, + { + checkId: "linear_broker_auth", + onMissing: "guide", + reason: + "Autosync also depends on broker-backed subscription and delivery flows, so broker auth should repair up front.", + }, + ], + }, +] as const satisfies readonly CommandPrerequisiteContract[]; + +export const COMMANDS_THAT_INVOKE_PREREQUISITE_CHECKS = [ + ...new Set( + COMMAND_PREREQUISITE_CONTRACTS.flatMap((contract) => contract.commands) + ), +] as const; + +export function getCommandPrerequisiteContracts(input: { + readonly command: string; +}): readonly CommandPrerequisiteContract[] { + return COMMAND_PREREQUISITE_CONTRACTS.filter((contract) => + contract.commands.includes(input.command) + ); +} diff --git a/tests/prerequisites-matrix.test.ts b/tests/prerequisites-matrix.test.ts new file mode 100644 index 000000000..40322cbd1 --- /dev/null +++ b/tests/prerequisites-matrix.test.ts @@ -0,0 +1,209 @@ +import { expect, test } from "bun:test"; + +import { + COMMAND_PREREQUISITE_CONTRACTS, + COMMANDS_THAT_INVOKE_PREREQUISITE_CHECKS, + getCommandPrerequisiteContracts, + PREREQUISITE_CHECKS, +} from "../src/cli/prerequisites.ts"; + +test("every prerequisite check is referenced by at least one command contract", () => { + const referencedChecks = new Set( + COMMAND_PREREQUISITE_CONTRACTS.flatMap((contract) => + contract.rules.map((rule) => rule.checkId) + ) + ); + + for (const check of PREREQUISITE_CHECKS) { + expect(referencedChecks.has(check.id)).toBe(true); + } +}); + +test("command coverage includes the major first-run and integration paths", () => { + expect(COMMANDS_THAT_INVOKE_PREREQUISITE_CHECKS).toEqual([ + "global install", + "global up", + "global status", + "global logs", + "global logs-reset", + "global down", + "global ca", + "global trust", + "up", + "down", + "restart", + "ps", + "run", + "tui", + "open", + "logs", + "logs --loki", + "logs --query", + "session", + "session list", + "session start", + "session attach", + "session exec", + "session stop", + "ssh", + "session panes", + "session capture", + "session tail", + "setup tmux", + "x github status", + "x github profiles", + "x github use", + "x github oauth-connect", + "x github pr-upsert", + "linear status", + "linear profiles", + "linear use", + "linear connect", + "linear oauth-connect", + "linear connections", + "linear seed-local-access", + "linear deliveries", + "linear apply-delivery", + "linear subscriptions", + "linear set-subscription", + "linear remove-subscription", + "linear projects", + "linear sync-issue", + "linear sync-project", + "linear project-bind", + "linear project-link", + "linear run-autosync", + ]); +}); + +test("global install guides Docker problems but only warns on mux availability", () => { + const [contract] = getCommandPrerequisiteContracts({ + command: "global install", + }); + expect(contract).toBeDefined(); + + const rulesByCheck = new Map( + contract?.rules.map((rule) => [rule.checkId, rule.onMissing]) + ); + + expect(rulesByCheck.get("docker_cli")).toBe("guide"); + expect(rulesByCheck.get("docker_daemon")).toBe("guide"); + expect(rulesByCheck.get("mux_backend")).toBe("warn"); +}); + +test("status-style auth commands warn instead of intercepting missing auth", () => { + const [githubStatus] = getCommandPrerequisiteContracts({ + command: "x github status", + }); + const [linearStatus] = getCommandPrerequisiteContracts({ + command: "linear status", + }); + + expect(githubStatus?.rules.every((rule) => rule.onMissing === "warn")).toBe( + true + ); + expect(linearStatus?.rules.every((rule) => rule.onMissing === "warn")).toBe( + true + ); +}); + +test("action commands escalate to guidance for missing integration auth", () => { + const [githubAction] = getCommandPrerequisiteContracts({ + command: "x github pr-upsert", + }); + const [linearAction] = getCommandPrerequisiteContracts({ + command: "linear sync-issue", + }); + + expect(githubAction?.rules.map((rule) => rule.onMissing)).toEqual([ + "guide", + "guide", + ]); + expect(linearAction?.rules.map((rule) => rule.onMissing)).toEqual([ + "guide", + "guide", + ]); +}); + +test("session commands require tmux instead of only resolving a mux backend", () => { + const [sessionStart] = getCommandPrerequisiteContracts({ + command: "session start", + }); + + expect(sessionStart?.rules.map((rule) => rule.checkId)).toEqual([ + "tmux_binary", + ]); +}); + +test("cleanup and local-config commands stay out of shared prerequisite interception", () => { + expect( + getCommandPrerequisiteContracts({ command: "x github disconnect" }) + ).toEqual([]); + expect( + getCommandPrerequisiteContracts({ command: "linear disconnect" }) + ).toEqual([]); + expect( + getCommandPrerequisiteContracts({ command: "linear assignee-mappings" }) + ).toEqual([]); + expect( + getCommandPrerequisiteContracts({ command: "linear project-unlink" }) + ).toEqual([]); +}); + +test("broker-backed linear commands guide through Hack auth", () => { + const [connections] = getCommandPrerequisiteContracts({ + command: "linear connections", + }); + const [deliveries] = getCommandPrerequisiteContracts({ + command: "linear deliveries", + }); + + expect(connections?.rules.map((rule) => rule.checkId)).toEqual([ + "linear_broker_auth", + ]); + expect(deliveries?.rules.map((rule) => rule.checkId)).toEqual([ + "linear_broker_auth", + ]); + expect(connections?.rules.map((rule) => rule.onMissing)).toEqual(["guide"]); + expect(deliveries?.rules.map((rule) => rule.onMissing)).toEqual(["guide"]); +}); + +test("conditional linear project binding rules only apply to remote lookup paths", () => { + const [projectBind] = getCommandPrerequisiteContracts({ + command: "linear project-bind", + }); + const [projectLink] = getCommandPrerequisiteContracts({ + command: "linear project-link", + }); + + expect(projectBind?.rules.map((rule) => rule.checkId)).toEqual([ + "linear_profile", + "linear_token", + ]); + expect(projectLink?.rules.map((rule) => rule.checkId)).toEqual([ + "linear_profile", + "linear_token", + ]); + expect(projectBind?.rules.every((rule) => rule.when)).toBe(true); + expect(projectLink?.rules.every((rule) => rule.when)).toBe(true); +}); + +test("conditional overlays remain addressable through multi-contract lookup", () => { + const upContracts = getCommandPrerequisiteContracts({ command: "up" }); + const restartContracts = getCommandPrerequisiteContracts({ + command: "restart", + }); + + expect(upContracts.length).toBe(2); + expect(restartContracts.length).toBe(2); + expect( + upContracts.flatMap((contract) => + contract.rules.map((rule) => rule.checkId) + ) + ).toEqual(["docker_cli", "docker_daemon", "mux_backend"]); + expect( + restartContracts.flatMap((contract) => + contract.rules.map((rule) => rule.checkId) + ) + ).toEqual(["docker_cli", "docker_daemon", "mux_backend"]); +}); From eaab6df623878f2bb9b5e577a67cc4f349c9f5e4 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sat, 14 Mar 2026 04:00:59 +0000 Subject: [PATCH 013/134] feat(env): define portable managed env artifact schema --- docs/env.md | 85 +++++ ...able-project-env-artifact-schema-design.md | 334 ++++++++++++++++++ ...rtable-project-env-artifact-schema-plan.md | 241 +++++++++++++ scripts/build-release.ts | 5 + src/commands/global.ts | 13 +- src/constants.ts | 2 + src/templates.ts | 64 ++++ tests/global-command.test.ts | 10 + tests/project-config-schema.test.ts | 53 ++- 9 files changed, 803 insertions(+), 4 deletions(-) create mode 100644 docs/plans/2026-03-13-portable-project-env-artifact-schema-design.md create mode 100644 docs/plans/2026-03-13-portable-project-env-artifact-schema-plan.md diff --git a/docs/env.md b/docs/env.md index 3314d7971..6dc012dd8 100644 --- a/docs/env.md +++ b/docs/env.md @@ -34,6 +34,7 @@ hack supports a project-scoped env contract (shareable, no values) plus safe sec ``` Fields: + - `key`: uppercase snake-case env var name (e.g. `AWS_PROFILE`). - `required`: if true, `hack up/run/restart` fails when missing (for targeted services). - `source`: @@ -42,6 +43,83 @@ Fields: - `services`: `null` (or omitted) means all services; otherwise a list of Compose service names. - `description`: optional, for humans/UI. +## Portable managed artifact + +Portable env management uses three layers: + +- `.hack/hack.env.json`: committed declaration contract for keys, required flags, service scope, and descriptions +- `PortableProjectEnvArtifactV1`: canonical managed-value artifact for portability, import/export, and future remote publish/apply flows +- local compatibility storage: `.hack/.env` for plaintext values plus the configured secret backend for secret values + +The managed artifact is intentionally separate from `.hack/hack.env.json`. The contract remains safe to commit. The managed artifact carries actual values plus intent metadata and should be encrypted as a whole when stored outside the local machine. + +Suggested import/export filename: + +- `.hack/hack.env.managed.json` +- schema URL: `https://schemas.hack/hack.env.managed.schema.json` + +Artifact shape: + +```json +{ + "$schema": "https://schemas.hack/hack.env.managed.schema.json", + "version": 1, + "environment": "default", + "metadata": { + "description": "Shared dev environment for Hack App", + "updatedAt": "2026-03-13T18:00:00Z", + "updatedBy": "cli", + "source": "hack-cli" + }, + "entries": [ + { + "key": "AWS_REGION", + "value": { + "kind": "plaintext", + "text": "us-east-1" + }, + "required": true, + "services": ["api"], + "description": "AWS region used by the API" + }, + { + "key": "DATABASE_URL", + "value": { + "kind": "secret", + "text": "postgres://..." + }, + "required": true, + "services": ["api", "worker"], + "description": "Primary application database" + } + ] +} +``` + +Managed artifact rules: + +- `entries[].value.kind` distinguishes logical plaintext from logical secrets +- `entries[].value.text` is the canonical string value before outer-envelope encryption +- `services` belongs in the managed artifact because portability must preserve service intent outside the originating machine +- `metadata` captures human and CLI/Desktop provenance, not machine-local backend details +- the artifact must not store local absolute paths, keychain service names, backend provider configuration, or runtime resolution history such as `resolvedFrom` + +Local compatibility rules: + +- `.hack/.env` contains only plaintext values +- the configured secret backend contains only secret values keyed by env var name +- neither compatibility target stores descriptions, service scope, timestamps, or actor metadata +- compatibility targets are derived outputs and may be regenerated from the managed artifact at any time + +CLI behavior when a managed artifact is active: + +1. Reads validate artifact keys against `.hack/hack.env.json` when present. +2. Reads report artifact intent, local materialization state, and drift between them. +3. `hack env set` mutates the canonical artifact first, then materializes the entry to `.hack/.env` or the configured secret backend based on `value.kind`. +4. If an entry changes kind, Hack removes stale local state from the old compatibility target before writing the new one. + +See [docs/plans/2026-03-13-portable-project-env-artifact-schema-design.md](plans/2026-03-13-portable-project-env-artifact-schema-design.md) for the full design rationale and implementation boundary. + ## CLI - `hack env list [--json] [--show-secrets]` @@ -59,6 +137,7 @@ Fields: - sets global backend strategy for multi-node/env secret storage Notes: + - `hack env set` also supports interactive prompting when `KEY` or `VALUE` is omitted. - Keychain mode uses service name `hack-` (project name from `.hack/hack.config.json`). - Encrypted-file mode uses `HACK_SECRETS_FILE_KEY` (if set) before keychain key lookup (`hack-secrets-backend/encrypted-file-key`). @@ -89,6 +168,7 @@ Global config supports backend selection for secret portability: ``` Fields: + - `backend`: `keychain` | `encrypted_file` | `cloud`. - `allowEnvAuthRefs`: allow `env:VAR_NAME` auth/secret references for node/controller workflows. - `encryptedFile.path`: target path for encrypted local store mode. @@ -106,23 +186,28 @@ When you run `hack up`, `hack restart`, or `hack run`, hack: 4. Invokes `docker compose` with an environment that includes resolved values (including secret backend values). Security posture: + - Secret values are never written into `.hack/` YAML files. - Plain env values live in `.hack/.env` (expected to be gitignored in most repos). ## Remote node secret behavior Current behavior (today): + - Node auth tokens and extension auth refs are stored through the configured secret backend. - `env:VAR_NAME` auth refs are allowed only when `controlPlane.secrets.allowEnvAuthRefs=true`. - Remote dispatch/workspace bootstrap does **not** automatically copy all host env values to remote nodes. Recommended setup: + - Keep controller secrets in `keychain` or `encrypted_file` backend (not shell env). - Configure each remote node to use the same secret backend strategy where possible. - Use project env contract (`.hack/hack.env.json`) to make required keys explicit before runs. Planned direction: + - Add an explicit host-to-node secret sync command with least-privilege scoping, encrypted payload delivery, and audit events per run. +- Keep the committed contract (`.hack/hack.env.json`) separate from the future portable managed-values artifact. See `docs/plans/2026-03-13-portable-project-env-artifact-schema-design.md`. ## Daemon/gateway API (UI integration) diff --git a/docs/plans/2026-03-13-portable-project-env-artifact-schema-design.md b/docs/plans/2026-03-13-portable-project-env-artifact-schema-design.md new file mode 100644 index 000000000..488f98861 --- /dev/null +++ b/docs/plans/2026-03-13-portable-project-env-artifact-schema-design.md @@ -0,0 +1,334 @@ +# Portable Project Env Artifact Schema Design + +## Context + +The current env model has two clear layers: + +- `.hack/hack.env.json` is a committed contract with key names, required flags, service scope, and descriptions. +- `.hack/.env` plus the configured secret backend hold local runtime values. + +That split works for local development, but it is not a portable managed-value model. It does not give Hack a stable artifact that can be exported, encrypted, published remotely, re-applied on another machine, or used as the canonical source for future CLI and desktop env workflows. + +This design introduces a separate portable managed env artifact instead of expanding `.hack/hack.env.json` in place. + +## Decision + +Keep `.hack/hack.env.json` as the declaration contract and introduce a second artifact for managed values: + +- Contract file: `.hack/hack.env.json` +- Portable artifact format: `PortableProjectEnvArtifactV1` +- Suggested import/export filename: `.hack/hack.env.managed.json` +- Default durable storage: encrypted bundle transport or remote storage, not a committed plaintext repo file + +This avoids schema churn in the existing contract parser and keeps the portable artifact focused on values, scope, and intent. + +## Approaches Considered + +### 1. Separate portable artifact + +Recommended. + +Pros: + +- preserves compatibility with current `hack init`, parser, docs, and tests +- keeps the committed contract safe to share and review +- gives portability its own stable schema and lifecycle +- makes `.hack/.env` and the secret backend clearly derived compatibility targets + +Cons: + +- introduces one more schema to document and maintain +- requires explicit reconciliation rules between contract and artifact + +### 2. Expand `.hack/hack.env.json` into the managed artifact + +Pros: + +- fewer files to explain +- one JSON document to parse + +Cons: + +- mixes safe shareable contract data with managed values +- creates migration pressure for existing repos and templates +- makes future remote encryption and local compatibility logic harder to reason about + +### 3. Keep only local compatibility files and derive portability ad hoc + +Pros: + +- no new schema immediately + +Cons: + +- `.hack/.env` cannot represent secrets safely or service intent +- the secret backend cannot represent descriptions or service scope +- export/apply/publish behavior would stay heuristic and brittle + +## Layer Model + +The env system should be treated as three distinct layers. + +### 1. Declaration contract + +`.hack/hack.env.json` + +Purpose: + +- defines which env keys exist +- captures default required flags, service scope, and descriptions +- remains safe to commit + +### 2. Portable managed artifact + +`PortableProjectEnvArtifactV1` + +Purpose: + +- canonical managed-value snapshot for publish, fetch, apply, import, and export +- contains logical plaintext and secret values plus the metadata needed to materialize them anywhere +- should be encrypted as a whole when stored outside the local machine + +### 3. Local compatibility materialization + +- `.hack/.env` +- configured secret backend (`keychain`, `encrypted_file`, `cloud`) + +Purpose: + +- runtime compatibility for Compose, local shells, and existing commands +- derived output only + +## Artifact Schema + +```json +{ + "$schema": "https://schemas.hack/hack.env.managed.schema.json", + "version": 1, + "environment": "default", + "metadata": { + "description": "Shared dev environment for Hack App", + "updatedAt": "2026-03-13T18:00:00Z", + "updatedBy": "cli", + "source": "hack-cli" + }, + "entries": [ + { + "key": "AWS_REGION", + "value": { + "kind": "plaintext", + "text": "us-east-1" + }, + "required": true, + "services": ["api"], + "description": "AWS region used by the API" + }, + { + "key": "DATABASE_URL", + "value": { + "kind": "secret", + "text": "postgres://..." + }, + "required": true, + "services": ["api", "worker"], + "description": "Primary application database" + } + ] +} +``` + +## Field Semantics + +### Top-level fields + +- `$schema`: schema URL for tooling and validation +- `version`: artifact schema version; starts at `1` +- `environment`: named managed environment, defaulting to `"default"` +- `metadata.description`: human summary for the artifact as a whole +- `metadata.updatedAt`: RFC 3339 timestamp for the last artifact mutation +- `metadata.updatedBy`: actor class such as `cli`, `desktop`, or `api` +- `metadata.source`: producing surface such as `hack-cli` or `hack-desktop` +- `entries`: stable, key-sorted list of managed env entries + +### Entry fields + +- `key`: uppercase snake-case env var name +- `value.kind`: + - `plaintext`: write to `.hack/.env` during compatibility materialization + - `secret`: write to the configured secret backend during compatibility materialization +- `value.text`: canonical string value +- `required`: whether the value is required when targeted services use it +- `services`: `null` or omitted means all services; otherwise a sorted list of Compose service names +- `description`: human intent note shown in CLI and UI + +## Why Secrets Are Still Plaintext Inside The Artifact + +The artifact is the normalized logical payload. Encryption belongs to the outer transport or storage envelope, not to individual entry fields. + +That means: + +- the artifact stays simple and deterministic +- plaintext and secret entries are distinguished by intent, not by nested encryption machinery +- the entire artifact can be encrypted, versioned, and audited as one unit for remote portability + +This design keeps the schema stable even if the remote encryption envelope changes later. + +## Portable Artifact Rules + +The portable artifact must include: + +- canonical key names +- canonical values +- whether each value is `plaintext` or `secret` +- required flag +- service scope +- human-readable description +- environment name +- artifact-level update metadata + +The portable artifact must not include: + +- local absolute paths +- machine hostnames +- keychain service names +- encrypted file paths +- cloud backend provider config +- `resolvedFrom` state such as `dotenv`, `process`, or `keychain` +- prompt history, interactive defaults, or shell fallback behavior +- per-machine compose override output + +## Local Compatibility Rules + +`.hack/.env` should contain only: + +- plaintext values +- no secret entries +- no descriptions +- no service scope metadata +- no timestamps or actor metadata + +The configured secret backend should contain only: + +- secret values keyed by env var name +- no plaintext entries +- no descriptions +- no service scope metadata + +Local compatibility files are not canonical. The CLI may regenerate them from the managed artifact at any time. + +## Reconciliation With `.hack/hack.env.json` + +The contract and artifact intentionally overlap on a few fields. The rules should be: + +- contract defines the expected shape of the project env surface +- artifact carries the managed snapshot used for portability +- when both exist, key names must be validated against the contract +- if `required`, `services`, or `description` disagree, the artifact wins for apply/export behavior and the CLI should surface drift +- unknown artifact keys should warn by default and require explicit confirmation before apply + +This keeps the artifact self-contained while still treating the committed contract as the project declaration surface. + +## CLI Read Behavior + +When portable env management is enabled for a project, CLI reads should use this order: + +1. load the portable artifact or fetched remote bundle payload +2. validate entry keys against `.hack/hack.env.json` when present +3. materialize compatibility state in memory: + - plaintext entries to `.env`-style map + - secret entries to secret-store map +4. compare local compatibility files against the canonical artifact +5. report drift, missing values, and unknown keys explicitly + +If no managed artifact is configured, current local-only behavior remains unchanged. + +## CLI Write Behavior + +`hack env set` and `hack env set --secret` should behave as follows when a managed artifact is active: + +1. mutate the canonical artifact entry +2. update top-level metadata timestamps/source +3. regenerate the corresponding local compatibility target: + - plaintext to `.hack/.env` + - secret to the configured secret backend +4. keep the other compatibility target untouched unless the entry kind changed + +If an entry changes kind: + +- `plaintext -> secret`: remove it from `.hack/.env`, then write it to the secret backend +- `secret -> plaintext`: remove it from the secret backend, then write it to `.hack/.env` + +`hack env list` should show: + +- contract intent +- artifact intent and value kind +- local compatibility resolution status +- drift when local files do not match the canonical artifact + +## Service Scope Rules + +Service scope belongs in the artifact because apply and export need stable intent even outside the originating machine. + +Rules: + +- omit or set `services` to `null` for all services +- store service names sorted and de-duplicated +- never infer service scope from `.hack/.env` +- when targeting a service subset, only required entries whose scope intersects that subset should block execution + +## Migration And Compatibility + +No existing repo should be forced to adopt the portable artifact immediately. + +Initial compatibility rules: + +- existing projects with only `.hack/hack.env.json` plus local values continue to work unchanged +- the portable artifact is opt-in +- import/export and remote publish/apply should use the new artifact +- local-only projects can be upgraded by normalizing current values into the artifact on first publish or explicit `env managed init` + +## Validation And Error Handling + +The schema should fail closed on structural errors and fail loud on semantic drift. + +Rules: + +- reject unknown schema versions +- reject entries without a valid uppercase env key +- reject duplicate keys inside one artifact +- reject invalid `value.kind` values +- reject non-array `services` values other than `null`/omitted +- sort and de-duplicate service names at serialization boundaries +- warn, rather than silently discard, when artifact keys are not declared in `.hack/hack.env.json` +- require explicit confirmation before applying artifact entries that are unknown to the contract + +This keeps corruption visible and makes CLI behavior predictable for publish, fetch, and apply flows. + +## Recommended Implementation Boundary + +Implement the artifact as a typed model and serializer first, before wiring remote storage. + +Suggested first implementation units: + +- parser and serializer for `PortableProjectEnvArtifactV1` +- normalization from contract + `.hack/.env` + secret store into artifact shape +- materialization from artifact back into local compatibility files +- drift reporting between artifact and local compatibility state + +## Testing Expectations + +The first implementation should be verifiable with deterministic unit tests around the artifact model before any remote transport work lands. + +Minimum test coverage: + +- parse valid plaintext and secret entries +- reject invalid versions and duplicate keys +- preserve `required`, `services`, and `description` through parse/serialize round trips +- keep entry ordering stable by key and service ordering stable within each entry +- normalize current local contract + value state into the artifact without leaking secret entries into `.hack/.env` +- materialize artifact entries back into local compatibility storage, including kind changes and stale-key cleanup + +## Notes + +- The referenced Linear spec URL was not directly accessible from this environment, so this design is inferred from the issue text, current repo docs, and the existing remote env portability plan. +- This design intentionally keeps the remote encryption envelope out of the inner artifact schema to minimize version churn. diff --git a/docs/plans/2026-03-13-portable-project-env-artifact-schema-plan.md b/docs/plans/2026-03-13-portable-project-env-artifact-schema-plan.md new file mode 100644 index 000000000..2a2c27092 --- /dev/null +++ b/docs/plans/2026-03-13-portable-project-env-artifact-schema-plan.md @@ -0,0 +1,241 @@ +# Portable Project Env Artifact Schema Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Add a stable portable managed env artifact model that can normalize plaintext and secret values, materialize local compatibility files, and support future remote publish/apply flows without changing the existing contract file format. + +**Architecture:** Keep `.hack/hack.env.json` as the committed declaration contract. Introduce a typed managed artifact model for canonical values plus metadata, then add read/write helpers that translate between the artifact and existing local compatibility targets (`.hack/.env` and the configured secret backend). + +**Tech Stack:** Bun, TypeScript, existing CLI env commands, Bun test, docs in `docs/env.md` and `docs/plans/` + +--- + +### Task 1: Add managed artifact types and parsing + +**Files:** + +- Create: `src/lib/managed-env-artifact.ts` +- Test: `tests/managed-env-artifact.test.ts` +- Modify: `src/constants.ts` + +**Step 1: Write the failing test** + +Add parser/serializer tests that cover: + +- valid artifact parse for plaintext entries +- valid artifact parse for secret entries +- service scope parsing with `null` and specific service lists +- stable serialization ordering +- invalid version rejection + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/managed-env-artifact.test.ts` Expected: FAIL because the module and types do not exist yet. + +**Step 3: Write minimal implementation** + +Create: + +- artifact version constant +- `PortableProjectEnvArtifact` +- `PortableProjectEnvEntry` +- `readManagedEnvArtifact` +- `parseManagedEnvArtifact` +- `serializeManagedEnvArtifact` + +Keep the schema narrow: + +- top-level metadata +- entries with `key`, `value.kind`, `value.text`, `required`, `services`, `description` + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/managed-env-artifact.test.ts` Expected: PASS. + +**Step 5: Commit** + +```bash +git add src/lib/managed-env-artifact.ts src/constants.ts tests/managed-env-artifact.test.ts +git commit -m "feat: add managed env artifact schema" +``` + +### Task 2: Normalize current local env state into the artifact + +**Files:** + +- Modify: `src/lib/hack-env.ts` +- Test: `tests/managed-env-artifact.test.ts` +- Test: `tests/env-backend-command.test.ts` + +**Step 1: Write the failing test** + +Add tests that normalize: + +- contract + `.hack/.env` into plaintext entries +- contract + secret store into secret entries +- mixed plaintext and secret entries +- contract descriptions and service scope copied into the artifact + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/managed-env-artifact.test.ts tests/env-backend-command.test.ts` Expected: FAIL because normalization helpers do not exist. + +**Step 3: Write minimal implementation** + +Add helpers that: + +- resolve current local values using existing env logic +- build a canonical artifact entry list +- preserve contract metadata in the artifact +- keep entries sorted by key and services sorted alphabetically + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/managed-env-artifact.test.ts tests/env-backend-command.test.ts` Expected: PASS. + +**Step 5: Commit** + +```bash +git add src/lib/hack-env.ts tests/managed-env-artifact.test.ts tests/env-backend-command.test.ts +git commit -m "feat: normalize local env state into managed artifacts" +``` + +### Task 3: Materialize artifact values back into local compatibility files + +**Files:** + +- Modify: `src/lib/hack-env.ts` +- Modify: `src/lib/secret-store.ts` +- Test: `tests/env-backend-command.test.ts` + +**Step 1: Write the failing test** + +Add tests that: + +- write plaintext artifact entries into `.hack/.env` +- write secret artifact entries into the configured secret backend +- delete old local state when an entry changes kind +- leave unrelated keys untouched + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/env-backend-command.test.ts` Expected: FAIL because artifact materialization does not exist. + +**Step 3: Write minimal implementation** + +Add a helper that takes a managed artifact and: + +- writes plaintext entries to `.hack/.env` +- writes secret entries to the configured secret store +- removes stale keys from the wrong local target when kind changes + +Do not add remote storage in this task. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/env-backend-command.test.ts` Expected: PASS. + +**Step 5: Commit** + +```bash +git add src/lib/hack-env.ts src/lib/secret-store.ts tests/env-backend-command.test.ts +git commit -m "feat: materialize managed env artifacts locally" +``` + +### Task 4: Add CLI visibility for managed artifact state + +**Files:** + +- Modify: `src/commands/env.ts` +- Test: `tests/env-backend-command.test.ts` + +**Step 1: Write the failing test** + +Add tests for: + +- artifact-aware `hack env list --json` +- drift reporting between artifact and local compatibility state +- explicit reporting of `plaintext` vs `secret` kind + +**Step 2: Run test to verify it fails** + +Run: `bun test tests/env-backend-command.test.ts` Expected: FAIL because the CLI output does not expose managed artifact state yet. + +**Step 3: Write minimal implementation** + +Extend `hack env list` output to show: + +- managed artifact presence +- value kind +- contract metadata +- local drift state + +Keep existing output compatible where possible. + +**Step 4: Run test to verify it passes** + +Run: `bun test tests/env-backend-command.test.ts` Expected: PASS. + +**Step 5: Commit** + +```bash +git add src/commands/env.ts tests/env-backend-command.test.ts +git commit -m "feat: surface managed env artifact state in env list" +``` + +### Task 5: Document artifact rules and upgrade path + +**Files:** + +- Modify: `docs/env.md` +- Modify: `docs/guides/init-project.md` +- Modify: `src/templates.ts` + +**Step 1: Write the failing test** + +No code test is required for this task. Use doc review plus template output review. + +**Step 2: Run verification to confirm the current docs are incomplete** + +Run: `grep -n \"managed artifact\\|portable artifact\" docs/env.md docs/guides/init-project.md src/templates.ts` Expected: incomplete or missing references to the new artifact model. + +**Step 3: Write minimal implementation** + +Document: + +- contract vs artifact vs local compatibility split +- opt-in upgrade path +- any template notes needed for future initialization behavior + +Do not start scaffolding the artifact by default unless the implementation requires it. + +**Step 4: Run verification to confirm docs are updated** + +Run: `grep -n \"managed artifact\\|portable artifact\" docs/env.md docs/guides/init-project.md src/templates.ts` Expected: matches in the updated docs. + +**Step 5: Commit** + +```bash +git add docs/env.md docs/guides/init-project.md src/templates.ts +git commit -m "docs: describe managed env artifact model" +``` + +## Recommended Execution Order + +1. Add the standalone artifact module first. +2. Normalize current local state into the artifact. +3. Add local materialization helpers. +4. Surface the new state in CLI output. +5. Update docs only after the code contract is settled. + +## Risks To Watch + +- accidental breaking changes to existing `.hack/hack.env.json` parsing +- leaking secret values into `.hack/.env` during kind changes +- making `hack env list --json` incompatible for existing consumers +- treating local compatibility files as canonical instead of derived + +## Links + +- `docs/plans/2026-03-13-portable-project-env-artifact-schema-design.md` +- `docs/plans/2026-03-06-remote-encrypted-project-env-portability-plan.md` diff --git a/scripts/build-release.ts b/scripts/build-release.ts index 5f18e9e6d..ad7384791 100644 --- a/scripts/build-release.ts +++ b/scripts/build-release.ts @@ -7,6 +7,7 @@ import { renderProjectBranchesSchemaJson, renderProjectConfigSchemaJson, renderProjectEnvSchemaJson, + renderProjectManagedEnvSchemaJson, } from "../src/templates.ts"; interface BuildArgs { @@ -109,6 +110,10 @@ async function main({ args }: { readonly args: BuildArgs }): Promise { resolve(schemasDir, "hack.env.schema.json"), renderProjectEnvSchemaJson() ); + await Bun.write( + resolve(schemasDir, "hack.env.managed.schema.json"), + renderProjectManagedEnvSchemaJson() + ); await Bun.write( resolve(schemasDir, "hack.branches.schema.json"), renderProjectBranchesSchemaJson() diff --git a/src/commands/global.ts b/src/commands/global.ts index 08ed8fc9e..57bcf7e20 100644 --- a/src/commands/global.ts +++ b/src/commands/global.ts @@ -4,9 +4,6 @@ import { confirm, isCancel, note, spinner } from "@clack/prompts"; import type { CliContext, CommandArgs } from "../cli/command.ts"; import { defineCommand, defineOption, withHandler } from "../cli/command.ts"; -/** Regex to split strings on whitespace. */ -const WHITESPACE_PATTERN = /\s+/; - import { optFollow, optJson, @@ -40,6 +37,7 @@ import { GLOBAL_LOGGING_COMPOSE_FILENAME, GLOBAL_LOGGING_DIR_NAME, GLOBAL_LOKI_CONFIG_FILENAME, + GLOBAL_MANAGED_ENV_SCHEMA_FILENAME, GLOBAL_SCHEMAS_DIR_NAME, } from "../constants.ts"; import { resolveGatewayConfig } from "../control-plane/extensions/gateway/config.ts"; @@ -82,6 +80,7 @@ import { renderProjectBranchesSchemaJson, renderProjectConfigSchemaJson, renderProjectEnvSchemaJson, + renderProjectManagedEnvSchemaJson, } from "../templates.ts"; import { display } from "../ui/display.ts"; import { dockerComposeLogsPretty } from "../ui/docker-logs.ts"; @@ -89,6 +88,9 @@ import { ensureBundledGumInstalled } from "../ui/gum.ts"; import { logger } from "../ui/logger.ts"; import { resolvePreferredHostDnsTarget } from "./doctor-utils.ts"; +/** Regex to split strings on whitespace. */ +const WHITESPACE_PATTERN = /\s+/; + const globalLogsOptions = [optFollow, optNoFollow, optTail, optPretty] as const; const globalLogsPositionals = [{ name: "service", required: false }] as const; @@ -283,6 +285,7 @@ function getGlobalPaths() { grafanaDashboard: resolve(loggingDir, GLOBAL_GRAFANA_DASHBOARD_FILENAME), configSchema: resolve(schemasDir, GLOBAL_CONFIG_SCHEMA_FILENAME), envSchema: resolve(schemasDir, GLOBAL_ENV_SCHEMA_FILENAME), + managedEnvSchema: resolve(schemasDir, GLOBAL_MANAGED_ENV_SCHEMA_FILENAME), branchesSchema: resolve(schemasDir, GLOBAL_BRANCHES_SCHEMA_FILENAME), }; } @@ -607,6 +610,10 @@ async function globalInstall(): Promise { paths.envSchema, renderProjectEnvSchemaJson() ); + await writeWithPromptIfDifferent( + paths.managedEnvSchema, + renderProjectManagedEnvSchemaJson() + ); await writeWithPromptIfDifferent( paths.branchesSchema, renderProjectBranchesSchemaJson() diff --git a/src/constants.ts b/src/constants.ts index 1bffc7df3..cfe53f435 100644 --- a/src/constants.ts +++ b/src/constants.ts @@ -88,5 +88,7 @@ export const PROJECT_BRANCHES_FILENAME = "hack.branches.json" as const; export const GLOBAL_CONFIG_SCHEMA_FILENAME = "hack.config.schema.json" as const; export const GLOBAL_ENV_SCHEMA_FILENAME = "hack.env.schema.json" as const; +export const GLOBAL_MANAGED_ENV_SCHEMA_FILENAME = + "hack.env.managed.schema.json" as const; export const GLOBAL_BRANCHES_SCHEMA_FILENAME = "hack.branches.schema.json" as const; diff --git a/src/templates.ts b/src/templates.ts index fcca682dc..b46b8365d 100644 --- a/src/templates.ts +++ b/src/templates.ts @@ -669,6 +669,70 @@ export function renderProjectEnvSchemaJson(): string { return `${JSON.stringify(schema, null, 2)}\n`; } +export function renderProjectManagedEnvSchemaJson(): string { + const schema = { + $schema: "https://json-schema.org/draft/2020-12/schema", + title: "hack.env.managed.json", + type: "object", + additionalProperties: false, + required: ["version", "environment", "metadata", "entries"], + properties: { + $schema: { type: "string" }, + version: { type: "integer", const: 1 }, + environment: { type: "string", minLength: 1 }, + metadata: { + type: "object", + additionalProperties: false, + required: ["updatedAt", "updatedBy", "source"], + properties: { + description: { type: "string" }, + updatedAt: { type: "string", format: "date-time" }, + updatedBy: { type: "string", minLength: 1 }, + source: { type: "string", minLength: 1 }, + }, + }, + entries: { + type: "array", + items: { + type: "object", + additionalProperties: false, + required: ["key", "value", "required"], + properties: { + key: { + type: "string", + pattern: "^[A-Z_][A-Z0-9_]*$", + minLength: 1, + }, + value: { + type: "object", + additionalProperties: false, + required: ["kind", "text"], + properties: { + kind: { type: "string", enum: ["plaintext", "secret"] }, + text: { type: "string" }, + }, + }, + required: { type: "boolean" }, + services: { + anyOf: [ + { type: "null" }, + { + type: "array", + items: { type: "string", minLength: 1 }, + uniqueItems: true, + }, + ], + }, + description: { type: "string" }, + }, + }, + }, + }, + } as const; + + return `${JSON.stringify(schema, null, 2)}\n`; +} + export function renderProjectBranchesSchemaJson(): string { const schema = { $schema: "https://json-schema.org/draft/2020-12/schema", diff --git a/tests/global-command.test.ts b/tests/global-command.test.ts index 47e250905..2d727fb2a 100644 --- a/tests/global-command.test.ts +++ b/tests/global-command.test.ts @@ -13,6 +13,8 @@ import { GLOBAL_HACK_DIR_NAME, GLOBAL_LOGGING_COMPOSE_FILENAME, GLOBAL_LOGGING_DIR_NAME, + GLOBAL_MANAGED_ENV_SCHEMA_FILENAME, + GLOBAL_SCHEMAS_DIR_NAME, } from "../src/constants.ts"; let tempDir: string | null = null; @@ -273,9 +275,17 @@ test("global install writes compose files and starts stacks", async () => { ); const hasCaddy = await Bun.file(caddyCompose).exists(); const hasLogging = await Bun.file(loggingCompose).exists(); + const managedEnvSchema = join( + tempDir!, + GLOBAL_HACK_DIR_NAME, + GLOBAL_SCHEMAS_DIR_NAME, + GLOBAL_MANAGED_ENV_SCHEMA_FILENAME + ); + const hasManagedEnvSchema = await Bun.file(managedEnvSchema).exists(); expect(hasCaddy).toBe(true); expect(hasLogging).toBe(true); + expect(hasManagedEnvSchema).toBe(true); expect( runCalls.some((call) => call.includes(caddyCompose) && call.includes("up")) ).toBe(true); diff --git a/tests/project-config-schema.test.ts b/tests/project-config-schema.test.ts index 95faf3e11..69cebb781 100644 --- a/tests/project-config-schema.test.ts +++ b/tests/project-config-schema.test.ts @@ -1,5 +1,8 @@ import { expect, test } from "bun:test"; -import { renderProjectConfigSchemaJson } from "../src/templates.ts"; +import { + renderProjectConfigSchemaJson, + renderProjectManagedEnvSchemaJson, +} from "../src/templates.ts"; test("project config schema includes startup validation", () => { const schema = JSON.parse(renderProjectConfigSchemaJson()) as Record< @@ -49,3 +52,51 @@ test("project config schema includes strict lifecycle hooks and processes", () = expect(processItems.additionalProperties).toBe(false); expect(processItems.required).toEqual(["name", "command"]); }); + +test("managed env schema models entry kind, metadata, and service scope", () => { + const schema = JSON.parse(renderProjectManagedEnvSchemaJson()) as Record< + string, + unknown + >; + const properties = schema.properties as Record; + + expect(schema.additionalProperties).toBe(false); + expect(schema.required).toEqual([ + "version", + "environment", + "metadata", + "entries", + ]); + + const metadata = properties.metadata as Record; + expect(metadata.additionalProperties).toBe(false); + const metadataProperties = metadata.properties as Record; + expect(metadataProperties.updatedAt).toEqual({ + type: "string", + format: "date-time", + }); + + const entries = properties.entries as Record; + const entry = entries.items as Record; + expect(entry.additionalProperties).toBe(false); + expect(entry.required).toEqual(["key", "value", "required"]); + + const entryProperties = entry.properties as Record; + const value = entryProperties.value as Record; + expect(value.additionalProperties).toBe(false); + const valueProperties = value.properties as Record; + expect(valueProperties.kind).toEqual({ + type: "string", + enum: ["plaintext", "secret"], + }); + + const services = entryProperties.services as Record; + expect(services.anyOf).toEqual([ + { type: "null" }, + { + type: "array", + items: { type: "string", minLength: 1 }, + uniqueItems: true, + }, + ]); +}); From daa5ed3dedbcfc0e6a26d35ebbfbc8152e2020c9 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sat, 14 Mar 2026 04:07:52 +0000 Subject: [PATCH 014/134] docs: clarify core offer and docs ia --- README.md | 960 +++----------------------- apps/macos/README.md | 4 +- docs/README.md | 75 +- docs/cli.md | 125 +++- docs/extensions.md | 8 +- docs/gateway-api.md | 3 +- docs/gateway.md | 3 +- docs/guides/remote-cloudflare.md | 3 +- docs/guides/remote-node-container.md | 4 +- docs/guides/remote-node-laptop-e2e.md | 4 +- docs/guides/remote-node-quickstart.md | 4 +- docs/guides/remote-node-railway.md | 4 +- docs/guides/remote-ssh.md | 3 +- docs/guides/remote-supervisor.md | 4 +- docs/guides/remote-tailscale.md | 4 +- docs/integrations.md | 126 ++++ docs/supervisor.md | 4 +- src/cli/command.ts | 3 + src/cli/help.ts | 25 +- src/cli/spec.ts | 8 +- src/commands/auth.ts | 14 +- src/commands/dispatch.ts | 10 +- src/commands/env.ts | 4 +- src/commands/gateway.ts | 10 +- src/commands/linear.ts | 4 +- src/commands/node.ts | 4 +- src/commands/remote.ts | 12 +- src/commands/session.ts | 4 +- src/commands/ssh.ts | 4 +- src/commands/tickets.ts | 4 +- tests/cli-help.test.ts | 52 ++ 31 files changed, 532 insertions(+), 964 deletions(-) create mode 100644 docs/integrations.md create mode 100644 tests/cli-help.test.ts diff --git a/README.md b/README.md index 31d64cc93..3c134f4f4 100644 --- a/README.md +++ b/README.md @@ -1,942 +1,204 @@ - +# Hack - -
-
-
- █████   █████   █████████     █████████  █████   ████
-░░███   ░░███   ███░░░░░███   ███░░░░░███░░███   ███░ 
- ░███    ░███  ░███    ░███  ███     ░░░  ░███  ███   
- ░███████████  ░███████████ ░███          ░███████    
- ░███░░░░░███  ░███░░░░░███ ░███          ░███░░███   
- ░███    ░███  ░███    ░███ ░░███     ███ ░███ ░░███  
- █████   █████ █████   █████ ░░█████████  █████ ░░████
-░░░░░   ░░░░░ ░░░░░   ░░░░░   ░░░░░░░░░  ░░░░░   ░░░░ 
-

-
-
-

- Opinionated local-dev orchestration for running multiple projects at the same time without port conflicts. -
-

+Hack is local development without the port-collision tax. ---- -**Network isolation per repo / branch**: every instance runs on its own Docker network (so Postgres/Redis/etc can stay on default ports *inside* the project). +It gives each repo or branch its own network, a stable HTTPS URL, and a single place to manage +the workflows that usually end up scattered across shell scripts, terminal tabs, and sidecar tools. -**Stable HTTPS hostnames**: `https://.hack` (and subdomains like `https://api..hack`) routed by a global Caddy proxy. +## Three core promises -**Good logs UX**: instant `docker compose logs` tailing, plus Loki/Grafana for querying + history. +1. Run multiple repos or branches at the same time without port conflicts. +2. Open every project on a stable HTTPS URL like `https://myapp.hack`. +3. Keep project workflows such as env, sessions, tickets, and integrations close to the repo. -**Opt-in per repo**: no invasive changes to your codebase; config lives in `.hack/`. - ---- - -### Why this exists - -Most of my projects run the same stack. That’s fine until you want to: - -- run two projects at the same time -- run two branches of the same repo -- run multiple worktrees in parallel - -At that point everything fights over `localhost` and default ports. - -The daily choice: - -- **Option A:** stop A → start B → ship fix → stop B → restart A - you just spent 5–10 minutes paying the orchestration tax and nuking your focus. -- **Option B:** don’t run it. code blind. push. let CI tell you what you broke. - -Neither scales. Both slow you down in dumb, repeatable ways. - - -## Installation +## Five-minute start ### Prerequisites -- **macOS**: supported. -- **Docker + Compose**: [Orbstack](https://docs.orbstack.dev/quick-start) | [Docker Desktop](https://www.docker.com/get-started/) + +- macOS +- Docker + Compose via [OrbStack](https://docs.orbstack.dev/quick-start) or + [Docker Desktop](https://www.docker.com/get-started/) ### Install -**Option A: Shell script (CLI only)** +CLI: ```bash curl -fsSL https://github.com/hack-dance/hack/releases/latest/download/hack-install.sh | bash ``` -**Option B: macOS DMG (CLI + Desktop App)** +macOS app: -Download the latest DMG from [GitHub Releases](https://github.com/hack-dance/hack/releases/latest). +- Download the latest DMG from + [GitHub Releases](https://github.com/hack-dance/hack/releases/latest) +- `Hack Desktop for macOS` is currently **Beta** -The DMG includes: -- **Hack Desktop** - macOS menu bar app for managing projects -- **Install Hack CLI** - double-click to install the CLI to `~/.hack/bin` +### Start a project -### Setup ```bash hack global install -``` - -### Quick start -Manual (CLI): -```bash cd /path/to/your-repo hack init hack up --detach hack open ``` -Agent-assisted (Cursor/Claude/Codex with shell access): - -```bash -hack setup cursor # or hack setup claude / hack setup codex -hack setup agents # optional: adds AGENTS.md + CLAUDE.md snippets -hack setup sync --all-scopes # optional: refresh project + user integrations -hack agent init --client cursor # or --client claude / --client codex -hack agent patterns # optional: dependency/ops checklist -``` - -If you omit `--client`, `hack agent init` will prompt you to choose (TTY only). -If your agent does not auto-open, paste the output into the chat. Example: - -``` -I ran `hack agent init` and pasted the output below. Please follow it to set up hack for this repo, -then run: -- `hack init` (use --auto if dev scripts are detected) -- `hack up --detach` -- `hack open --json` -If anything fails, use `hack logs --pretty` and summarize next steps. - - -``` - -If the agent cannot run shell commands, use MCP instead: `hack setup mcp` and `hack mcp serve`. - -### Remote node runtime image (multi-node) - -Use the prebuilt node-runtime container to bootstrap remote nodes quickly (Linux/macOS hosts with Docker): - -```bash -bun run build:node-runtime-image --tag hack-node-runtime:dev -``` - -Full operator guide: [`docs/guides/remote-node-container.md`](docs/guides/remote-node-container.md) - -### Tickets (optional, git-backed issue tracking) +What those commands do: -`hack` includes a lightweight, git-backed ticket system for tracking work without leaving your repo. Tickets are stored in a hidden git ref (`refs/hack/tickets`) so they sync with your code but don't clutter your branch list. +- `hack global install` boots the machine-level DNS, TLS, and logging services +- `hack init` creates the `.hack/` project config +- `hack up --detach` starts your stack on an isolated network +- `hack open` opens the project URL in your browser -**Quick setup:** +Useful next commands: ```bash -# Enable tickets for this repo -hack x tickets setup - -# Create a ticket -hack x tickets create --title "Add dark mode support" - -# List tickets -hack x tickets list - -# Interactive TUI for managing tickets -hack x tickets tui - -# Update status -hack x tickets status T-00001 in_progress - -# Sync to remote -hack x tickets sync -``` - -**Why use it:** -- No external service required - tickets live in git -- Agent-friendly - Claude/Cursor/Codex can create and manage tickets -- Syncs with `git push/pull` via hidden refs -- Works offline - -Tickets are ideal for solo devs, small teams, or agent-driven workflows where you want lightweight task tracking without context-switching to Jira/Linear/GitHub Issues. - -Full docs: [`docs/guides/tickets.md`](docs/guides/tickets.md) - -### macOS app (experimental) -Source lives in `apps/macos`. See `apps/macos/README.md` for XcodeGen and build/run steps. -Quick commands (repo root): - -```bash -bun run macos:project-gen -bun run macos:open -bun run macos:dev -bun run macos:build -bun run macos:test -``` - -### Configuration (.hack/hack.config.json) -- `name`: project slug (also used for Docker Compose project name) -- `dev_host`: base hostname (`.hack`) - -*Optional `[logs]` settings* -- `follow_backend`: `compose|loki` (default: `compose`) -- `snapshot_backend`: `compose|loki` (default: `loki`) -- `clear_on_down`: when running `hack down`, request Loki delete for this project (best-effort) -- `retention_period`: e.g. `"24h"`; on `hack down`, prune older logs (best-effort) - -*Optional `[internal]` settings (container DNS/TLS)* -- `dns`: use CoreDNS to resolve `*.hack` inside containers (default: `true`) -- `tls`: mount Caddy Local CA + set common SSL env vars (default: `true`) -- `extra_hosts`: static Compose `extra_hosts` entries (hostname → IP/target), merged into the internal override - -If you need **dynamic** `extra_hosts` (e.g. Pulumi outputs / local tunnels that change), use: - -```bash -hack internal extra-hosts set -hack internal extra-hosts unset -hack internal extra-hosts list -``` - -This writes `.hack/.internal/extra-hosts.json` and is merged into `extra_hosts` when you run `hack up` / `hack restart`. - -*Optional `[oauth]` settings (OAuth-safe alias host)* -- `enabled`: when true, `hack init` generates Caddy labels so routed services answer on both: -- primary: `https://` -- OAuth alias: `https://.` (e.g. `https://sickemail.hack.gy`) -- `tld`: optional (default: `"gy"`). Only `*.hack.gy` is bootstrapped automatically by `hack global install`; other TLDs require manual DNS setup. - -The file includes a JSON Schema reference for editor validation: - -```json -{ - "$schema": "https://schemas.hack/hack.config.schema.json" -} -``` - -Schemas are served locally by the global Caddy proxy at `https://schemas.hack`. - -Quick edits: - -```bash -hack config get dev_host -hack config set dev_host "myapp.hack" -hack config set logs.snapshot_backend "compose" -``` - -## Commands (high level) - -Run `hack help` (or `hack help `) for full usage. - -Common: -- `hack global install|up|down` -- `hack init|up|down|logs|open|tui` -- `hack status` -- `hack remote setup` -- `hack gateway enable` - -Full command table + flags: `docs/cli.md`. - -Run `hack help ` for detailed help. - -Project commands that call Docker Compose accept `--profile` (up/down/restart/ps/logs/run). - -## JSON output - -Use `--json` for machine-readable output: - -- `hack projects --json` -- `hack ps --json` -- `hack logs --json` (NDJSON stream; use `--no-follow` for snapshots) -- `hack open --json` (returns `{ "url": "..." }`) - -When the daemon is running, `hack projects --json` and `hack ps --json` use it for faster results. - -`hack logs --json` emits event envelopes (`start`, `log`, `end`) so MCP/TUI consumers can stream safely. - -## Daemon (optional) - -`hackd` is a local daemon that caches Docker state for fast status/ps queries. - -```bash -hack daemon start -hack daemon status -hack daemon metrics -hack daemon stop -hack daemon logs +hack logs --pretty +hack run +hack status +hack down ``` -### launchd integration (macOS) +## What is core today -On macOS, you can install `hackd` as a launchd service for automatic management: +These workflows are the main product: -```bash -# Install (auto-start on login by default) -hack daemon install +- Project setup and runtime: `hack init`, `hack up`, `hack down`, `hack restart`, `hack ps` +- Stable local access: `hack open`, `hack logs`, `hack run`, `hack status` +- Parallel project work: branch instances, isolated networks, and per-project hostnames +- Repo-local workflow support: `hack env`, `hack session`, `hack tickets` -# Install without auto-start (manual start/stop via launchd) -hack daemon install --no-run-at-load +Supporting capabilities: -# Uninstall the launchd service -hack daemon uninstall -``` +- GitHub and Linear integrations +- Agent setup for Cursor, Claude Code, and Codex +- CLI and SDK reference surfaces for extensions and automation -Options: -- `--run-at-load` / `--no-run-at-load`: Start automatically on login -- `--gui-only` / `--no-gui-only`: Only run in GUI sessions (default: GUI only) +Beta surfaces: -The service uses label `dance.hack.hackd` and writes to `~/Library/LaunchAgents/`. +- Remote control plane and gateway flows +- Multi-node execution and dispatch +- Hack Desktop for macOS -Use it when: -- You run `hack projects --json` / `hack ps --json` frequently (scripts, agent workflows, TUI). -- You want faster status snapshots without shelling out to Docker each time. +## Integrations overview -Skip it when: -- You prefer zero background processes. -- You rarely use JSON status/ps outputs. +Hack has a few supporting surfaces that matter after your project is already running. -If it is not running (or version-mismatched), the CLI falls back to direct Docker calls. +### GitHub -## Control plane + extensions (optional) +Use GitHub integration when you want Hack to authenticate repo workflows or create and update pull +requests for you. -`hack` includes a small control-plane kernel so features like jobs, tickets, and remote access can -ship as extensions without bloating the core CLI. +- Main surface: `hack x github ...` +- Common use: connect a profile, then use PR or repo automation from Hack workflows +- Reference: [docs/extensions.md](docs/extensions.md) -- Extension commands run via `hack x `. -- Global control-plane config lives at `~/.hack/hack.config.json` (`hack config set --global ...`). -- Per-project overrides live in `.hack/hack.config.json` and win over global values. -- `controlPlane.gateway.enabled` is project-scoped and implicitly enables the gateway extension. +### Linear -See `SPECS/control-plane/consolidated.md` and `docs/extensions.md` for the extension SDK surface. +Use Linear when you want selected Linear issues or projects to sync into your repo-local workflow +instead of treating Linear as a separate universe. -### Gateway (remote access) +- Main surface: `hack linear ...` +- Common use: connect a profile, bind a project, sync issues or project state +- Reference: [docs/integrations.md](docs/integrations.md) -The gateway exposes `hackd` over HTTP/WS with token auth. It binds to `127.0.0.1` by default and -should be exposed via a Zero Trust/VPN or SSH tunnel when needed. `hack remote setup` is the -one-command flow that enables the gateway, creates a token, and can configure + start exposure -(Cloudflare/Tailscale/SSH) via prompts. It prints a QR by default (use `--no-qr` to skip). +### Tickets -```bash -hack remote setup -# or: -hack gateway setup -# or manually: -hack gateway enable -hack daemon stop && hack daemon start -hack x gateway token-create -``` +Use tickets when you want lightweight work tracking in git without leaving the repo. -Gateway tokens default to read-only. For non-GET requests, set -`controlPlane.gateway.allowWrites = true` globally and create a write-scoped token: +- Main surface: `hack tickets ...` +- Common use: create follow-up work, track status, sync hidden ticket refs +- Reference: [docs/guides/tickets.md](docs/guides/tickets.md) -```bash -hack config set --global 'controlPlane.gateway.allowWrites' true -``` +### Sessions -```bash -hack x gateway token-create --scope write -``` +Use sessions when you want a durable terminal workspace for yourself or an agent. -Current gateway API (HTTP/WS): -- status/metrics/projects/ps (`/v1/*`) -- supervisor jobs: list/create/show/cancel + log/event stream (`/control-plane/*`) -- supervisor shells: create/show + PTY stream (write token + allowWrites) -- CLI shell client: `hack x supervisor shell` (gateway + write token required) +- Main surface: `hack session ...` +- Common use: keep a project shell alive, reattach later, or run long-lived agent work +- Reference: [docs/sessions.md](docs/sessions.md) -Interactive shells are available over the gateway WebSocket; the CLI can attach with -`hack x supervisor shell` (write token + allowWrites required). Use SSH/Zero Trust for a full -terminal UI, or run commands via supervisor jobs. +### Env -See `docs/gateway-api.md` for full API usage, structured workflow patterns, and a runnable demo. +Use env management when you want the repo to declare what configuration it needs without committing +the secret values themselves. -Remote access options (recommended order): -1) SSH tunnel to the gateway port for quick, ad-hoc access. -2) Zero Trust/VPN (Tailscale, Cloudflare, etc.) for persistent access. -3) Optional Caddy route (`https://gateway.hack`) for local convenience. +- Main surface: `hack env ...` +- Common use: define required vars in `.hack/hack.env.json`, keep values in `.hack/.env` or a + secret backend +- Reference: [docs/env.md](docs/env.md) -Note: Cloudflare Tunnel is ideal for the gateway HTTP/WS surface. It is not a direct SSH -replacement on iOS; use Tailscale/VPN for SSH access from mobile clients. -If you already use Cloudflare WARP, configure a private network route to your laptop and -use that IP/hostname for SSH in your mobile client. +## Docs map -Remote helper: -- `hack remote` shows status and offers to run setup when needed. -- `hack remote status` prints gateway + exposure status. -- `hack remote qr` prints a QR payload for SSH or gateway usage (confirm before sharing). -- `hack remote monitor` opens a mini TUI (status + gateway audit log tail). +Start here when you want the product story before the low-level reference: -Cloudflare tunnel helper: +- Core workflows: [docs/README.md](docs/README.md) +- CLI quick map and full reference: [docs/cli.md](docs/cli.md) +- Integrations overview: [docs/integrations.md](docs/integrations.md) -```bash -hack x cloudflare tunnel-setup --hostname gateway.example.com -hack x cloudflare tunnel-start -``` +Beta surfaces are intentionally separate: -Tailscale helper: +- Remote control plane and gateway: [docs/gateway.md](docs/gateway.md) +- Multi-node quickstart: [docs/guides/remote-node-quickstart.md](docs/guides/remote-node-quickstart.md) +- Hack Desktop for macOS: [apps/macos/README.md](apps/macos/README.md) -```bash -hack x tailscale setup -hack x tailscale status -``` - -DNS note: `cloudflared tunnel route dns ` creates the required CNAME to -`.cfargotunnel.com` in your Cloudflare zone (proxied). +Reference surfaces: -### Supervisor (jobs + shells) +- Extensions and SDK: [docs/extensions.md](docs/extensions.md) +- Gateway API: [docs/gateway-api.md](docs/gateway-api.md) +- Architecture: [docs/architecture.md](docs/architecture.md) -The supervisor is the job/shell runner that powers agent workflows and remote execution. It can run -commands, stream logs, and host PTY-backed shells. Use it locally with `hack x supervisor` or -remotely over the gateway. +## `hack help` guide -Docs: `docs/supervisor.md`. +Root help is organized in the same order as the product: -## Docs +- `Global commands`: machine setup and shared services +- `Core workflows`: the day-to-day local-dev path +- `Collaboration & integrations`: env, sessions, tickets, Linear, and SSH access +- `Beta workflows`: remote control plane, multi-node, and dispatch +- `Extension commands`: lower-level extension namespace access -Start here: -- `docs/README.md` (index) -- `docs/architecture.md` -- `docs/gateway.md` -- `docs/extensions.md` +Run `hack help` or `hack help ` for the CLI version of that map. -## Agent setup (CLI-first) +## Agent setup -Use `hack setup` to install local integrations. Default is project scope; add `--global` for user scope. +If you want Hack to teach coding agents how to work in a repo: ```bash hack setup cursor hack setup claude hack setup codex -hack setup agents hack setup sync --all-scopes +hack agent init --client codex ``` -What each setup command does: -- `hack setup cursor`: installs Cursor rules in `.cursor/rules/hack.mdc` -- `hack setup claude`: installs Claude Code hooks in `.claude/settings.local.json` (or user scope) -- `hack setup codex`: installs the Codex skill in `.codex/skills/hack-cli/SKILL.md` -- `hack setup agents`: adds/updates hack usage snippets in `AGENTS.md` and `CLAUDE.md` -- `hack setup mcp`: writes MCP configs for no-shell clients -- `hack setup sync --all-scopes`: refreshes docs, skills, and MCP config at project + user scope - -Automatic guardrail: -- Project-level `hack` commands (interactive sessions) auto-check docs/skills/MCP drift and attempt auto-sync. -- To warn-only: `HACK_SETUP_SYNC_MODE=warn` -- To disable: `HACK_SETUP_SYNC_MODE=off` - -Primer helpers: -- `hack agent prime`: short CLI-first primer used by Claude Code hooks -- `hack agent init`: repo-specific setup prompt agents can follow to scaffold/verify hack config -- `hack agent init --client cursor|claude|codex`: open the prompt directly in an agent client -- `hack agent patterns`: dependency/ops checklist for agents - -Recommended flow: -- Use `hack setup cursor|claude|codex` for your agent client -- Use `hack setup agents` to document hack usage inside the repo -- Use `hack setup sync --all-scopes` after `hack update` to refresh generated integrations -- Use `hack setup mcp` only when the agent has no shell access - -## MCP (no-shell clients) - -Use MCP only when the CLI is not available (e.g. no shell access). - -Run the MCP server locally over stdio: - -```bash -hack mcp serve -``` - -Most MCP clients spawn this on demand. Running it directly will wait for a client connection. - -Install MCP configs (Cursor, Claude CLI, Codex): +If your agent does not have shell access, use MCP instead: ```bash -# Project-scoped (default via setup) hack setup mcp - -# User-scoped (default; writes to your home config directories) -hack mcp install --all - -# Project-scoped (writes .cursor/.claude/.codex in the repo) -hack mcp install --all --scope project -``` - -Project scope writes `.cursor/mcp.json`, `.claude/settings.json`, and `.codex/config.toml`. -For `--scope project`, `hack mcp install` also updates `AGENTS.md` and `CLAUDE.md` by default. - -Print config snippets without writing: - -```bash -hack mcp print --codex -``` - -`hack init` can prompt to install local agent integrations after scaffolding a repo. -For agent-driven scaffolding without prompts, use `hack init --auto` and run `hack setup` manually. - -## Branch builds (worktree-friendly) - -Use `--branch ` on project commands to run isolated instances with unique hostnames and compose -project names: - -```bash -hack up --branch feature-x --detach -hack logs --branch feature-x -hack open --branch feature-x -``` - -Using `--branch` will create/update `.hack/hack.branches.json` with a `last_used_at` timestamp. Branch -instances show up in `hack projects --details`. - -Optional: track branch aliases in `.hack/hack.branches.json` for quick lookup: - -```bash -hack branch add feature-x --note "worktree for PR 123" -hack branch list -hack branch open feature-x -``` - - -## Service-to-service connections (HTTP vs DB/Redis) - -If your app runs in Docker (the default in `hack`), don’t connect to `127.0.0.1` / `localhost` for Postgres/Redis. -Inside a container, `localhost` is that container, not the other compose services. - -If you previously ran everything on your host and used `localhost:PORT`, update those references when you -move into containers: - -- **HTTP services**: use the same `https://*.hack` hostname you open from the host (whatever you configured in Caddy labels, e.g. `https://api.myapp.hack`). -- **Non-HTTP services** (DB/Redis/etc.): use the Compose service hostname (e.g. `db`, `redis`). - -For HTTP services, use the same `https://*.hack` URLs you use on the host. `hack up` injects internal DNS, -TLS trust, and `extra_hosts` mappings so `*.hack` resolves reliably inside containers. If you see `ENOTFOUND` -inside containers, run `hack restart` to refresh the host mappings. - -For non-HTTP services, use the Compose service hostname on the default network: - -- `Postgres: db:5432` -- `Redis: redis:6379` - -Note: Caddy’s CA is mounted into containers when `internal.tls: true` so HTTPS calls to `*.hack` work for most runtimes. -If you’re using Java/Kotlin, you’ll need to import the CA into the JVM truststore manually. - -Example: - -```yaml -environment: - DATABASE_URL: postgres://postgres:postgres@db:5432/mydb - REDIS_URL: redis://redis:6379 -``` - -If you need host access for debugging, prefer `docker compose exec` so you don’t reintroduce port conflicts: - -```bash -docker compose -f .hack/docker-compose.yml exec db psql -U postgres -d mydb -docker compose -f .hack/docker-compose.yml exec redis redis-cli -``` - -## Logs (why both Compose and Loki) - -By default, `hack logs` uses `docker compose logs` because it’s the lowest latency tail. -The daemon does not proxy logs yet; `hack logs` still talks directly to Docker Compose or Loki. - -Loki is still valuable for: - -- querying across time (history) -- filtering by labels (project/service/container) -- Grafana Explore / dashboards - -### CLI - -```bash -# Tail (fast) -hack logs --pretty - -# Snapshot -hack logs --no-follow --pretty - -# Snapshot JSON -hack logs --no-follow --json - -# Query/history (force Loki) -hack logs --loki --pretty - -# Range (Loki only) -hack logs --loki --since 2h --pretty -hack logs --loki --since 4h --until 1h --pretty - -# Filter Loki by service -hack logs --loki --services api,worker --pretty - -# Raw LogQL -hack logs --loki --query '{project="my-project"} |= "error"' --pretty -``` - -### Grafana - -- Open: `hack open logs` or visit `https://logs.hack` -- Explore queries: - -```logql -{project="my-project"} -{project="my-project", service="api"} -``` - -Alloy labels logs with: - -- project: `Docker Compose project name` -- service: `Docker Compose service name` -- container: `Docker container name` - - -## Projects registry (bird’s-eye view) - -`hack` maintains a best-effort registry under `~/.hack/projects.json` so you can target a project from anywhere: - -```bash -hack projects -hack logs --project my-project --pretty -hack up --project my-project -``` - -## .hack and valid tld requirements - -OAuth providers (notably Google) require `localhost` or a host that ends with a real public suffix. - -We keep `.hack` as the primary local dev domain, and optionally expose an alias domain for OAuth flows. -If the OAuth alias is enabled, `hack global install` configures `*.hack.gy` to resolve to the Caddy -container IP via dnsmasq + the OS resolver (bypasses port forwarding issues with Tailscale/VPNs). - -Routing rules: - -- `https://*.hack` is the default local-dev hostname space. -- `https://*.hack.gy` is an OAuth-safe alias host space (public-suffix-style domain). -- Only services with Caddy labels and attached to `hack-dev` are exposed via these hostnames. -- These domains are local routes by default; use tunnels/remote ingress separately for internet-reachable webhooks. - -If you use Next.js (or another dev server that cares about dev origins), configure its dev allowlist to include the proxy domains. -Next.js supports `allowedDevOrigins` (wildcards supported) in `next.config.js`: - -```js -module.exports = { - allowedDevOrigins: ["*.hack", "*.hack.gy"], -} -``` - -Optionally you can pass in your own custom `dev_host` to the config. - - -## SSL - -`hack` uses Caddy’s internal PKI to issue certs for `*.hack` (and any OAuth alias host). This covers -HTTPS for services routed through Caddy, but it does not create cert/key files for services running -outside of Caddy. - -- macOS: run `hack global trust` to trust the Caddy Local CA in the System keychain. -- Other OS: run `hack global ca` to export the CA cert path, then add it to your OS/browser trust store. -- If you need the PEM directly: `hack global ca --print`. -- If you are running a local service outside of Caddy, use `hack global cert ` (mkcert required) to generate a cert/key under `~/.hack/certs` and wire it into your service. This is only needed for non-Caddy services that still want trusted TLS. -- macOS: `hack global install` can optionally install mkcert (needed for `hack global cert`). - -Install mkcert if you don't already have it (macOS example): - -```bash -brew install mkcert -mkcert -install -``` - -Example (non-Caddy service): - -```bash -hack global cert --install api.myapp.hack -``` - -Use `--out ` if you want certs written somewhere else. - -## Internal DNS (containers) - -`hack global install` runs CoreDNS on the `hack-dev` network. CoreDNS answers `*.hack` and `*.hack.*` with -Caddy’s current IP so containers can use the same `https://*.hack` URLs as the host. - -This is the core isolation model: each repo keeps its own compose network, while routed HTTP services get stable hostnames -through shared Caddy ingress. That means you can call local services by hostname from host or container without relying on -host-published `localhost:` patterns. - -Some runtimes don’t honor custom DNS for `*.hack` reliably, so `hack up` also injects `extra_hosts` mappings -to the Caddy IP. If the Caddy IP changes, `hack status`, `hack doctor`, and the TUI show a warning; fix it -with `hack restart` to refresh the mapping. - -If you also need `extra_hosts` for non-`*.hack` hostnames (common when you run host-local tunnels/proxies -and want containers to reach them by their real domain), use `hack internal extra-hosts set` to write a -repo-local `.hack/.internal/extra-hosts.json` that gets merged into the generated Compose override. - -When `internal.tls` is enabled, `hack up` mounts the Caddy Local CA into each container and sets common -SSL env vars so HTTPS to `*.hack` is trusted inside containers. - -If you update `hack`, rerun `hack global install` once to refresh the CoreDNS config. - - - - -## Why not just use X? - -### Docker / Compose alone - -They run containers. They don’t give you stable hostnames, HTTPS, or a way to run many isolated copies of the same stack without custom glue. - -You can build that layer yourself. I did. That’s this. - -### Kubernetes - -Kubernetes solves cluster orchestration. This problem is local parallelism. -It adds complexity without fixing ports, routing, or developer feedback loops. - -### Different ports - -This is the default answer and it doesn’t scale. -Ports leak into config, break OAuth and cookies, and turn into debt. -Hostnames scale. Ports don’t. - -There isn’t an off-the-shelf tool that gives you full local network isolation, real HTTPS, and near zero per-repo setup. - -If you want that, you have to build it yourself. - - -## How it works - -`hack` is a thin layer on top of Docker Compose plus a tiny global proxy. - -- each project/branch runs in its own Docker network -- services use their normal ports inside that network -- a shared proxy routes `https://*.hack` and handles HTTPS -- logs are captured centrally - -Your code doesn’t change. Your mental overhead does. - -### Global (once per machine) - -`hack global install` provisions `~/.hack/` and starts: - -- **Caddy** (`lucaslorentz/caddy-docker-proxy`) on ports `80/443` - - watches Docker labels and auto-routes `https://*.hack` -- **Logging stack**: Grafana + Loki + Alloy - - reachable via `https://logs.hack` - -### Per-project (per repo) - -`hack init` creates `.hack/` in the repo root: - -- `.hack/docker-compose.yml`: your project services -- `.hack/hack.config.json`: project config (name, dev host, log preferences) - -Each project’s compose network stays isolated; only services you want “public” get attached to the shared ingress network so Caddy can reach them. - - -## Common patterns (deps + ops) - -Agents: run `hack agent patterns` for a compact checklist based on this section. - -### 1) Containerized dependency installs (recommended) - -If you run `bun install` on the host and then start Linux containers, you can hit platform -mismatches (native modules, postinstall scripts, OS-specific binaries). The clean pattern is to -install dependencies **inside** Docker and share them via a volume. - -Add a one-shot deps service and make your app services depend on it: - -```yaml -deps: - image: imbios/bun-node:latest - working_dir: /app - volumes: - - ..:/app - - node_modules:/app/node_modules - command: bun install - networks: - - default - -www: - image: imbios/bun-node:latest - working_dir: /app/apps/www - volumes: - - ..:/app - - node_modules:/app/node_modules - depends_on: - deps: - condition: service_completed_successfully -``` - -This keeps dependency resolution in the same OS as your containers and avoids host/guest drift. - -### 2) Networked ops commands (DB schema tooling, migrations) - -Because `hack` avoids publishing DB ports to your host, run schema/ops commands inside the -compose network. - -Option A (recommended): add an ops-only service: - -```yaml -db-ops: - image: imbios/bun-node:latest - working_dir: /app/packages/db # where your db schema + package.json live - volumes: - - ..:/app - - node_modules:/app/node_modules - environment: - DATABASE_URL: postgres://postgres:postgres@db:5432/mydb - depends_on: - - db - - deps - networks: - - default - profiles: ["ops"] - # Examples: - # - Prisma: bunx prisma migrate deploy - # - Drizzle: bunx drizzle-kit push - command: bun run db:push -``` - -Run it on demand: - -```bash -docker compose -f .hack/docker-compose.yml --profile ops run --rm db-ops -``` - -Option B: run via `hack run` (thin wrapper over `docker compose run --rm`): - -```bash -hack run --workdir /app/packages/db email-sync -- bunx prisma generate -hack run --workdir /app/packages/db email-sync -- bunx prisma migrate dev -hack run --workdir /app/packages/db email-sync -- bunx drizzle-kit push -hack run --workdir /app bun run turbo db:migrate -``` - -If your ops service is behind a compose profile, enable it: - -```bash -hack run --profile ops --workdir /app/packages/db db-ops -- bun run db:push +hack mcp serve ``` -See examples: -- `examples/next-app/README.md` - - -## Troubleshooting - -- `*.hack` doesn’t resolve: run `hack doctor`, then `hack global install` (macOS: ensure dnsmasq is running). - -- Stale global setup / CoreDNS issues: run `hack doctor --fix` (refreshes network + CoreDNS + CA). - -- TLS warnings: run `hack global trust` (macOS). - -- Logs missing in Grafana: ensure Alloy is running (`hack global status`) and try `{app="docker"}` in Explore. - -- `ENOTFOUND` for `*.hack` inside containers: run `hack restart` to refresh `extra_hosts` mappings (check - `hack status` or the TUI for Caddy IP mismatch warnings). - -- `EAI_AGAIN` for external domains inside containers (e.g. `api.clerk.com`): CoreDNS isn’t forwarding. - Run `hack global install` and restart CoreDNS: - `docker compose -f ~/.hack/caddy/docker-compose.yml restart coredns`. - -- `hack global up` warns about `hack-dev` network labels or missing subnet: remove the network and reinstall: - `docker network rm hack-dev` then `hack global install`. - -- OAuth redirect errors: use the OAuth alias host (`*.hack.gy`) or `localhost` (providers may reject non-public suffixes like `.hack`). - - - ## Development -### From source - -```bash -bun install -bun run install:dev -hack --help -``` -This installs a small `hack` shim into `~/.hack/bin/hack` that runs your working tree directly (no rebuild needed). - -If `hack` isn’t found, add this to your shell config: - -```bash -export PATH="$HOME/.hack/bin:$PATH" -``` - -### Compiled binary (release-like) +If you are working on Hack itself: ```bash bun install -bun run install:bin -hack --help -``` -This builds `dist/hack` via `bun build --compile` and installs it to `~/.hack/bin/hack`. - -### Check which install is active - -```bash -bun run install:status -``` -Reports whether `hack` is a dev shim or a compiled binary (and where it points). - - -### Run in place - -```bash -bun dev --help -``` - -### Tests - -```bash -bun run test -``` - -Runs workspace tests via Turbo (`@hack/cli`, `@hack/auth-broker`, `@hack/db`). - -### Workspace command model - -Root commands are orchestration wrappers. For package-local work, run commands directly: - -```bash -bun run --cwd packages/cli build -bun run --cwd packages/cli macos:ghostty:bundle -bun run --cwd services/auth-broker test -bun run --cwd packages/db db:migrate -``` - -### Conventional commits - -```bash -bun run commit -``` - -Commitlint runs on `git commit` via husky, and semantic-release uses Conventional Commits to -compute versions. - -### Build a standalone binary - -```bash bun run build -./dist/hack --help -``` - -### Build a release bundle - -```bash -bun run build:release +bun test +bun x ultracite check ``` -Produces `dist/release/hack--release/`, a tarball, and `hack-install.sh`. - -### Release (tag + GitHub workflow) - -```bash -bun run release:prepare -git push --follow-tags -``` - -Updates `CHANGELOG.md` plus root/workspace `package.json` versions, creates the release commit and tag, and triggers the GitHub Release workflow on push. - -See `PACKAGING.md` for details. +Useful repo-level commands: -See also: -- [Examples](examples/next-app/README.md) -- [Docs](docs/README.md) -- [Architecture](docs/architecture.md) +- `bun run build` +- `bun run test` +- `bun run check` +- `bun run macos:build` +## Beta note -

(back to top)

+Remote control plane, multi-node execution, and the macOS app are all still **Beta**. They are +useful, but they are not the core path a new user needs to learn first. diff --git a/apps/macos/README.md b/apps/macos/README.md index d6dd111c5..a9b3ec9eb 100644 --- a/apps/macos/README.md +++ b/apps/macos/README.md @@ -1,7 +1,9 @@ -# Hack Desktop (macOS) +# Hack Desktop for macOS (Beta) Native macOS app for managing local hack projects and daemon status. +> Beta: the macOS app is useful, but it is still outside Hack's core getting-started path. + ## Requirements - macOS 14+ diff --git a/docs/README.md b/docs/README.md index e493be242..b99b525e0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,45 +1,52 @@ # Docs -This directory contains the public documentation for hack. Specs remain in `SPECS/` (working notes). +This directory is organized around the product story instead of the implementation tree. -## Core docs +If you are new to Hack, start with the core path first, then move into integrations, then only +visit the beta and reference surfaces you actually need. +## Core workflows + +- [Initialize a project](guides/init-project.md) - [CLI reference](cli.md) +- [Env & secrets](env.md) +- [Sessions](sessions.md) +- [Lifecycle](lifecycle.md) +- [Global settings](guides/global-settings.md) - [Architecture](architecture.md) -- [Sessions](sessions.md) - mux session management (tmux or zellij) + API -- [Env & secrets](env.md) - env contract + `.hack/.env` + OS keychain secrets -- [Lifecycle](lifecycle.md) - project startup hooks + managed host processes -- [Gateway overview](gateway.md) -- [Gateway API](gateway-api.md) -- [Supervisor](supervisor.md) -- [Extensions](extensions.md) -- [Control-plane SDK](sdk.md) -Monorepo note: -- Root scripts orchestrate workspace tasks through Turbo. -- Package-local commands remain available via `bun run --cwd