diff --git a/labs/12-product-engineering-loop/boatstack-distribution/TROUBLESHOOTING.md b/labs/12-product-engineering-loop/boatstack-distribution/TROUBLESHOOTING.md index 38aea6cc..c0693888 100644 --- a/labs/12-product-engineering-loop/boatstack-distribution/TROUBLESHOOTING.md +++ b/labs/12-product-engineering-loop/boatstack-distribution/TROUBLESHOOTING.md @@ -137,6 +137,8 @@ Use the installer for the target release in update mode. It downloads and verifi If the ignored local install lock says `dev`, has an unknown source commit, is malformed, or is absent, rerun the same verified target installer with `BOATSTACK_REPAIR=1`. The target helper recovers the prior stable version from the repository's committed generated pin and repairs only verified Boatstack-owned state. Do not delete repository files or move detached state by hand. Detached operation receipts and runtime slots are validated against their external Boatstack ownership root, so a path under `Application Support/boatstack` is not treated as a repository escape. +If a previous update succeeded but its generated diff was later discarded, rerun the verified target installer. Boatstack now checks the current repository and runtime postcondition before consuming the detached success receipt. A clean old-pin worktree reopens only the same local update and regenerates it; publication and other external terminal receipts remain closed. + ## A tool call repeats or publication appears stuck Run `.product-loop/bin/boatstack-helper operation-status --repo . --json`. `EXECUTING` means the exact call already has a live lease, so wait instead of launching it again. `RECONCILE_REQUIRED` means Boatstack did not observe completion; verify the reported Git, GitHub, file, browser, or MCP postcondition before retrying. A successful operation whose response was lost is recovered from that observation. Do not reset the task, repeat a denied push, or open another PR. diff --git a/labs/12-product-engineering-loop/boatstack-distribution/release-notes/2026-08-01-terminal-update-postcondition.md b/labs/12-product-engineering-loop/boatstack-distribution/release-notes/2026-08-01-terminal-update-postcondition.md new file mode 100644 index 00000000..013fcff8 --- /dev/null +++ b/labs/12-product-engineering-loop/boatstack-distribution/release-notes/2026-08-01-terminal-update-postcondition.md @@ -0,0 +1,3 @@ +### Update receipts verify current state + +Boatstack no longer treats a detached `SUCCEEDED` update receipt as permanent proof after its generated repository diff has been discarded. A clean retry verifies the target bundle, hooks, runtime pin, helper identity, and preserved integrations; when that postcondition is missing, it safely reopens only the bounded local update and regenerates the infrastructure diff. External publication receipts retain their existing duplicate-suppression behavior. diff --git a/labs/12-product-engineering-loop/product-engineering-loop/init.go b/labs/12-product-engineering-loop/product-engineering-loop/init.go index de385e6e..d5f047cb 100644 --- a/labs/12-product-engineering-loop/product-engineering-loop/init.go +++ b/labs/12-product-engineering-loop/product-engineering-loop/init.go @@ -793,7 +793,8 @@ func RunUpdate(options InitOptions) error { return reexecUpdate(options.BinaryPath, options) } } - config, _, configErr := LoadConfig(WorkspaceFor(repo).SourceConfigPath()) + configPath := WorkspaceFor(repo).SourceConfigPath() + config, rawConfig, configErr := LoadConfig(configPath) if configErr != nil { return configErr } @@ -827,15 +828,27 @@ func RunUpdate(options InitOptions) error { if err != nil { return err } + if receipt.State == OperationSucceeded { + postconditionErr := verifyInstalledUpdatePostcondition(repo, configPath, config, rawConfig, preflight.PreservedIntegrations) + if postconditionErr == nil { + return nil + } + receipt, err = reconcileSucceededInstallUpdate(repo, receipt.OperationID, + "the previously successful local update no longer matches its target postcondition", + postconditionErr.Error()) + if err != nil { + return err + } + } begin, err := BeginOperation(repo, receipt.OperationID, SHA256Bytes([]byte("install-update\x00"+packageFingerprint)), "boatstack-helper update") if err != nil { if begin.Receipt.State == OperationSucceeded { - return nil + return verifyInstalledUpdatePostcondition(repo, configPath, config, rawConfig, preflight.PreservedIntegrations) } return err } if begin.Receipt.State == OperationSucceeded { - return nil + return verifyInstalledUpdatePostcondition(repo, configPath, config, rawConfig, preflight.PreservedIntegrations) } options.Update = true if err := RunInit(options); err != nil { @@ -845,3 +858,47 @@ func RunUpdate(options InitOptions) error { _, err = CompleteOperation(repo, receipt.OperationID, begin.LeaseToken, "SUCCEEDED", "post-install doctor and generated projections passed", Version) return err } + +func verifyInstalledUpdatePostcondition(repo, configPath string, config ProjectConfig, rawConfig []byte, expectedIntegrations map[string]IntegrationState) error { + bundle, err := BuildExportBundle(configPath, config, rawConfig, "boatstack") + if err != nil { + return err + } + if err := CheckExport(repo, bundle.Files); err != nil { + return err + } + if err := CheckHostHooks(repo, config.Adapters); err != nil { + return err + } + for _, host := range config.Adapters { + path := executionInterceptorPath(host) + if path == "" { + continue + } + items := classifyExecutionInterceptor(repo, host) + if len(items) != 1 || items[0].Classification != RepairCurrent { + return fmt.Errorf("target execution interceptor is not current: %s", path) + } + } + if err := verifyGeneratedRuntime(repo); err != nil { + return err + } + manifest, _, err := loadSharedRuntime(repo) + if err != nil { + return err + } + if err := verifyLocalRuntime(repo); err != nil { + return err + } + installedIntegrations, err := readInstalledIntegrations(repo, config) + if err != nil { + return err + } + if expectedIntegrations == nil { + expectedIntegrations = config.Integrations + } + if !sameJSON(installedIntegrations, expectedIntegrations) || !sameJSON(manifest.Integrations, expectedIntegrations) { + return fmt.Errorf("installed integration state does not match the update postcondition") + } + return nil +} diff --git a/labs/12-product-engineering-loop/product-engineering-loop/installation_repair_test.go b/labs/12-product-engineering-loop/product-engineering-loop/installation_repair_test.go index d6281e51..819b18f4 100644 --- a/labs/12-product-engineering-loop/product-engineering-loop/installation_repair_test.go +++ b/labs/12-product-engineering-loop/product-engineering-loop/installation_repair_test.go @@ -375,6 +375,99 @@ func TestDetachedUpdateRepairsInvalidLocalProvenanceEndToEnd(t *testing.T) { } } +func TestDetachedUpdateReconcilesSucceededReceiptAfterCommittedPinIsRestored(t *testing.T) { + now := time.Date(2026, 8, 1, 13, 0, 0, 0, time.UTC) + withUpdateGlobals(t, "v0.4.0", now, func() (ReleaseInfo, error) { return ReleaseInfo{}, nil }) + repo, _ := updateInstalledRepo(t) + t.Setenv(stateRootEnv, t.TempDir()) + invalidateWorkspaceCache() + if _, err := AttachDetached(AttachOptions{Repo: repo}); err != nil { + t.Fatal(err) + } + runGit(t, repo, "switch", "-c", "chore/update-boatstack-v0.5.0") + Version = "v0.5.0" + SourceCommit = "update-test-0.5.0" + options := InitOptions{Repo: repo, Repair: true, Yes: true, Input: strings.NewReader(""), Output: &bytes.Buffer{}} + if err := RunUpdate(options); err != nil { + t.Fatal(err) + } + receipts, err := operationReceipts(repo) + if err != nil || len(receipts) == 0 { + t.Fatalf("first update receipt missing: %#v %v", receipts, err) + } + var updateID string + for _, receipt := range receipts { + if receipt.Kind == "install-update" && receipt.State == OperationSucceeded { + updateID = receipt.OperationID + } + } + if updateID == "" { + t.Fatalf("successful update receipt missing: %#v", receipts) + } + + // Simulate an operator discarding the generated update diff while ignored + // runtime state and the detached terminal receipt survive. + runGit(t, repo, "restore", ".") + if status := gitPorcelain(t, repo); status != "" { + t.Fatalf("fixture did not restore a clean old-pin worktree: %s", status) + } + if err := verifyGeneratedRuntime(repo); err == nil { + t.Fatal("restored old pin unexpectedly matched the target runtime") + } + + options.Input = strings.NewReader("") + options.Output = &bytes.Buffer{} + if err := RunUpdate(options); err != nil { + t.Fatalf("clean retry did not reconcile the stale terminal receipt: %v", err) + } + if err := verifyGeneratedRuntime(repo); err != nil { + t.Fatalf("clean retry did not regenerate the target pin: %v", err) + } + reconciled, err := loadOperation(repo, updateID) + if err != nil || reconciled.State != OperationSucceeded || reconciled.Attempt != 1 || reconciled.Observation.Status != "SUCCEEDED" { + t.Fatalf("reconciled update did not finish as a fresh bounded attempt: %#v %v", reconciled, err) + } +} + +func TestUpdatePostconditionVerifierIsReadOnlyWhenCurrent(t *testing.T) { + now := time.Date(2026, 8, 1, 13, 0, 0, 0, time.UTC) + withUpdateGlobals(t, "v0.4.0", now, func() (ReleaseInfo, error) { return ReleaseInfo{}, nil }) + repo, _ := updateInstalledRepo(t) + runGit(t, repo, "switch", "-c", "chore/update-boatstack-v0.5.0") + Version = "v0.5.0" + SourceCommit = "update-test-0.5.0" + options := InitOptions{Repo: repo, Yes: true, Input: strings.NewReader(""), Output: &bytes.Buffer{}} + if err := RunUpdate(options); err != nil { + t.Fatal(err) + } + receipts, err := operationReceipts(repo) + if err != nil { + t.Fatal(err) + } + var before OperationReceipt + for _, receipt := range receipts { + if receipt.Kind == "install-update" { + before = receipt + } + } + configPath := WorkspaceFor(repo).SourceConfigPath() + config, rawConfig, err := LoadConfig(configPath) + if err != nil { + t.Fatal(err) + } + preflight, err := ClassifyInstallationRepair(repo, config.Adapters, false) + if err != nil { + t.Fatal(err) + } + if err := verifyInstalledUpdatePostcondition(repo, configPath, config, rawConfig, preflight.PreservedIntegrations); err != nil { + t.Fatalf("current update postcondition did not verify: %v", err) + } + after, err := loadOperation(repo, before.OperationID) + if err != nil || after.Attempt != before.Attempt || after.UpdatedAt != before.UpdatedAt { + t.Fatalf("current terminal receipt was needlessly reopened: before=%#v after=%#v err=%v", before, after, err) + } +} + func TestRepairReconstructsCorruptGeneratedProvenance(t *testing.T) { now := time.Date(2026, 7, 23, 12, 0, 0, 0, time.UTC) withUpdateGlobals(t, "v0.4.0", now, func() (ReleaseInfo, error) { return ReleaseInfo{}, nil }) diff --git a/labs/12-product-engineering-loop/product-engineering-loop/operation.go b/labs/12-product-engineering-loop/product-engineering-loop/operation.go index 1f24cfe2..7f8f4dfe 100644 --- a/labs/12-product-engineering-loop/product-engineering-loop/operation.go +++ b/labs/12-product-engineering-loop/product-engineering-loop/operation.go @@ -8,6 +8,7 @@ import ( "os" "path/filepath" "regexp" + "runtime" "sort" "strings" "time" @@ -253,16 +254,28 @@ func withOperationLock(repo, id string, apply func() error) error { return fmt.Errorf("operation %s is busy", id) } -// Windows can report ERROR_ACCESS_DENIED when another process owns an O_EXCL -// lock file. Treat that as contention only when the lock path actually exists; -// genuine directory/ACL permission failures still fail closed. +// Windows can report ERROR_ACCESS_DENIED while another process owns or has just +// released an O_EXCL lock file. Retry that condition within the caller's bounded +// budget; other platforms require the lock path to exist. func isLockContention(openErr error, lock string) bool { + return isLockContentionForOS(openErr, lock, runtime.GOOS) +} + +func isLockContentionForOS(openErr error, lock, goos string) bool { if os.IsExist(openErr) { return true } if !os.IsPermission(openErr) { return false } + // Windows can keep an exclusive lock handle alive briefly after the owner + // removes its directory entry. During that interval OpenFile reports + // ERROR_ACCESS_DENIED while a following Stat can already report not-exist. + // Retry within the caller's fixed budget; a real ACL failure still exhausts + // that budget without entering the critical section. + if goos == "windows" { + return true + } _, statErr := os.Stat(lock) return statErr == nil } @@ -442,6 +455,42 @@ func BeginOperation(repoPath, id, attemptKey, tool string) (OperationBeginResult return result, err } +// reconcileSucceededInstallUpdate reopens only a local atomic Boatstack update +// whose previously observed postcondition no longer holds. A terminal receipt is +// evidence about an observation in time, not permanent authority to suppress a +// later explicit update after the repository was restored or otherwise regressed. +// Other operation kinds keep their existing terminal replay semantics. +func reconcileSucceededInstallUpdate(repoPath, id, detail, evidence string) (OperationReceipt, error) { + repo, err := ResolveRepository(repoPath) + if err != nil { + return OperationReceipt{}, err + } + var result OperationReceipt + err = withOperationLock(repo, id, func() error { + receipt, loadErr := loadOperation(repo, id) + if loadErr != nil { + return loadErr + } + if receipt.State != OperationSucceeded { + return fmt.Errorf("operation %s is no longer a succeeded update", id) + } + if receipt.Kind != "install-update" || receipt.RetryClass != "ATOMIC_LOCAL" { + return fmt.Errorf("operation %s does not support terminal postcondition reconciliation", id) + } + receipt.State = OperationRetryable + receipt.Attempt = 0 + receipt.Lease = nil + receipt.Observation = OperationObservation{ + Status: "POSTCONDITION_MISSING", Detail: boundedObservation(detail), + Evidence: boundedObservation(evidence), At: operationTimestamp(), + } + receipt.UpdatedAt = operationTimestamp() + result = receipt + return saveOperation(repo, receipt) + }) + return result, err +} + func completeOperation(repoPath, id, leaseToken, attemptKey, outcome, detail, evidence string, trustedAttempt bool) (OperationReceipt, error) { repo, err := ResolveRepository(repoPath) if err != nil { diff --git a/labs/12-product-engineering-loop/product-engineering-loop/operation_test.go b/labs/12-product-engineering-loop/product-engineering-loop/operation_test.go index e9342024..6e1eb249 100644 --- a/labs/12-product-engineering-loop/product-engineering-loop/operation_test.go +++ b/labs/12-product-engineering-loop/product-engineering-loop/operation_test.go @@ -264,6 +264,21 @@ func TestOperationLifecycleAndReplayProtection(t *testing.T) { } } +func TestTerminalPostconditionReconciliationIsUpdateLocalOnly(t *testing.T) { + repo := operationTestRepo(t) + receipt := preparedOperation(t, repo, "generic-package", "ATOMIC_LOCAL", 2) + begin, err := BeginOperation(repo, receipt.OperationID, "generic-attempt", "Write") + if err != nil { + t.Fatal(err) + } + if _, err := CompleteOperation(repo, receipt.OperationID, begin.LeaseToken, "SUCCEEDED", "observed", "artifact"); err != nil { + t.Fatal(err) + } + if _, err := reconcileSucceededInstallUpdate(repo, receipt.OperationID, "missing", "artifact"); err == nil || !strings.Contains(err.Error(), "does not support") { + t.Fatalf("generic terminal operation was reopened: %v", err) + } +} + func TestOperationRejectsChangedPackageAndAuthorization(t *testing.T) { repo := operationTestRepo(t) receipt := preparedOperation(t, repo, "package-auth", "ATOMIC_LOCAL", 2) @@ -357,6 +372,18 @@ func TestOperationLedgerIsolatedPerWorktree(t *testing.T) { } } +func TestWindowsAccessDeniedLockRaceIsContentionAfterEntryDisappears(t *testing.T) { + lock := filepath.Join(t.TempDir(), "removed.lock") + openErr := &os.PathError{Op: "open", Path: lock, Err: os.ErrPermission} + + if !isLockContentionForOS(openErr, lock, "windows") { + t.Fatal("Windows access-denied race was not classified as bounded contention") + } + if isLockContentionForOS(openErr, lock, "linux") { + t.Fatal("missing Unix lock path was misclassified as contention") + } +} + // TestSameVersionUpdateFromTwoWorktreesDoesNotCollide reproduces the reported // incident: a same-version install-update prepared from a second worktree used to // fail with "existing operation identity does not match the prepared package" diff --git a/labs/12-product-engineering-loop/product-engineering-loop/references/failure-moves.md b/labs/12-product-engineering-loop/product-engineering-loop/references/failure-moves.md index 8895bc8f..f7d608e9 100644 --- a/labs/12-product-engineering-loop/product-engineering-loop/references/failure-moves.md +++ b/labs/12-product-engineering-loop/product-engineering-loop/references/failure-moves.md @@ -28,6 +28,7 @@ The `root-cause` operation operationalizes this taxonomy for a single bug: it cl | Cross-clone runtime-absence lockout | Only pointers (the guard's baked version path, the committed version pin) travel through Git; the version-keyed runtime bytes are gitignored and delivered out of band. So a teammate who pulls a merged version bump — or clones fresh — holds the new pointers but an empty slot, and the guard fail-closes ("shared runtime is missing") before any Go runs, stranding every teammate on every bump until each manually re-installs | The guard auto-hydrates an absent slot by running the tag-pinned, `.sha256`-verified installer in a branch-free, slot-only `hydrate-runtime` mode, serialized clone-wide by an atomic `mkdir` lock and bounded by a timeout, then falls through to the existing gates which stay authoritative and fail-closed; the deny message embeds the exact one-line self-heal, and `BOATSTACK_AUTO_HYDRATE=0` is the kill switch. Hydration refuses any running-vs-pin identity mismatch and touches no committed generated file | Running a fetched installer on cold start (bounded by tag pinning, HTTPS, sidecar verification, the guard's own checksum re-verify before `exec`, and the kill switch), or falling open — hydration is additive only, never a new authority for `exec` | | Post-publication correction routing | CI, review, or a denied push targets work already marked published — including a published-but-open *earlier slice* inside a still-active delivery, which the pointer-based advisors mis-routed to the active slice | Resolve the target through the same addressable-slice set the actuators use ({active slice} ∪ {published slices whose PR is not terminal}), keyed off the correction's branch: route a non-terminal published slice to an in-place re-gate/`publish-pr --action update` of *that* slice (not the active one), and only a terminal PR to an independently approved corrective child. Run recovery routing, change recording, and the publication-bypass finding through that one resolver so the advisor layer cannot drift from the actuator layer | Treating PR creation as completion, asking the user to bypass the guard, or an advisor/actuator addressability split-brain that repairs the wrong slice | | Unobserved side-effect completion | The same visible state could mean not started, executing, succeeded with a lost response, or failed | Durable operation receipt; exact lease; observe completion; reconcile the expected postcondition before retry | Conversation-scoped retry loops, duplicate PRs, or phantom success | +| Postcondition-regressed terminal receipt | A durable local operation once reached `SUCCEEDED`, but its repository postcondition was later restored or regressed while the detached receipt survived; a retry consumes the terminal identity and returns success without rebuilding the missing state | Before consuming terminal success, verify the operation's declared postcondition against current repository and runtime state; keep success when it holds, otherwise reopen only the same bounded atomic-local operation and re-execute under its existing authority | Globally reopening terminal receipts, retrying irreversible external effects, or trusting the receipt instead of the current postcondition | | Unregistered malformed draft lockout | A hand-authored feature `plan.md` never passed through the helper, so a `CheckPlan` failure escalates to `INVALID_STATE` and the guard denies every product mutation, including the prescribed recovery | `repair-state` quarantines the draft out of `features/` and returns the workflow to `auto-plan`, refusing any directory with a lock, `pr.md`, delivery state, or tracked files | Loosening candidate selection so a genuinely invalid plan silently unblocks product edits | | Premature supervisory pointer advance | A durable supervisory pointer/state advances on request-success and revokes the correction actuator for a target whose postcondition (CI, merge) is not yet observed, so the stranded target can never be re-addressed | Separate the advance from correctability: keep a bounded in-place actuator for a non-terminal target (re-gate/re-publish the same open PR) and a bounded forward actuator once it is terminal (corrective child); resolve addressability network-free from a persisted terminal-state cache, never advance a supervisory pointer past an unobserved postcondition | Serializing legitimately-parallel work by refusing to advance, or persisting an identity/status that deadlocks the corrected retry | | Non-transactional multi-file promote | A managed artifact spans files that must land together (e.g. the compiled `tasks.json`, `test-matrix.json`, `evidence.md`, and the `plan.lock.json` that binds them), but independent non-atomic writes can leave a partial set on a crash or a failed post-write check | Promote the whole set through the transactional mutation boundary as one mutation: base-hash preconditions, supervisor-authority binding, atomic all-or-nothing write, post-write verification with automatic rollback, and a reversible receipt whose inverse bytes make the boundary closed under inversion — `undo` re-applies the inverse as a mutation (with redo as undo-of-the-undo), and a domain guard refuses reversal once a delivery gate would be stranded | Patching consistency after the fact with hash guards instead of making the promote atomic, persisting a rejected identity so a corrected retry deadlocks, or undoing an activation that strands live delivery state | diff --git a/labs/12-product-engineering-loop/product-engineering-loop/references/workflow.md b/labs/12-product-engineering-loop/product-engineering-loop/references/workflow.md index a8b9b50b..a528e198 100644 --- a/labs/12-product-engineering-loop/product-engineering-loop/references/workflow.md +++ b/labs/12-product-engineering-loop/product-engineering-loop/references/workflow.md @@ -442,6 +442,8 @@ For an available version, create `chore/update-boatstack-v` and downloa If the ignored local install lock is missing, malformed, or carries a development identity, the verified target helper derives the prior stable version only from `HEAD:.product-loop/generated.lock.json`. That committed pin makes the local provenance path repairable without trusting drifted worktree bytes. Every mutable controller target is paired with its owning storage boundary: embedded worktree state uses the worktree Git directory, embedded shared state uses the Git common directory, and detached state uses the external Boatstack control root. Effectful callers validate the paired boundary and never reconstruct it from the repository path. +A terminal update receipt is consumed only while its target postcondition still holds. Before returning success for a prior `install-update`, Boatstack checks the target generated bundle, host hooks, execution interceptors, committed runtime pin, shared and local runtime identity, and preserved integrations. If an operator restored the old committed pin or otherwise removed that local atomic result, Boatstack records `POSTCONDITION_MISSING`, reopens only that `ATOMIC_LOCAL` update, and performs a fresh bounded attempt. PR publication and other external operations retain terminal replay suppression and are never reopened by this rule. + `update -binary ` installs the passed binary's **own self-reported version**, not the running helper's. Because each helper embeds its own version-bound generated bundle and compile-time constants, an older helper cannot correctly install a newer one in-process; when the passed binary self-reports a different identity, the whole update is re-executed by that binary so it installs itself — its bundle, constants, version-keyed shared-runtime slot, and durable receipt are then authoritative by construction, and the hand-off terminates in a single hop. The write boundary refuses to install a `-binary` whose self-report disagrees with the process running it, and re-hashes the freshly written slot against its manifest, rolling back on mismatch — so a runtime can never be labeled one version while carrying another's bytes. The runtime bytes never travel through Git — only the guard's baked version path and the committed version pin do — so a teammate who pulls a merged version bump, or clones fresh, starts with the new pointers but an **empty**, gitignored, version-keyed shared slot. Rather than fail-close every such teammate until they re-install by hand, the safety guard **auto-hydrates** an absent slot: it runs the tag-pinned, `.sha256`-verified installer in a branch-free, slot-only `hydrate` mode, serialized clone-wide by an atomic `mkdir` lock (peers wait briefly for the slot to appear) and bounded by a timeout, then falls through to the existing missing/symlink/manifest/checksum gates. Hydration is strictly additive: those gates remain the sole authority for execution and stay fail-closed, so a disabled, timed-out, or failed hydration simply denies — now with the exact one-line self-heal command embedded in the message. The `hydrate-runtime` helper subcommand it invokes rewrites no committed generated file and requires no dedicated branch; it refuses to populate a slot whose identity disagrees with the worktree's pin, and since the installer downloads the exact pinned version first, running equals installed by construction (the runtime-cache re-hash-and-rollback is the backstop). This is a deliberate posture change — the guard runs a fetched installer on cold start — bounded by tag pinning, HTTPS, sidecar verification, the guard's own checksum re-verify before `exec`, the clone-wide lock, the timeout, and the `BOATSTACK_AUTO_HYDRATE=0` kill switch (with a `BOATSTACK_HYDRATE_COMMAND` override). It never becomes a new authority for execution.