Skip to content

fix(check): detect generated-output drift#99

Closed
VVeb1250 wants to merge 4 commits into
sampleXbro:masterfrom
VVeb1250:fix-check-output-drift
Closed

fix(check): detect generated-output drift#99
VVeb1250 wants to merge 4 commits into
sampleXbro:masterfrom
VVeb1250:fix-check-output-drift

Conversation

@VVeb1250

Copy link
Copy Markdown

Summary

  • make agentsmesh check fail when generated outputs are modified, missing, or stale
  • distinguish canonical drift from generated-output drift in JSON with canonicalDrift and outputDrift
  • report actionable outputModified, outputRemoved, and outputStale paths
  • reuse the existing managed-output cleanup inventory for read-only stale detection
  • update the CLI and CI drift documentation and add a patch changeset

Root cause

agentsmesh check only compared canonical files under .agentsmesh/ with .agentsmesh/.lock. It never projected the current canonical state and compared the generated files on disk, so direct output edits could pass CI.

Stale outputs also do not appear in the current generation result. The existing cleanup path knew which output locations AgentsMesh manages, but that discovery was only used by write-mode generation. This change extracts read-only stale discovery for check; deletion remains exclusive to normal agentsmesh generate.

Validation

  • integration JSON suite: 15/15 passed
  • focused check, renderer, target-contract, and stale-cleanup suites: 178/178 passed
  • build passed
  • lint passed
  • typecheck passed
  • documentation site build passed

Fixes #98

@VVeb1250
VVeb1250 marked this pull request as ready for review July 18, 2026 19:07
@sampleXbro sampleXbro added the bug Something isn't working label Jul 19, 2026
@sampleXbro
sampleXbro self-requested a review July 19, 2026 07:45
@sampleXbro

Copy link
Copy Markdown
Owner

Hi @VVeb1250 — thank you for the excellent issue report and for following it up with a working PR. The repro in #98 was precise, and the read-only findStaleGeneratedOutputs refactor here is genuinely clean work.

I want to be upfront: we had a fix for #98 in flight on our side, and after comparing the two approaches carefully I'm going to land ours as the base — but there are pieces of your PR I'd like to bring in on top of it. Here's the reasoning.

Where the approaches differ

Your PR makes check run the full generation pipeline (generate --check internally) and compare the live canonical projection against disk. Ours records checksums of every generated output in .agentsmesh/.lock at generate time (a new outputs map), and check simply re-hashes those paths.

We went with the lock-checksum route for four reasons:

  1. It preserves the check vs generate --check contract. check stays the fast, hash-only CI gate; the full-pipeline verification remains generate --check's job. With the projection approach, check roughly doubles CI cost on every run for larger configs and the "fast check" story disappears.
  2. Clean drift separation. With the projection approach, editing a canonical file without regenerating flips both canonicalDrift and outputDrift, and every affected output lands in outputModified — in the most common drift scenario the output list is noise. With recorded checksums, a canonical edit reports only modified, and a hand-edited output reports only outputsModified.
  3. An escape hatch for gitignored outputs. Some teams gitignore generated outputs; after a CI checkout every output would report as missing under the projection approach, with no way out. Our version ships check --no-outputs for exactly that workflow.
  4. Coverage of the other surfaces. The MCP check tool and the programmatic API needed the same fix — the MCP handler was silently skipping output verification entirely.

The one real trade-off in favor of your approach: it works on existing locks with zero migration. Ours needs one agentsmesh generate to upgrade the lock — until then check reports outputsChecked: false with a hint, rather than failing.

What I'd like to adopt from this PR

  • outputStale detection — hand-added/orphaned files in managed output locations are something our implementation deliberately punted on, and your read-only split of cleanupStaleGeneratedOutputs into findStaleGeneratedOutputs is exactly the right shape for it. If you're up for it, I'd welcome a rebased version of this PR scoped to that piece once our base lands; otherwise I'll follow up and credit you.
  • The canonicalDrift / outputDrift booleans — your naming matches what the issue asked for, and they're cheap to add additively on top of our per-path arrays.

The implementation commit is here and you can compare it directly. Thanks again — both the report and the PR moved this forward materially, and I'd be glad to have the stale-detection piece land under your name.

@sampleXbro sampleXbro closed this Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs/behavior mismatch: agentsmesh check ignores generated-output drift

2 participants