Skip to content

fix(tests): purge deleted tests, bill their storage, and record versions as loaded - #8858

Merged
TheodoreSpeaks merged 2 commits into
stagingfrom
fix/tests-release-review
Oct 9, 2026
Merged

TheodoreSpeaks merged 2 commits into
stagingfrom
fix/tests-release-review

Conversation

@TheodoreSpeaks

@TheodoreSpeaks TheodoreSpeaks commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Addresses the workflow-test review comments on the v0.9.17 release PR (#8853).

Changes

  • Deleted tests block cleanup (Greptile P1): retention cleanup deleted a test's source file while the soft-deleted workflow_test row still referenced it, so the foreign key rejected the delete on every run. Cleanup now removes expired test rows (their runs cascade) in the same transaction, before the file.
  • Deleted tests keep consuming quota (Greptile P1): test sources are billed on create, but cleanup treated every non-workspace context as unbilled. test files now go through the billed path, which releases their versions and decrements the payer.
  • Changed workflows look tested (Greptile P1): the run recorded draft timestamps and child deployments after it finished. The executor now reports the version it loaded (currentWorkflow on its delegation origin) through enterWorkflow, and the session records each draft's updatedAt at that point.
  • Org-chat submenus offered tests (Cubic): workspace submenus now exclude the composer's excluded types.
  • Open test (Cubic): opens a new tab on web, like other internal links.
  • Doc fixes: hook inheritance exceptions (custom-block sources, workflows called as Agent tools) and mockSampleOutput wording.

Verification

  • New background/cleanup-soft-deletes.integration.ts against real Postgres: an expired test, its run, and its source are purged, a live one is kept, and workspace + payer storage drop by the source's bytes. It fails on the pre-fix cleanup.
  • Real in-process test-file run of a parent → child workflow: recorded versions match the DB exactly for both workflows, in draft (updatedAt) and deployed (active deployment ids).
  • lint:check, check:audits, and the executor / workflow-tests / cleanup unit suites pass.

Companion PRs

  • simstudioai/mothership#665

🤖 Generated with Claude Code

…ons as loaded

- Retention cleanup removes an expired test's row before its source file, which its
  foreign key blocked, and treats test sources as billed so their bytes are released.
- A run records each workflow's deployment, and a draft's timestamp, as the executor
  loads it, so an edit or redeploy mid-run no longer makes the run look current.
- Organization-chat workspace submenus leave tests out, like the composer.
- Open test opens a new tab on web; doc fixes for hook inheritance and mockSampleOutput.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Oct 9, 2026 6:03pm UTC

Request Review

@github-actions github-actions Bot added the requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

  • ❌ simstudioai/mothership#665 — OPEN, not merged (targets staging) — fix(worker): describe mockSampleOutput as setting the mock's answer

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@greptile review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/workflow-tests/session.ts Outdated
@greptile-apps

greptile-apps Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Critical impact] No new blocking issue was found in the changes since the previous review.

Summary

This PR fixes deleted-test cleanup and storage accounting, records loaded workflow versions, and adjusts test links and composer menus. The latest revision reads draft timestamps before starting the executor.

  • Expired tests and their sources are removed together, freeing their storage.
  • Test results name the workflow versions that actually ran.
  • Organization chat menus leave excluded resources out of workspace submenus.
  • Embedded tests open in a new web tab and stay in place on desktop.
  • Test help describes sample outputs as mock answers.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Start a draft test] --> B[Read workspace draft timestamps]
  B --> C[Load and execute workflow]
  C --> D[Record entered workflow versions]
  D --> E[Save test result and execution links]
  E --> F[Compare saved timestamps with current drafts]
Loading

Reviews (2) · Last reviewed commit: "fix(tests): take a draft's timestamp fro..." · Reviewed by Greptile

Comment thread apps/sim/lib/workflow-tests/session.ts Outdated
Comment thread apps/sim/background/cleanup-soft-deletes.test.ts
Reading it after the load let an edit landing in between look tested; the snapshot from
before the load can only err toward stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@greptile review

@TheodoreSpeaks
TheodoreSpeaks merged commit 1266679 into staging Oct 9, 2026
46 checks passed
@TheodoreSpeaks
TheodoreSpeaks deleted the fix/tests-release-review branch October 9, 2026 18:46

This branch was previously deployed

1 inactive deployment
Preview — 2ab0a2c4 Deployed Oct 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant