Skip to content

Test suites leave temporary directories behind on every run #124

Description

@shibayan

What happens

Several test suites create temporary directories under the OS temp directory and never remove them. They accumulate across runs on a developer's machine.

Measured on a Windows developer machine after ordinary use of the repository:

1560  afjs-fdry-*      (packages/foundry)
 288  afjs-replay-*
  61  afjs-bg-*        (packages/agentserver background tests)
  15  afjs-conv-*
  15  afjs-cont-*

A single vitest run packages/agentserver packages/foundry adds seven more (six afjs-fdry-*, one afjs-replay-*), so the count grows with every run rather than being bounded.

For contrast, the state-root isolation added for the response-store default (scripts/test-state-root.ts) removes its directory in afterAll and leaves nothing behind — afjs-state-root-* measures zero after a full run. The suites above predate it and have no equivalent cleanup.

Why this matters

  • The directories hold whatever those tests wrote, including serialized transcripts and response envelopes, in the clear, indefinitely.
  • On a long-lived machine the count reaches thousands, which slows directory listings and temp-cleaner passes.
  • A test that reads "everything under the root" can observe another run's leftovers if a future change ever shares a root.

Required implementation

  • Find every suite that creates a temporary directory and give it a cleanup that runs even when the suite fails.
  • Prefer the pattern scripts/test-state-root.ts uses: name the directory rather than creating it eagerly, so a fully skipped test file leaves nothing (Vitest runs no afterAll for a file whose tests are all skipped), and remove it in afterAll with { recursive: true, force: true }.
  • Where several tests in one file each make their own directory, one per-file root with per-test subdirectories is enough.
  • Do not delete anything outside the directory the suite itself created.

Acceptance criteria

  • A reproduction-first measurement records the leftover count before and after a full pnpm check, and the after-count for newly created directories is zero.
  • Every suite that creates a temporary directory removes it, including when a test in that file fails.
  • A fully skipped test file leaves no directory behind.
  • The prefixes above (afjs-fdry, afjs-replay, afjs-bg, afjs-conv, afjs-cont) are each covered, and a sweep confirms no other prefix is left uncovered.
  • Cleanup failures do not fail an otherwise passing suite.
  • pnpm check passes.

Out of scope

Removing directories left by earlier runs. This issue stops the growth; existing leftovers are the developer's to clear.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agentserverUsage: [Issues, PRs], Target: packages/agentserverfoundryUsage: [Issues, PRs], Target: Microsoft Foundry integrationsmaintenanceUsage: [Issues, PRs], Target: CI, tooling, configuration, and internal maintenance

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions