🤖 feat: add background bash process execution with SSH support (#920)

### Stack:

1. #923 

1. #920 (base) <- This PR

---

## Summary

Adds `run_in_background=true` option to the bash tool, enabling agents
to spawn long-running processes (dev servers, builds, file watchers)
that persist independently.

## Why This Approach

We needed background process support that works identically for both
local and SSH runtimes. A few considerations drove the design:

1. **PTY-based output doesn't fit remote use cases.** For SSH,
maintaining a persistent PTY connection just to let agents read stdout
would be fragile and complex. Agents need to search, filter, and tail
output—not consume a live stream.

2. **File-based output lets agents use standard tools.** By writing
stdout/stderr to files on the target machine, agents can use `tail -f`,
`grep`, `head`, etc. to inspect output. We don't need to reimplement
these filtering capabilities in our own tooling.

3. **A proper long-lived remote daemon is future work.** Ideally, SSH
remotes would have a persistent mux process (or agent binary) that
manages background jobs directly. The user's frontend would just connect
to it. That's a significant architectural change. This PR provides
background shell support without requiring that investment—the
file-based approach works today with no remote-side dependencies.

## Architecture

```
AI Tools (bash, bash_background_list, bash_background_terminate)
    ↓
BackgroundProcessManager (lifecycle, in-memory tracking)
    ↓
Runtime.spawnBackground() (LocalBaseRuntime / SSHRuntime)
    ↓
BackgroundHandle (file-based output & status)
    ↓
backgroundCommands.ts (shared shell builders for Local/SSH parity)
```

## Key Design Decisions

| Decision | Rationale |
|----------|-----------|
| **File-based output** | Works identically for local and SSH, agents
read via `tail`/`cat`/`grep`. |
| **set -m + nohup** | Robust process isolation, PID === PGID for clean
group termination |
| **Workspace-scoped** | Processes tied to workspace, cleaned up on
workspace removal |
| **Lazy status refresh** | No polling overhead, reads `exit_code` file
on list() |
| **Cleanup on compaction** | Background processes terminated before
compaction, as they're not guaranteed to be included (Claude Code does
this too) |

## Tools

- `bash(run_in_background=true)` — spawns process, returns
`stdout_path`/`stderr_path`
- `bash_background_list` — lists processes with status and file paths
- `bash_background_terminate` — kills process group (SIGTERM → wait →
SIGKILL)

## Output Structure

```
/tmp/mux-bashes/{workspaceId}/{bg-xxx}/
├── stdout.log    # Process stdout
├── stderr.log    # Process stderr  
├── exit_code     # Written by trap on exit
└── meta.json     # Process metadata
```

## Technical Details

### Process Spawning

The spawn command uses a subshell with job control enabled:

```bash
(set -m; nohup bash -c 'WRAPPER_SCRIPT' > stdout.log 2> stderr.log < /dev/null & echo $!)
```

**Key elements:**
- `set -m` — Enables bash job control, which makes backgrounded
processes become their own process group leader (PID === PGID). This is
a bash builtin available on all platforms.
- `nohup` — Prevents SIGHUP from killing the process when the terminal
closes.
- Subshell `(...)` — Isolates the process group so the outer shell exits
immediately after echoing the PID.
- `< /dev/null` — Detaches stdin so the process doesn't block waiting
for input.

### Exit Code Detection

The wrapper script sets up a trap to capture the exit code:

```bash
trap 'echo $? > exit_code' EXIT && cd /path && export ENV=val && USER_SCRIPT
```

When the process exits (normally or via signal), the trap writes `$?` to
the `exit_code` file. Mux reads this file to determine if the process is
still running (`exit_code` doesn't exist) or has exited (file contains
the code).

### Process Group Termination

Because `set -m` ensures PID === PGID, we can kill the entire process
tree using a negative PID:

```bash
kill -15 -PID; sleep 2; if kill -0 -PID; then kill -9 -PID; echo 137 > exit_code; else echo 143 > exit_code; fi
```

**Sequence:**
1. Send SIGTERM (`-15`) to the process group (`-PID` targets the group)
2. Wait 2 seconds for graceful shutdown
3. Check if any process in the group survives (`kill -0 -PID`)
4. If still alive, send SIGKILL (`-9`) and record exit code 137
5. Otherwise, record exit code 143 (SIGTERM)

This ensures child processes spawned by the background job are also
terminated, preventing orphaned processes.

### Cross-Platform Compatibility

| Feature | Linux | macOS | Windows (MSYS2) |
|---------|-------|-------|-----------------|
| `set -m` | ✓ bash builtin | ✓ bash builtin | ✓ bash builtin |
| `kill -15/-9 -PID` | ✓ | ✓ | ✓ |
| `nohup` | ✓ | ✓ | ✓ |
| Path format | POSIX | POSIX | Converted via `cygpath` |

Using `set -m` instead of platform-specific tools like `setsid`
(Linux-only) ensures the same code works everywhere.

## Platform Support

- **Linux/macOS/Windows MSYS2**: set -m + nohup pattern (universal)
- **SSH**: Same pattern executed remotely

## Testing

- 20 unit tests in `backgroundProcessManager.test.ts` (including process
group termination)
- 7 unit tests for background execution in `bash.test.ts`
- 6 unit tests in `bash_background_list.test.ts` (including
display_name)
- 5 unit tests in `bash_background_terminate.test.ts`
- 19 unit tests in `backgroundCommands.test.ts`
- 7 runtime tests in `tests/runtime/runtime.test.ts` (Local & SSH)
- 3 end-to-end integration tests in `tests/ipc/backgroundBash.test.ts`
(real AI calls)

_Generated with mux_

Author: ethanndickson

bun.lock

@@ -1,5 +1,6 @@
 {
   "lockfileVersion": 1,
+  "configVersion": 0,
   "workspaces": {
     "": {
       "name": "mux",

src/browser/components/RightSidebar/CodeReview/ReviewPanel.tsx

@@ -266,7 +266,8 @@ export const ReviewPanel: React.FC<ReviewPanelProps> = ({
         }
 
         const diffOutput = diffResult.data.output ?? "";
-        const truncationInfo = diffResult.data.truncated;
+        const truncationInfo =
+          "truncated" in diffResult.data ? diffResult.data.truncated : undefined;
 
         const fileDiffs = parseDiff(diffOutput);
         const allHunks = extractAllHunks(fileDiffs);

src/cli/run.ts

@@ -17,6 +17,7 @@ import { PartialService } from "@/node/services/partialService";
 import { InitStateManager } from "@/node/services/initStateManager";
 import { AIService } from "@/node/services/aiService";
 import { AgentSession, type AgentSessionChatEvent } from "@/node/services/agentSession";
+import { BackgroundProcessManager } from "@/node/services/backgroundProcessManager";
 import {
   isCaughtUpMessage,
   isStreamAbort,
@@ -267,7 +268,14 @@ async function main(): Promise<void> {
   const historyService = new HistoryService(config);
   const partialService = new PartialService(config, historyService);
   const initStateManager = new InitStateManager(config);
-  const aiService = new AIService(config, historyService, partialService, initStateManager);
+  const backgroundProcessManager = new BackgroundProcessManager();
+  const aiService = new AIService(
+    config,
+    historyService,
+    partialService,
+    initStateManager,
+    backgroundProcessManager
+  );
   ensureProvidersConfig(config);
 
   const session = new AgentSession({
@@ -277,6 +285,7 @@ async function main(): Promise<void> {
     partialService,
     aiService,
     initStateManager,
+    backgroundProcessManager,
   });
 
   await session.ensureMetadata({

src/common/orpc/schemas/runtime.ts

@@ -12,24 +12,33 @@ export const RuntimeModeSchema = z.enum(["local", "worktree", "ssh"]);
  *
  * This allows two-way compatibility: users can upgrade/downgrade without breaking workspaces.
  */
+// Common field for background process output directory
+const bgOutputDirField = z
+  .string()
+  .optional()
+  .meta({ description: "Directory for background process output (e.g., /tmp/mux-bashes)" });
+
 export const RuntimeConfigSchema = z.union([
   // Legacy local with srcBaseDir (treated as worktree)
   z.object({
     type: z.literal("local"),
     srcBaseDir: z.string().meta({
       description: "Base directory where all workspaces are stored (legacy worktree config)",
     }),
+    bgOutputDir: bgOutputDirField,
   }),
   // New project-dir local (no srcBaseDir)
   z.object({
     type: z.literal("local"),
+    bgOutputDir: bgOutputDirField,
   }),
   // Explicit worktree runtime
   z.object({
     type: z.literal("worktree"),
     srcBaseDir: z
       .string()
       .meta({ description: "Base directory where all workspaces are stored (e.g., ~/.mux/src)" }),
+    bgOutputDir: bgOutputDirField,
   }),
   // SSH runtime
   z.object({
@@ -40,6 +49,7 @@ export const RuntimeConfigSchema = z.union([
     srcBaseDir: z
       .string()
       .meta({ description: "Base directory on remote host where all workspaces are stored" }),
+    bgOutputDir: bgOutputDirField,
     identityFile: z
       .string()
       .optional()

src/common/types/tools.ts

@@ -7,6 +7,8 @@
 export interface BashToolArgs {
   script: string;
   timeout_secs?: number; // Optional: defaults to 3 seconds for interactivity
+  run_in_background?: boolean; // Run without blocking (for long-running processes)
+  display_name?: string; // Human-readable name for background processes
 }
 
 interface CommonBashFields {
@@ -26,6 +28,14 @@ export type BashToolResult =
         totalLines: number;
       };
     })
+  | (CommonBashFields & {
+      success: true;
+      output: string;
+      exitCode: 0;
+      backgroundProcessId: string; // Background spawn succeeded
+      stdout_path: string; // Path to stdout log file
+      stderr_path: string; // Path to stderr log file
+    })
   | (CommonBashFields & {
       success: false;
       output?: string;
@@ -190,6 +200,33 @@ export interface StatusSetToolArgs {
   url?: string;
 }
 
+// Bash Background Tool Types
+export interface BashBackgroundTerminateArgs {
+  process_id: string;
+}
+
+export type BashBackgroundTerminateResult =
+  | { success: true; message: string; display_name?: string }
+  | { success: false; error: string };
+
+// Bash Background List Tool Types
+export type BashBackgroundListArgs = Record<string, never>;
+
+export interface BashBackgroundListProcess {
+  process_id: string;
+  status: "running" | "exited" | "killed" | "failed";
+  script: string;
+  uptime_ms: number;
+  exitCode?: number;
+  stdout_path: string; // Path to stdout log file
+  stderr_path: string; // Path to stderr log file
+  display_name?: string; // Human-readable name (e.g., "Dev Server")
+}
+
+export type BashBackgroundListResult =
+  | { success: true; processes: BashBackgroundListProcess[] }
+  | { success: false; error: string };
+
 export type StatusSetToolResult =
   | {
       success: true;

src/common/utils/tools/toolDefinitions.ts

@@ -52,6 +52,26 @@ export const TOOL_DEFINITIONS = {
         .describe(
           `Timeout (seconds, default: ${BASH_DEFAULT_TIMEOUT_SECS}). Start small and increase on retry; avoid large initial values to keep UX responsive`
         ),
+      run_in_background: z
+        .boolean()
+        .default(false)
+        .describe(
+          "Run this command in the background without blocking. " +
+            "Use for processes running >5s (dev servers, builds, file watchers). " +
+            "Do NOT use for quick commands (<5s), interactive processes (no stdin support), " +
+            "or processes requiring real-time output (use foreground with larger timeout instead). " +
+            "Returns immediately with process_id (e.g., bg-a1b2c3d4), stdout_path, and stderr_path. " +
+            "Read output with bash (e.g., tail -50 <stdout_path>). " +
+            "Terminate with bash_background_terminate using the process_id. " +
+            "Process persists until terminated or workspace is removed."
+        ),
+      display_name: z
+        .string()
+        .optional()
+        .describe(
+          "Human-readable name for background processes (e.g., 'Dev Server', 'TypeCheck Watch'). " +
+            "Only used when run_in_background=true."
+        ),
     }),
   },
   file_read: {
@@ -229,6 +249,26 @@ export const TOOL_DEFINITIONS = {
       })
       .strict(),
   },
+  bash_background_list: {
+    description:
+      "List all background processes started with bash(run_in_background=true). " +
+      "Returns process_id, status, script, stdout_path, stderr_path for each process. " +
+      "Use to find process_id for termination or check output file paths.",
+    schema: z.object({}),
+  },
+  bash_background_terminate: {
+    description:
+      "Terminate a background process started with bash(run_in_background=true). " +
+      "Use process_id from the original bash response or from bash_background_list. " +
+      "Sends SIGTERM, waits briefly, then SIGKILL if needed. " +
+      "Output files remain available after termination.",
+    schema: z.object({
+      process_id: z
+        .string()
+        .regex(/^bg-[0-9a-f]{8}$/, "Invalid process ID format")
+        .describe("Background process ID to terminate"),
+    }),
+  },
   web_fetch: {
     description:
       `Fetch a web page and extract its main content as clean markdown. ` +
@@ -272,6 +312,8 @@ export function getAvailableTools(modelString: string): string[] {
   // Base tools available for all models
   const baseTools = [
     "bash",
+    "bash_background_list",
+    "bash_background_terminate",
     "file_read",
     "file_edit_replace_string",
     // "file_edit_replace_lines", // DISABLED: causes models to break repo state

src/common/utils/tools/tools.ts

@@ -1,6 +1,8 @@
 import { type Tool } from "ai";
 import { createFileReadTool } from "@/node/services/tools/file_read";
 import { createBashTool } from "@/node/services/tools/bash";
+import { createBashBackgroundListTool } from "@/node/services/tools/bash_background_list";
+import { createBashBackgroundTerminateTool } from "@/node/services/tools/bash_background_terminate";
 import { createFileEditReplaceStringTool } from "@/node/services/tools/file_edit_replace_string";
 // DISABLED: import { createFileEditReplaceLinesTool } from "@/node/services/tools/file_edit_replace_lines";
 import { createFileEditInsertTool } from "@/node/services/tools/file_edit_insert";
@@ -12,6 +14,7 @@ import { log } from "@/node/services/log";
 
 import type { Runtime } from "@/node/runtime/Runtime";
 import type { InitStateManager } from "@/node/services/initStateManager";
+import type { BackgroundProcessManager } from "@/node/services/backgroundProcessManager";
 
 /**
  * Configuration for tools that need runtime context
@@ -31,6 +34,10 @@ export interface ToolConfiguration {
   runtimeTempDir: string;
   /** Overflow policy for bash tool output (optional, not exposed to AI) */
   overflow_policy?: "truncate" | "tmpfile";
+  /** Background process manager for bash tool (optional, AI-only) */
+  backgroundProcessManager?: BackgroundProcessManager;
+  /** Workspace ID for tracking background processes (optional for token estimation) */
+  workspaceId?: string;
 }
 
 /**
@@ -101,6 +108,8 @@ export async function getToolsForModel(
     // and line number miscalculations. Use file_edit_replace_string instead.
     // file_edit_replace_lines: wrap(createFileEditReplaceLinesTool(config)),
     bash: wrap(createBashTool(config)),
+    bash_background_list: wrap(createBashBackgroundListTool(config)),
+    bash_background_terminate: wrap(createBashBackgroundTerminateTool(config)),
     web_fetch: wrap(createWebFetchTool(config)),
   };
 

src/desktop/main.ts

@@ -564,6 +564,30 @@ if (gotTheLock) {
     }
   });
 
+  // Track if we're in the middle of disposing to prevent re-entry
+  let isDisposing = false;
+
+  app.on("before-quit", (event) => {
+    // Skip if already disposing or no services to clean up
+    if (isDisposing || !services) {
+      return;
+    }
+
+    // Prevent quit, clean up, then quit again
+    event.preventDefault();
+    isDisposing = true;
+
+    // Race dispose against timeout to ensure app quits even if disposal hangs
+    const disposePromise = services.dispose().catch((err) => {
+      console.error("Error during ServiceContainer dispose:", err);
+    });
+    const timeoutPromise = new Promise<void>((resolve) => setTimeout(resolve, 5000));
+
+    void Promise.race([disposePromise, timeoutPromise]).finally(() => {
+      app.quit();
+    });
+  });
+
   app.on("window-all-closed", () => {
     if (process.platform !== "darwin") {
       app.quit();

src/node/runtime/LocalBackgroundHandle.ts

@@ -0,0 +1,81 @@
+import type { BackgroundHandle } from "./Runtime";
+import { parseExitCode, buildTerminateCommand } from "./backgroundCommands";
+import { log } from "@/node/services/log";
+import { execAsync } from "@/node/utils/disposableExec";
+import { getBashPath } from "@/node/utils/main/bashPath";
+import * as fs from "fs/promises";
+import * as path from "path";
+
+/**
+ * Handle to a local background process.
+ *
+ * Uses file-based status detection (same approach as SSHBackgroundHandle):
+ * - Process is running if exit_code file doesn't exist
+ * - Exit code is read from exit_code file (written by bash trap on exit)
+ *
+ * Output is written directly to files via shell redirection (nohup ... > file),
+ * so the process continues writing even if mux closes.
+ */
+export class LocalBackgroundHandle implements BackgroundHandle {
+  private terminated = false;
+
+  constructor(
+    private readonly pid: number,
+    public readonly outputDir: string
+  ) {}
+
+  /**
+   * Get the exit code from the exit_code file.
+   * Returns null if process is still running (file doesn't exist yet).
+   */
+  async getExitCode(): Promise<number | null> {
+    try {
+      const exitCodePath = path.join(this.outputDir, "exit_code");
+      const content = await fs.readFile(exitCodePath, "utf-8");
+      return parseExitCode(content);
+    } catch {
+      // File doesn't exist or can't be read - process still running or crashed
+      return null;
+    }
+  }
+
+  /**
+   * Terminate the process by killing the process group.
+   * Sends SIGTERM (15), waits 2 seconds, then SIGKILL (9) if still running.
+   *
+   * Uses buildTerminateCommand for parity with SSH - works on Linux, macOS, and Windows MSYS2.
+   */
+  async terminate(): Promise<void> {
+    if (this.terminated) return;
+
+    try {
+      const exitCodePath = path.join(this.outputDir, "exit_code");
+      const terminateCmd = buildTerminateCommand(this.pid, exitCodePath);
+      log.debug(`LocalBackgroundHandle: Terminating process group ${this.pid}`);
+      using proc = execAsync(terminateCmd, { shell: getBashPath() });
+      await proc.result;
+    } catch (error) {
+      // Process may already be dead - that's fine
+      log.debug(
+        `LocalBackgroundHandle.terminate: Error: ${error instanceof Error ? error.message : String(error)}`
+      );
+    }
+
+    this.terminated = true;
+  }
+
+  /**
+   * Clean up resources.
+   * No local resources to clean - process runs independently via nohup.
+   */
+  async dispose(): Promise<void> {
+    // No resources to clean up - we don't own the process
+  }
+
+  /**
+   * Write meta.json to the output directory.
+   */
+  async writeMeta(metaJson: string): Promise<void> {
+    await fs.writeFile(path.join(this.outputDir, "meta.json"), metaJson);
+  }
+}