tycoon serve / ps / kill: user-registerable process manager for opted-in background tasks

[db-tycoon-stephen]

Problem

Tycoon has a process manager — tycoon start / tycoon stop use ServiceManager (src/tycoon/services/manager.py) with a PID file at .tycoon/run/pids.json, signal handling, port-readiness probing, and process-tree termination on stop. It's a solid foundation.

But it's hardcoded to four built-in services: tycoon, rill, dagster, nao (_SERVER_NAMES in src/tycoon/commands/start.py:16). End users can't register their own long-running tasks to be managed alongside these. If you want to background, say, a Streamlit app you wrote on top of your warehouse, or a custom FastAPI route, or a watch-mode dbt run — you're on your own with &, nohup, tmux, or pm2.

There are two more rough edges:

1. tycoon start blocks the terminal (shutdown.wait() at start.py:78). PIDs are persisted so tycoon stop works from a different shell, but you have to either keep a terminal pinned or background it yourself with &. No --daemon flag.
2. No introspection. No tycoon ps, no tycoon logs <name>, no "is rill actually still running or did it crash and the PID is stale." stdout/stderr go to DEVNULL (manager.py:57-58), so when a service dies silently there's nothing to read.

What end users actually need

The user-facing story we want:

# Register an arbitrary long-running task in tycoon.yml
tycoon serve register my-streamlit \
    --command \"streamlit run app.py --server.port 8501\" \
    --port 8501

# Run it in the background
tycoon serve start my-streamlit

# See what's running (built-ins + user-registered together)
tycoon ps
# NAME            STATUS    PID    PORT   UPTIME   COMMAND
# tycoon          running   12345  8000   0:14:22  uvicorn tycoon.server.app:app
# rill            running   12346  9009   0:14:22  rill start ./rill
# my-streamlit    running   12380  8501   0:02:01  streamlit run app.py ...

# Tail logs
tycoon serve logs my-streamlit

# Kill it
tycoon serve stop my-streamlit
# or:
tycoon kill my-streamlit       # alias, end-user friendly verb

Proposal

1. New services block in tycoon.yml

services:
  my-streamlit:
    command: \"streamlit run app.py --server.port 8501\"
    port: 8501
    cwd: ./streamlit_app          # optional, defaults to project root
    env:                          # optional
      MY_VAR: foo
    autostart: false              # if true, included in plain `tycoon start`

Built-in services (tycoon, rill, dagster, nao) keep their current registration in services/definitions.py — the YAML block is purely additive. Names collide with built-ins → clear error.

2. New commands

• tycoon serve register NAME --command \"...\" [--port N] [--cwd path] [--env K=V] — write the entry to tycoon.yml. Equivalent to hand-editing the YAML; just removes friction.
• tycoon serve unregister NAME — remove the entry.
• tycoon serve start NAME — start a single user-registered service, daemonized. (Built-ins still go through tycoon start --only NAME; or unify under tycoon serve start NAME and treat the four built-ins as just more entries.)
• tycoon serve stop NAME (alias tycoon kill NAME) — terminate. Reuse the existing process-tree kill logic from commands/stop.py:_kill_pid.
• tycoon serve logs NAME [--tail N] [--follow] — read log file. Logs land at .tycoon/run/logs/<name>.log instead of being piped to DEVNULL.
• tycoon ps — top-level command (not under serve), shows everything. Status column reflects "running / stopped / crashed (PID file stale)". Unlike today's silent crash, we detect dead PIDs.
• tycoon serve restart NAME — convenience wrapper.

3. Daemonization for tycoon start

Add --daemon / -d to the existing tycoon start. Without it, current blocking behavior is preserved (good for live demos and Ctrl-C). With it, fork to background, return shell prompt immediately. PID file already exists; just change the foreground/background story.

4. Crash detection in tycoon ps

When showing status: os.kill(pid, 0) to check if the process exists. If the PID file claims running but the PID is dead, status = crashed (stale). Optional: tycoon ps --prune to clean up stale entries.

5. Logs to disk, not DEVNULL

ServiceManager.start currently does stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL. Switch to .tycoon/run/logs/<name>.log opened in append mode. Rotate at some sane size (10 MB?) — basic, no need for logrotate-grade infrastructure. This is what makes tycoon serve logs possible.

Why this matters

The conference-talk pitch is "tycoon is the one tool for local-first analytics." When users build their own assets on top of tycoon's pipeline (a custom Streamlit app reading from the warehouse, a notebook server, a notification cron), they hit a UX cliff — tycoon manages its own four services flawlessly but their stuff is on its own. Closing that gap is the difference between "tycoon is a great demo runner" and "tycoon is the daily driver for my analytics stack."

The asks here are also concretely small: existing ServiceManager already does ~80% of the work. The missing pieces are (a) a YAML schema for user-registered services, (b) commands to read/write that schema, (c) logs-to-disk instead of DEVNULL, (d) a status view, and (e) --daemon mode.

Out of scope (this issue)

• Cross-host process management. Single machine only. If you need that, you need k8s or systemd; tycoon shouldn't compete.
• Auto-restart on crash. No supervisord-style respawn for v1. Crash detection in ps is enough; user re-runs serve start.
• Dependency graphs between services. "Wait for rill before starting my-thing" is not in scope. If your service genuinely needs another to be up, write that into your command (e.g. wait-for-it :9009 -- streamlit run ...).
• Per-service resource limits. Not in v1.
• Scheduled / cron-style jobs. Different problem; that's tycoon data sources run on a schedule, which is a Dagster concern, not a process-manager one.

Acceptance criteria

• Users can register arbitrary long-running services in tycoon.yml (programmatically via tycoon serve register or by hand-editing)
• tycoon serve start NAME runs the registered command, daemonized, with logs written to .tycoon/run/logs/<name>.log
• tycoon serve stop NAME / tycoon kill NAME kills the service and its full process tree (existing _kill_pid logic in commands/stop.py:82 already handles this — reuse it)
• tycoon ps shows built-ins and user services together; detects crashed-but-PID-file-stale state; column for uptime
• tycoon serve logs NAME [--tail N] [--follow] works
• tycoon start --daemon returns immediately instead of blocking
• Tests don't depend on real streamlit / rill binaries — use a fake long-running test command (e.g. sleep 60)
• Docs: new docs/commands/serve.md; docs/commands/start.md and docs/commands/stop.md updated to cross-link
• CHANGELOG + docs/releases/v<X>.md entry under whichever cycle this lands in

Cross-references

• src/tycoon/services/manager.py — existing ServiceManager, foundation for this work
• src/tycoon/services/definitions.py — where built-in services are registered; the user-services flow should slot into this same machinery
• src/tycoon/commands/start.py:16 — _SERVER_NAMES is the hardcode that needs to become dynamic
• src/tycoon/commands/stop.py:82 — _kill_pid already implements process-tree termination; reuse for user services
• .tycoon/run/pids.json — existing PID file location; extend to include user services

[db-tycoon-stephen]

Mirrored in Jira for roadmap tracking: PTC-12 (Epic: PTC-5 (AI / agent refinements)). Dev work continues here; the Jira issue exists so PRs/commits referencing PTC-12 show up on the Releases roadmap.