Add a task time-profiler: llm/tool/operator-wait gap analysis over transcripts

AGENTS.md

@@ -8,7 +8,7 @@ slice at a time (see ROADMAP "Definition of done — every slice").
 
 The control plane makes **no LLM calls**. All LLM calls happen **inside task containers**.
 
-- LLM-free packages: `core`, `taskservice`, `sessionservice`, `terminal`, `workflows`.
+- LLM-free packages: `core`, `taskservice`, `sessionservice`, `terminal`, `workflows`, `profiler`.
 - The **only** LLM-bearing package is `container/` (the agent runs there).
 
 If you add a package that orchestrates or renders, keep it LLM-free.
@@ -48,7 +48,22 @@ src/panopticon/
                    # description, unsent); daemon.py = the provision-only pull loop;
                    # host.py = the unified per-host daemon (spawn + provision each pass;
                    # `python -m panopticon.sessionservice.host`); `python -m panopticon.sessionservice`
-                   # spawns one task
+                   # spawns one task; transcripts.py = host-side read of a task's session
+                   # transcripts out of its per-task config volume, after the container is gone
+                   # (docker run + find/cat, injectable command-runner) — feeds `profiler/`
+  profiler/        # task time-profiler: pure gap-analysis over a claude session transcript's own
+                   # timestamps (no agent-side instrumentation) — categories.py = the tool-time
+                   # category table (one place, easily extended: Bash sub-classified by regex —
+                   # tests/vcs/deps/pty-verify —, Read/Write/Edit/Grep/Glob → code-nav, Task/Agent →
+                   # subagents [one bucket; its sidechain is never walked separately], mcp__* →
+                   # orchestration); parse.py = profile_transcripts(paths) -> a profile dict: merges
+                   # same-message.id assistant lines into one turn, classifies every gap as LLM time
+                   # (user→assistant), tool time (assistant-with-tool_use→its tool_result, by
+                   # category), or operator/system wait (assistant-with-no-tool_use→next user, incl.
+                   # AskUserQuestion spans and a same-file restart's abandoned-tool_result→
+                   # "interrupted, continue" gap) — between-session gaps (multiple transcript files)
+                   # reported separately; defensive throughout (malformed/old-format lines →
+                   # unattributed, never a crash)
   container/       # entrypoint (`python -m panopticon.container` = connect/register/slug/
                    # heartbeat liveness) + agent.py (`-m panopticon.container.agent` = the tmux
                    # pane's launcher: render skills + operations, point claude at the /mcp server,
@@ -219,6 +234,20 @@ on every PR (the same commands the Makefile wraps).
 - `tests/test_multi_workflow_acceptance.py` — Slice 8 acceptance: over REST (via `build_app`), a
   path-discovered workflow is selectable with no core change, and GithubPeerReviewed + the
   free-form (spike) workflow run concurrently with workflow-specific skills. No Docker, no LLM.
+- `tests/profiler/test_parse.py` — the golden spec for the **time-profiler**'s gap-analysis: a
+  hand-built fixture transcript exercises every category, an `AskUserQuestion` span (must land in
+  operator-wait, not a tool category), a `Task` subagent call (one bucket), a same-`message.id`
+  multi-line assistant turn, parallel tool calls, a mid-session restart (two consecutive `user`
+  lines, no assistant turn between — also operator-wait), a between-sessions restart, and
+  defensive old-format/malformed input (never crashes); `test_categories.py` covers the category
+  table's matcher rules in isolation.
+- `tests/sessionservice/test_transcripts.py` — unit tests pin the emitted `docker volume
+  inspect`/`docker run … find`/`cat` commands via a fake command-runner (incl. never issuing a
+  bare `docker run --volume <missing-name>`, which would silently create an empty volume); a
+  `skipif` no-docker integration test round-trips a real volume.
+- `tests/terminal/test_task_profile.py` — golden-output tests for the `panopticon profile` CLI's
+  formatting (single-task, `--all-tasks`, and the dashboard's one-line summary) plus
+  `run_profile_command`'s wiring (fake client + an injected session-path lookup, no real docker).
 
 ## Glossary
 

src/panopticon/profiler/__init__.py

@@ -0,0 +1,11 @@
+"""Pure, LLM-free transcript gap-analysis: where an agent's task time went.
+
+See :mod:`panopticon.profiler.parse` for the algorithm and :mod:`panopticon.profiler.categories`
+for the (single-place, easily-extended) tool category table.
+"""
+
+from __future__ import annotations
+
+from panopticon.profiler.parse import profile_transcripts
+
+__all__ = ["profile_transcripts"]

src/panopticon/profiler/categories.py

@@ -0,0 +1,80 @@
+"""The tool-time category table (single source of truth — extend it here, nowhere else).
+
+Classifies one tool call (name + its ``tool_use`` input) into a bucket. Ordered, first-match-wins:
+a ``Bash`` command is sub-classified by regex over its command string; every other tool is
+classified by name. Callers needing display order (CLI/dashboard) import :data:`DISPLAY_ORDER`.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Mapping
+from typing import Any
+
+#: Bash-command sub-categories, checked in order against the command string. First match wins;
+#: a command matching none of these falls through to ``other-tools``.
+_BASH_CATEGORY_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = (
+    ("tests", re.compile(r"\bpytest\b")),
+    ("vcs", re.compile(r"\b(git|gh)\b")),
+    ("deps", re.compile(r"\b(pip3?\s+install|uv\s+(?:add|sync)|uv\s+pip\s+install)\b")),
+    ("pty-verify", re.compile(r"\b(pexpect|pyte|verify[-_]skill)\b")),
+)
+
+#: Read/inspection tools that don't change repo state or wait on anything external.
+_CODE_NAV_TOOLS = frozenset({"Read", "Write", "Edit", "Grep", "Glob"})
+
+#: The subagent tool. Its tool_use→tool_result gap already spans the whole sidechain run (we never
+#: walk the sidechain transcript separately, so this is one bucket, not double-counted). Claude
+#: Code's built-in name is ``Task``; ``Agent`` covers harnesses (like this one) that rename it.
+_SUBAGENT_TOOLS = frozenset({"Task", "Agent"})
+
+#: A tool call whose PreToolUse/PostToolUse hooks flip the turn to the user while it's pending
+#: (see AGENTS.md's turn-flip contract) — the agent is blocked on a human answer, not doing tool
+#: work. Routed to operator-wait by the parser, not classified here as a tool category.
+ASK_USER_QUESTION_TOOL = "AskUserQuestion"
+
+#: Every category a tool call can land in, in the order the CLI/dashboard display them. ``llm`` and
+#: ``unattributed`` aren't tool categories (the parser computes them directly from gaps), but share
+#: this ordering so rendering stays in one place.
+DISPLAY_ORDER: tuple[str, ...] = (
+    "llm",
+    "tests",
+    "pty-verify",
+    "code-nav",
+    "vcs",
+    "deps",
+    "subagents",
+    "orchestration",
+    "other-tools",
+    "unattributed",
+)
+
+#: The tool-classified categories only (excludes ``llm``/``unattributed``) — what :func:`categorize`
+#: can return.
+TOOL_CATEGORIES: tuple[str, ...] = tuple(
+    c for c in DISPLAY_ORDER if c not in ("llm", "unattributed")
+)
+
+
+def categorize(tool_name: str, tool_input: Mapping[str, Any] | None) -> str:
+    """Classify one tool call into a category name (always one of :data:`TOOL_CATEGORIES`).
+
+    ``tool_name`` is matched defensively — an unrecognized or missing name falls through to
+    ``other-tools`` rather than raising, so an old-format or unfamiliar transcript is still
+    handled (never crashes the profiler)."""
+    if tool_name == "Bash":
+        command = (tool_input or {}).get("command")
+        if isinstance(command, str):
+            for category, pattern in _BASH_CATEGORY_PATTERNS:
+                if pattern.search(command):
+                    return category
+        return "other-tools"
+    if tool_name in _CODE_NAV_TOOLS:
+        return "code-nav"
+    if tool_name in _SUBAGENT_TOOLS:
+        return "subagents"
+    if isinstance(tool_name, str) and tool_name.startswith("mcp__"):
+        # The container's claude connects to the task service's MCP server and *only* it
+        # (`--strict-mcp-config`), so any `mcp__` call is by construction an orchestration call.
+        return "orchestration"
+    return "other-tools"