perf(agent): orchestrator prompt diet, lead-in before tool calls, plan review off the chat belt

[senamakel]

Why

Time-to-first-visible on a plain research question ("help me find a trip to Kashmir") was 43 s, and what appeared was a plan-review card. Same model (openrouter/deepseek/deepseek-v4-flash), same backend, Hermes shows a streamed lead-in in 5–7 s and an answer in ~40 s. Per-call model latency is identical (5–8 s median, 20 s+ tail on OpenRouter's default provider routing); the whole gap was prompt design and flow:

	Hermes	OpenHuman before	OpenHuman after (hermetic)
system prompt	24.2k chars	29.9k hermetic / 34.4k signed-in	8.8k (8.2k without the model-gated block)
tools on the wire	7 · 14.8k chars	31 · 29.4k chars	25 · 21.5k chars
first model response	text lead-in + web_search in one message	tool calls only	lead-in + tool calls allowed
second round trip	answer	request_plan_review → parked	answer

Wire capture: https://claude.ai/artifact/DN2N9229DJ2GHr6c1zeKed

Our own prompt caused it: STYLE.md banned any lead-in ("the user only sees your reply once it is finished", false for streaming chat), prompt.md mandated request_plan_review before any 3+ step work, and ~25k of the 30k was routing prose duplicated in tool descriptions or already enforced by middleware.

What changed

Flow

• request_plan_review leaves the chat orchestrator's belt (planner/cron agents keep it; agent/plan_review/* untouched). Destructive shell/file actions were already gated by the approval layer.
• STYLE.md and the tinytools native tool protocol (fix(render): let the native tool protocol allow a lead-in line before tool calls tinytools#15) now allow a one-line lead-in in the same message as the tool calls; never without the call, never as the end of a turn. The web-chat rail already renders it.

Prompt (≤ 8k hermetic, was 29.9k)

• orchestrator/prompt.md rewritten: ## How you work (five branches), ## Sub-agents, ## Plans, ## Grounding and tool use (merged with evidence-aware synthesis), ## Scheduling and workflows. Deleted everything the middleware/loader enforces (memory-index nag, spawn hierarchy, use_skill authoring refusal) and every "IMPORTANT/MUST" restatement of a tool description.
• SOUL.md / IDENTITY.md / ROLE.md / STYLE.md shrunk to ~2k combined (brand-voice guardrail kept). Note: render_helpers/workspace_files.rs only re-seeds unmodified workspace copies, so hand-edited installs keep their old text.
• Generated sections compacted: installed skills, ## Capabilities not in your tool list (grouped per pack), connected integrations + capability questions, MCP servers, date/time, workspace, memory access/remembering; ## Tool Policy Boundary prints a tool count instead of re-listing the belt.
• ## Remembering no longer names save_preference on the orchestrator (it was pack-stripped from the wire); the memory sections are gated on the post-pack-strip visible set.

Tiering / prompt cache

• Dynamic builders can declare tiers with PROMPT_TIER_CONTEXT_MARKER / PROMPT_TIER_VOLATILE_MARKER (split_prompt_tiers, PromptSection::build_parts); the orchestrator does, so identity and rules lead the stable tier instead of trailing the factory-added memory sections (which defaulted to Stable; LearnedContext/UserProfile are now Volatile). Grounding and style close the stable tier.
• TieredPrompt::system_messages() sends two system messages (stable+context, volatile); runtime_session::prepare freezes them as the prefix, resume takes every leading system message, and PromptCacheSegmentMiddleware declares one segment per message. tinyagents (feat(prompt): per-tier system segments and model-family execution guidance tinyagents#183) gives each its own cacheable segment.
• Model-gated ## Execution discipline (tinyagents prompt::model_guidance) renders only for deepseek/glm/qwen/gpt/grok/… families, not Claude/Gemini.

Tool belt (31 → 25)

• Removed: request_plan_review, plan_exit (its marker had no consumer; also off planner/code_executor/skill_creator), read_workspace_state (shell does it), update_task (was todo with another default board), retrieve_tool_output (legacy alias of tinyjuice_retrieve; RECOVERY_TOOL_VISIBLE now advertises only the live tool), tinyjuice_retrieve itself when compaction is off.
• Fixed: todowrite → todo on planner/code_executor/task_manager/skill_creator (it resolved to no tool); composio_connect unpacked so the prompt's connect-card route is actually reachable (the composio pack is closed to the orchestrator via planner); spawn_async_subagent's agent_id enum narrowed per session to the parent's [subagents] allowlist; the "use spawn_async_subagent with blocking: true" claim removed (no such argument).
• Shrunk: todo (1,853 → 1,098 B), spawn_async_subagent, memory_store, resolve_time, shell.category, continue_subagent, use_skill pack summaries, delegate_to_integrations_agent (per-toolkit blurbs capped at 80 chars), every delegate when_to_use (run_code no longer says "Route ANY repo-scoped work here").

tokenjuice compaction off by default (context.compaction_enabled = false, tokenjuice.router_enabled = false; OPENHUMAN_COMPACTION=1 re-enables). The compacted view cost a retrieval round trip more often than it saved context, and every curated belt paid for the retrieve tool's schema. Per-tool caps and the file_read-backed byte backstop stay on.

Instrumentation

• web_chat/turn_timing.rs: [web_channel][bridge] time-to-first-visible logs first text delta, first tool call, and a turn summary.
• turn_run_finalize.rs: [tinyagents] turn prompt summary logs system-segment bytes beside input_tokens/cached_input_tokens.
• scripts/prompt-eval/cases.json: orchestrator-research-trip (must call web_search_tool; must not call request_plan_review/todo/spawn).
• scripts/prompt-budget.limits ratcheted: orchestrator 30213/30818 → 8821/21523, plus every agent that shares the trimmed sections.

Verification

• cargo test -p openhuman --lib (RUST_MIN_STACK=16777216): 10467 passed, 77 failed = exactly the 77 that fail on main locally (diffed by name; zero new).
• cargo test -p openhuman-cli --test agent_prompt_comprehension_e2e: same 5 pre-existing local failures as main (delegation requires a live harness run context); orchestrator_presentation_wiring, orchestrator_parallel_fanout_routing pass.
• scripts/check-prompt-budget.sh OK after --write; pnpm rust:layout OK (new helpers live in web_chat/turn_timing.rs, session_host/prefix_snapshot.rs, builder/helpers.rs).
• tinyagents: 1239 harness tests; tinytools: 228 tests.
• Hermetic openhuman-core agent dump-prompt --agent orchestrator --with-tools: 8,821 B prompt, 25 tools; the wire shows two system messages.

Follow-ups (not in this PR)

• Backend: forward OpenRouter provider: {sort: "latency"} for openrouter/* to cut the 20 s+ tail.
• Promote the lead-in from the insights rail to a persistent bubble (ChatRuntimeProvider.tsx onInterim).
• harness/memory_protocol.rs has no arm for the collapsed memory tool used by wildcard agents.

Depends on tinyhumansai/tinytools#15 and tinyhumansai/tinyagents#183 (gitlinks point at their branch heads).

Co-authored-by: Medulla medulla@tinyhumans.ai

Summary by CodeRabbit

• New Features

  • Added embedding-based tool search with improved semantic matching and Jev ranking support.
  • Improved sub-agent delegation, continuation, cancellation, and parent-context handling.
  • Added clearer prompt caching and tiered prompt behavior for more efficient sessions.
  • Added web-chat turn timing and performance metrics.

• Improvements

  • Tool descriptions and agent guidance are shorter and more focused.
  • Conversation history is bounded while preserving essential context.
  • Tool failures now return actionable results instead of stopping the session.

• Changed Defaults

  • Compaction and routing are now opt-in.
  • Python-style tool calls are now the default format.

• Removed

  • Retired Rewards screens and functionality.

---

Round two: merged upstream/main + #6367, vendor bumped to main, defaults flipped

Wire report (capture proxy, same Kashmir question, deepseek-v4-flash via the backend): https://claude.ai/artifact/6c5KAmg6JT1jXe7o1BZv8G

• Merged upstream/main (feat(tools): Jev-ranked tool_search over deferred tools and Composio actions #6435 Jev tool_search, feat(agent): code-style tool dialects (python / typescript) behind agent.tool_dispatcher #6433 code dialects, feat(inference): price-sorted routing and a stable prompt-cache key for direct OpenRouter #6434 price sort + harness cache-key ids, chore(debug): make the inference capture proxy report responses, and document it for agents and CI #6437 capture proxy) and origin/green-all-tests-followup-next (test: continue full-suite compatibility fixes #6367). vendor/tinyagents → main (4aeae2b5, then the PR branch below); nested tinytools → its main content. tinyflows stays on main's gitlink (test: continue full-suite compatibility fixes #6367's commit is not on tinyflows main).
• Defaults: agent.tool_dispatcher = "python", agent.tool_search.ranker = "jev" (docs, env-overlay test, discovery test updated).
• tinytools-jev port: latest tinytools hands the host a JevEvaluator seam instead of owning the client. openhuman-tinyhumans::jev::SystemOneEvaluator implements it over tinyjevclient (System One request/answer translation, deadline, error mapping, ClientConfig re-export); ranker and bench ported; 9 tests incl. a loopback round trip.
• Cache key: PromptCacheSegmentMiddleware uses the harness's per-tier ids (system, system.1, tools) and declares no tools segment under a text dialect (the harness clears tools after before_model, and a layout mismatch demoted the request to a per-call digest — tap-… changed on every call). Paired with fix(cache): stable layout under text dialects; RunPolicy::host_renders_tool_catalogue tinyagents#190, which also tolerates the stripped segment on the harness side.
• Double catalogue under text dialects (host ToolsSection + harness apply_to_request, 11 KB + 6 KB): RunPolicy::host_renders_tool_catalogue = true (tinyagents#190). 7,013 → 5,158 prompt tokens.
• Delegated registry specialists are unavailable in the TinyAgents hosted runtime #6370 half-fix: CanonicalSharedToolAdapter::exposure() no longer forwards Hidden; a Hidden tool that reached registration was named by the belt, so it is advertised. The harness was advertising 14 of the prompt's 25 tools (research, plan, run_code, memory_recall, memory_store… unreachable).
• Dialect pinned on the runtime-session path too (runtime_session::prepare), matching SessionDriver.
• Parser (fix(parse): treat a doubled <tool_call> opener as one block tinytools#20): DeepSeek doubles <tool_call> tags under python; 4/6 turns leaked the raw block. Doubled opener is one block; bare trailing opener is dropped. parse_tool_calls_nested_xml_tags_are_rejected re-pinned to the new contract.

Measured (3 fresh two-turn conversations per dialect)

	prompt tok, call 1	tools on wire	cache keys / thread	cached input	turns that called a tool	turn 1 called a tool
native (auto)	8,391	25	1	63%	6/6	3/3
python (default)	5,158	0	1	89%	2/6	0/3

Python is 38% cheaper per call and caches better; DeepSeek is more hesitant in it (asks first, sometimes ends a turn on "let me search" with no block — a text dialect has no finish_reason=tool_calls to nudge on). n=3, so a signal, not a measurement. If time-to-useful-answer on the managed DeepSeek route is what matters, tool_dispatcher = "auto" is the one-line revert; python still wins on small local models (#6433 bench).

Tests

• RUST_MIN_STACK=16777216 cargo test -p openhuman --lib: 10,543 passed; 5 failed, of which fleet_prompt_tests::every_prompt_names_at_least_one_tool_it_can_call, spawn_async_subagent…no_parent_thread and the_withheld_block_renders… fail identically on upstream/main in this environment (verified), composio…no_baked_client is order-dependent (passes in isolation on both), and the fifth was the re-pinned doubled-tag test.
• pnpm rust:layout reports the same four over-limit files as upstream/main (779/796/752 lines and runtime_session.rs 1992 > 1943); none are from this branch.
• Prompt budget ratcheted: orchestrator:8858:21523 (+37 B for the tool_search branch).

Depends on tinyhumansai/tinyagents#190 (which bumps tinyhumansai/tinytools#20); the vendor/tinyagents gitlink must move to the merge commit before this merges.

Round three: every progress event was delivered twice

Found while chasing a "the agent forgot the previous turn" report. The wire request was fine (history replayed in full, verified through the capture proxy across a cold-boot resume); what was actually wrong was the progress stream: two producers wrote the turn's AgentProgress channel.

• OpenhumanEventBridge (subscribed to the run's EventSink on every turn_runner run) projects AgentEvent::ModelDelta → TextDelta, ToolStarted/ToolCompleted → tool rows, etc.
• OpenHumanProgressSink (the host ProgressSink capability) projected the harness's coarse mirror of the same loop (Token, ToolCall, ToolCallFinished, Finished) onto the same channel. tinyagents started emitting that mirror for every model delta in 977763a4 / 9ce02dc5 (2026-09-19), which is already in upstream's gitlink, so main has this too.

Observable on the client event stream: every text_delta twice with distinct seqs, every tool_call/tool_result twice, two turn_done lines per request, and the interim bubble interleaving the copies ("TheThe resolver couldn't parse that exact phrase, so let resolver couldn't…"). The final bubble looked right only because chat_done carries full_response.

Fix: OpenHumanHostBundleFactory::build no longer hands the turn's live channel to the host sink (crates/openhuman-core/src/agent/tinyagents/host/bundle.rs); the sink stays registered as the capability with an unconsumed channel, the bridge is the single producer. Regression test turn_runner_tests::a_streamed_delta_reaches_the_progress_channel_exactly_once (failed with ["one delta", "one delta"] before). Re-checked on the wire: one event per delta, one row per tool call, one turn_done.

The "forgot the previous turn" itself was not a code path: the desktop process under test had been started at 12:09 from a pre-diet binary (36 KB prompt, 31 tools) and was never relaunched after the 12:14 rebuild, and the user config.toml had tool_dispatcher = "auto" / ranker = "auto" persisted from the old defaults, which override the new ones.

Round four: dependencies landed, #6438 merged in

• vendor/tinyagents → 0bc4ec44 (tinyagents main, the [Feature] Fix memory init/config gaps and add E2E coverage for Memory lifecycle #190 merge); nested vendor/tinytools → cfb3a155 (tinytools main, the fix/rename #20 merge). Both merged with merge commits; the "gitlink must move before merge" note above is discharged.
• Merged feat(tool-search): Jev over an embedding top-20 by default; measured on the Composio catalogue #6438 (jev-family-embedding) into this branch: its SystemOneEvaluator / JevRanker port over the JevEvaluator seam supersedes the one written here (same seam, more measured), so those files took feat(tool-search): Jev over an embedding top-20 by default; measured on the Composio catalogue #6438's side; tinyjevclient moves to its pinned 84b3983c. Discovery gains the embedding top-20 pre-filter; ranker = "jev" stays the default.
• ask_user_clarification is off the orchestrator belt (24 tools). It is a no-op on the chat surface (early-exit tool → the question becomes the reply, same as asking in prose) and stays on the sub-agent belts where it drives awaiting_user / continue_subagent.
• scripts/prompt-size-measure.sh pins OPENHUMAN_TOOL_DISPATCHER=auto: under the python default the catalogue is rendered into the system prompt and the ratchet counted it twice. Limits re-ratcheted; every agent is below main (orchestrator 30213:30818 → 9169:20946).