feat(tools): Jev-ranked tool_search over deferred tools and Composio actions - #6435
Merged
Merged
Conversation
Bump the vendored tinyagents dependency to commit ae48b2c, incorporating upstream fixes and improvements. Auto-committed-on: macbook
The change consolidates the search logic by moving the BM25 implementation into a shared utility module, removing duplicated code from the skills and tools modules. This simplifies the codebase and ensures consistent search behavior across different components. Auto-committed-on: macbook
Added a README for the search utility module to explain its purpose and usage, making it easier for developers to understand and maintain the codebase. Auto-committed-on: macbook
This change adds a README file to the util module, documenting its purpose and usage. The file provides guidance for developers working with the utility functions in this crate. Auto-committed-on: macbook
Added README files for the tools module and its implementation directory to explain their purpose and structure. Auto-committed-on: macbook
The tool call state is now restored when a tool execution fails, ensuring that subsequent tool calls in the same turn are not incorrectly blocked. Previously, a failed tool call left the state marked as in-progress, which prevented any further tool calls from being processed. Auto-committed-on: macbook
Removed unused imports from the tools module to clean up the codebase and avoid compiler warnings. Auto-committed-on: macbook
The agent configuration schema previously declared several fields with incorrect types, causing validation failures when users provided valid values. This change updates the schema to match the actual data structures used by the agent runtime, ensuring that configuration files with the intended types are accepted without errors. Auto-committed-on: macbook
Introduce the tinyagents module with harness assembly and discovery capabilities, along with corresponding configuration schema updates. This establishes the foundational structure for managing tiny agent workflows within the core system. Auto-committed-on: macbook
The session host builder now applies the `tool_search` settings from its configuration during the build process, ensuring that every subsequent turn is treated as configured even when the turn harness is assembled without a config in hand. Auto-committed-on: macbook
Add a new `jev` feature gate to openhuman-tinyhumans that enables a Jev-backed `tool_search` ranker, using TypeSafe's Jev through the TinyHumans System One proxy. The feature is on by default and forwarded by the app and CLI crates, while the core falls back to BM25 ranking when disabled. Auto-committed-on: macbook
The redact_url_for_log function has moved from openhuman_core::api::config to openhuman_core::util::redact, so the ranker now calls the updated path to keep the log redaction working correctly. Auto-committed-on: macbook
The ranker now accepts a config loader, defaulting to the core's read path, so tests can supply a fixed config instead of relying on process-global state. This makes the no-credential test deterministic by removing its dependency on environment setup. Auto-committed-on: macbook
Add a `tool_ranker` option to `InstallOptions`, defaulting to true, that installs the Jev-backed ranker as the core's process-wide ranker when the `jev` feature is enabled. This replaces the default BM25 ranking for `tool_search` when permitted by the agent configuration, with the ranker resolving credentials per search so no login is required at install time. Auto-committed-on: macbook
Replace the let-chain condition with a filter call on the cached option, simplifying the code while preserving the same behavior of returning the cached ranker only when the fingerprint matches. Auto-committed-on: macbook
Add a test asserting that a wildcard belt withholds every `Deferred` registration from the wire while keeping it reachable: never advertised, always in the deferred set, and classified `Allow` by the policy. Also expose the deferred tool names set for test assertions. Auto-committed-on: macbook
Added a test-only accessor to expose the tool policy session for verification in unit tests, enabling assertions on its state without altering production behavior. Auto-committed-on: macbook
Use the fully qualified path for `ToolPolicySession` in the test-only accessor to avoid relying on a local import, making the type unambiguous and the code more robust to future import changes. Auto-committed-on: macbook
…lder/builder_tests_tool_exposur Auto-committed-on: macbook
The harness answers `tool_search` without running a tool, so no `ToolStarted`/`ToolCompleted` pair exists for it. Replay the same synthetic pair the live bridge emits, so a replayed trace has the `tool.tool_search` span with the ranking facts. Also emit the corresponding progress events and log lines in the live bridge, keeping the query text out of log lines as it is user content. Auto-committed-on: macbook
The orchestrator agent's named tool allowlist now includes `tool_search`, enabling on-demand tool discovery for rarely needed lookups. This addition ensures the deferred tool set remains reachable through the harness's intrinsic search mechanism, which ranks results by Jev with a TinyHumans credential or BM25 otherwise, and supports calling tools by name or via `tool_call` after discovery. Auto-committed-on: macbook
The orchestrator prompt now instructs the agent to call `tool_search` with the user's intent when a needed capability is not in the visible tool list, before delegating or declining. This reflects that the tool list is a core set, with many more registered tools discoverable via search, and clarifies that a single clear action on a connected service is a search-then-call rather than a delegation. The grounding section is updated to match, requiring a search attempt before claiming a capability does not exist. Auto-committed-on: macbook
…mputation The deferred tool name set now includes names from both durable and synthesized tools, ensuring per-action integration tools are kept off the wire and in the searchable set. A new `discovery_enabled` flag tracks whether the belt reaches deferred tools at all, and a shared recompute method keeps the deferred set consistent whenever either tool set changes. Auto-committed-on: macbook
…licy The runtime session snapshot now carries deferred tool names and a discovery flag, matching the session host's allowlist. This ensures the policy engine classifies deferred tools as reachable, so they remain callable through the tool_search bridge while not being advertised. Auto-committed-on: macbook
Add a `deferred` constructor that registers Composio actions as searchable, off-wire tools for parent sessions, resolving live config per call instead of anchoring to spawn-time config. This lets one clear action be a search and a call rather than a sub-agent run. Auto-committed-on: macbook
Register one deferred ComposioActionTool per connected integration action, sorted by toolkit and action name for stable ordering. This lets belts that opted into discovery reach actions through the harness's tool_search, avoiding an integrations_agent run, while non-opted belts never see them due to prompt-hiding and the direct-call gate. Gated actions are excluded, and slug collisions keep the first arrival. Auto-committed-on: macbook
When a wildcard belt is used, the durable `Hidden` members of the registry (such as collapsed `memory_*` and `todo_*` tools) were not being deferred, causing them to remain visible. This change ensures that for a wildcard belt, these hidden tools are stripped from the visible set and added to the deferred set, while named belts continue to exclude them entirely. The logic now correctly handles both durable and synthesised deferred tools, preserving the intended visibility behaviour. Auto-committed-on: macbook
…ndency The new `tool-search-bench` binary compares the `tool_search` rankers (bm25, overlap, and jev) over the real orchestrator registry and recorded Composio catalogues. The `jev` ranker is now compiled in by default via the `jev` feature, which also enables the optional `tinytools-jev` dependency, so the benchmark can measure it without requiring a Jev client at build time. Auto-committed-on: macbook
The Cargo.lock file has been updated to include the new `tinytools-jev` dependency, which is now required by the project. This change ensures that the lock file reflects the current dependency tree and maintains reproducibility of builds. Auto-committed-on: macbook
The token-overlap ranker used by the Composio sub-agent has been moved from the benchmark binary into the core discovery module, where it is now publicly exposed and callable. It also gains an explicit empty-intent guard that returns an error, and a test covering both its ranking behavior and kind name. Auto-committed-on: macbook
The `RankHit` type was no longer used in the benchmark code, so the import has been cleaned up to keep the codebase tidy. Auto-committed-on: macbook
Contributor
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (44)
Comment |
senamakel
marked this pull request as ready for review
September 22, 2026 02:46
Tiny Sweeper review
|
# Conflicts: # crates/openhuman-cli/Cargo.toml # crates/openhuman-core/src/agent/tinyagents/harness_assembly.rs # vendor/tinyagents
senamakel
added a commit
to senamakel/openhuman
that referenced
this pull request
Sep 22, 2026
Keeps the hermetic orchestrator prompt and adds the tool_search branch from tinyhumansai#6435; prompt_cache segments use the harness per-tier ids (system, system.1, tools) so the sticky prompt_cache_key from tinyhumansai#6434 stays constant across tiers. vendor/tinyagents -> main (4aeae2b5), nested tinytools -> main (aa811fe). Co-authored-by: Medulla <medulla@tinyhumans.ai>
senamakel
added a commit
to senamakel/openhuman
that referenced
this pull request
Sep 22, 2026
vendor/tinyagents is pinned at 0bc4ec44 (tinyagents main tip, PR tinyhumansai#190 merged), while the merge-base pin (eefef72b) was an ad hoc merge commit made while resolving PR tinyhumansai#6435 locally and was never pushed to tinyagents main. It sits on a sibling branch, so the monotonicity gate sees the two as diverged ("sideways") rather than a clean fast-forward. Diffing eefef72b against 0bc4ec44 confirms no work is lost: content unique to eefef72b (ToolRanker/BM25 discovery, the claude_code input builder, dialect docs) is present in 0bc4ec44 too, just reshaped by later commits on tinyagents main (net +1617/-151 lines across the submodule, almost entirely superseding rewrites of the same files). 0bc4ec44 is the correct pin to build against; keep it. Co-authored-by: Medulla <medulla@tinyhumans.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Jev-ranked, on-demand tool discovery for the orchestrator: the model keeps a core tool set on the wire, and everything else — rarely used lookups, and now every action of every connected Composio toolkit — is registered as
ToolExposure::Deferredand reached through the harness's intrinsictool_searchbridge, ranked by TypeSafe's Jev (BM25 retrieves top-20, one JevChoicedecides) when the process has a TinyHumans credential and by BM25 otherwise. One clear action on a connected service ("ping Alex on Slack that I'm late") becomes a search and a direct call instead of adelegate_to_integrations_agentsub-run.What changed
tool_search(tools/impl/meta/tool_search.rs,util/bm25.rs). It shadowed the harness's bridge, and a tool it found could not be called: the turn allowlist registered only advertised names.tools/impl/meta/deferred.rsnow only decides what is deferred.deferred_tool_namesbesidevisible_tool_names: the policy session classifiesvisible ∪ deferredasAllow(a deferred tool outside the union would beHideFromPrompt, which the direct-call gate refuses), the harness allowlist is the union, the prompt spec list stays the advertised set, andCanonicalSharedToolAdapterforwardsexposure()/family()so the harness advertises onlyDirectand indexes the rest. Both turn loops (session host and runtime session) are wired; the runtimeToolSnapshotcarries the deferred specs as the granted declaration set.[tools] namedbelt opts in by listingtool_search(not a registered tool — the builder strips it and treats it as the flag). The orchestrator opts in; every other named belt is unchanged and still cannot reach a deferred tool.collect_orchestrator_toolssynthesises oneComposioActionTool::deferred(toolkit, …)per action of each connected toolkit besidedelegate_to_integrations_agent; they follow the connected-integration reconcile and resolve the live config per call. Permission level,external_effectclassification and the approval gate apply per call exactly as on the sub-agent path.agent::tinyagents::discovery: process-wideinstall_tool_ranker,agent.tool_searchconfig (ranker: auto|bm25|jev|compare,top_k, defaultauto/ 3), and theToolDiscoveryPolicyevery turn harness runs with.OverlapRankerwraps the existingrank_tools_by_promptbehind the same trait for comparison.openhuman-tinyhumansjevfeature (default, forwarded by the app and CLI):TinyHumansJevRankerresolves the credential and backend URL per search (sign-in/out is honoured, client cached by fingerprint), andinstall()installs it. No credential →Backenderror → harness BM25 fallback, soautocosts nothing signed out.ToolSearchedis projected (live bridge + journal replay) as a synthetictool_searchstart/complete pair carryingranker,top_confidence,fallback,shadow_matched,latency_ms, so a Jev-vs-BM25 comparison can be read off traces;ToolsAdvertised/DeferredToolCallare logged.tool_searchfirst"; the grounding rule now allows what a search returns.cargo run -p openhuman-cli --bin tool-search-bench -- --ranker allranks the real orchestrator registry + the recorded Composio catalogues (≈1,000 actions, 9 toolkits) against a labelled intents fixture and reports top-1 / top-3 / recall@20 / needless rate / latency / tokens / USD per ranker.Still to do on this branch (why it is a draft)
tests/fixtures/tool_search/intents.jsonl(~150 labelled intents) and the bench rundocs/plans/jev-tool-search-baseline.md): prompt/schema bytes per agent, advertised vs registered, bench accuracy, and model calls / prompt tokens / sub-agent spawns on 10 scripted live prompts — onmainfirst, then here underjev,bm25andcomparescripts/prompt-budget.limitsratchet (the deferred tail must pay for the bridge; target a net orchestrator decrease)Deferredfamilies (Composio actions and the coreDeferredtools are in)x-sdk-nameon the Jev request (needs a smalltinyjevclientchange; every other backend request carries it)gitbooks/developing/architecture/agent-harness.md,platform/about_app/autodefault from the bench numbers (ships as Jev only if top-3 ≥ +10 pts over BM25 with the needless rate not worse)Depends on tinyhumansai/tinytools#17 and tinyhumansai/tinyagents#185 — the
vendor/tinyagentsgitlink points at that branch.Test plan
RUST_MIN_STACK=67108864 cargo test -p openhuman --lib: 10,456 passed; the 77 failures are the same 77 on untouchedmain(spawn/orchestration suites that need the mock backend) — diffed withcomm, zero newtool_search, named belt without it reaches nothing),tools::implementations::meta::deferred,agent::tinyagents::discovery(policy from settings, overlap ranker),openhuman-tinyhumans::jev(no credential → backend error, fingerprint)cargo clippy -p openhuman -p openhuman-cli -p openhuman-tinyhumans -- -D warningsclean (the--all-targetsform fails on pre-existing test files untouched here)cargo checkfor the core andcrates/openhuman-app(feature forwarded)openhuman-core servewith a TinyHumans key and Slack connected →tool.tool_searchspan (ranker=jev) then theSLACK_SEND_MESSAGEaction, nosubagent.*spanCo-authored-by: Medulla medulla@tinyhumans.ai
Summary by CodeRabbit