feat: add PubTator3 and Paperclip literature-evidence agents - #22
Open
Oguzhankokulu wants to merge 2 commits into
Open
feat: add PubTator3 and Paperclip literature-evidence agents#22Oguzhankokulu wants to merge 2 commits into
Oguzhankokulu wants to merge 2 commits into
Conversation
Two independent literature-evidence agents, each in its own package alongside
agent_tools/. Purely additive: no existing file is modified.
crossbar_llm/pubtator3_tools/ NCBI PubTator3 (autocomplete, relations,
search, BioC export)
crossbar_llm/paperclip_tools/ Paperclip full-text corpora, REST-primary
with an automatic MCP fallback
Each package holds its own client/adapter, LangGraph agent, prompts, schemas
and tests, and neither imports the other - verified by asserting that importing
one leaves the other absent from sys.modules. The two small provider-agnostic
helpers they both need (structured-output-with-JSON-fallback, and token usage
capture) are duplicated rather than shared, so a change to one agent cannot
break the other.
Both expose the same contract:
question (+ state) in -> {final_answer, citations, warnings, usage} out
where question_type == "out_of_scope" with final_answer None means the agent
declines, letting a caller distinguish that from an answer or a failure.
`build_graph` takes an already-built `chat_model`, which keeps the graph
independent of model configuration and is the seam the tests inject fakes at.
Each package's `llm.py` is the convenience path for callers, building that
model through agent_tools.llm_factory.
Nothing calls the agents yet. Wiring them into the API is a separate change.
New dependencies: aiolimiter>=1.2.1 and langchain-mcp-adapters>=0.3.2 at
runtime, pytest-asyncio>=1.4.0 and pytest-httpx>=0.36.2 for tests. Resolved
against this branch's lockfile on Python 3.12.11 with no change to existing
pins (langchain-core 1.5.0, langgraph 1.2.6, httpx 0.28.1, pydantic 2.13.4).
pytest -c crossbar_llm/pubtator3_tools/tests/pytest.ini crossbar_llm/pubtator3_tools/tests
pytest -c crossbar_llm/paperclip_tools/tests/pytest.ini crossbar_llm/paperclip_tools/tests
Note: Paperclip caps its full-text `map` step at 100 operations/day per API
key, shared across all users of a deployment.
There was a problem hiding this comment.
Pull request overview
This PR adds two new, self-contained literature-evidence agent packages under crossbar_llm/: one for NCBI PubTator3 (REST client + LangGraph agent + LangChain tools + offline tests/fixtures) and one for Paperclip (adapter-driven retrieval agent + robust wrappers + offline + optional live tests). The intent is to provide a consistent “question in → {final_answer, citations, warnings, usage} out” contract for future routing, without wiring either agent into the API yet.
Changes:
- Introduces
crossbar_llm/pubtator3_tools/with a typed async REST client, LangChain@toolwrappers, a LangGraph pipeline, prompts/schemas, usage capture, and extensive offline tests/fixtures. - Introduces
crossbar_llm/paperclip_tools/with never-raise adapter wrappers, a LangGraph pipeline with optional REST-only filter and map extraction, prompts/schemas, usage capture, and live-gated smoke tests. - Adds per-package pytest configuration and fixtures so each agent’s test suite can run independently of the repo’s root
pytest.ini.
Reviewed changes
Copilot reviewed 35 out of 37 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| crossbar_llm/pubtator3_tools/init.py | Package entrypoint docs for PubTator3 agent. |
| crossbar_llm/pubtator3_tools/agent.py | LangGraph orchestrator wiring router/search/export/synthesize/depth loop. |
| crossbar_llm/pubtator3_tools/client.py | Async PubTator3 REST client + Pydantic parsing/models. |
| crossbar_llm/pubtator3_tools/llm.py | Convenience chat-model builder via shared factory. |
| crossbar_llm/pubtator3_tools/nodes.py | Deterministic nodes for resolution/search/export plumbing. |
| crossbar_llm/pubtator3_tools/prompts.py | System prompts for router/synthesizer/depth evaluator. |
| crossbar_llm/pubtator3_tools/schemas.py | Router/evaluator schemas and shared state typing. |
| crossbar_llm/pubtator3_tools/structured_output.py | Structured-output helper with JSON fallback. |
| crossbar_llm/pubtator3_tools/tools.py | LangChain @tool wrappers around PubTator3 client endpoints. |
| crossbar_llm/pubtator3_tools/usage.py | Token usage aggregation + optional logging. |
| crossbar_llm/pubtator3_tools/tests/init.py | Test package marker. |
| crossbar_llm/pubtator3_tools/tests/conftest.py | Fixtures + pytest-asyncio mode enforcement for this suite. |
| crossbar_llm/pubtator3_tools/tests/pytest.ini | Scoped pytest config for PubTator3 suite. |
| crossbar_llm/pubtator3_tools/tests/test_models.py | Offline parsing/model round-trip tests with fixtures. |
| crossbar_llm/pubtator3_tools/tests/test_new_features.py | Tests for newer graph behaviors (abstracts_only/sections/refinement). |
| crossbar_llm/pubtator3_tools/tests/test_tools_split.py | Tool schema + wrapper behavior tests (httpx-mocked). |
| crossbar_llm/pubtator3_tools/tests/fixtures/pubtator3_autocomplete_example.json | Recorded autocomplete fixture. |
| crossbar_llm/pubtator3_tools/tests/fixtures/pubtator3_export_example.json | Recorded export fixture. |
| crossbar_llm/pubtator3_tools/tests/fixtures/pubtator3_search_example.json | Recorded search fixture. |
| crossbar_llm/paperclip_tools/init.py | Package entrypoint docs for Paperclip agent. |
| crossbar_llm/paperclip_tools/agent.py | LangGraph orchestrator for Paperclip retrieval/map/filter/sql paths. |
| crossbar_llm/paperclip_tools/llm.py | Convenience chat-model builder via shared factory. |
| crossbar_llm/paperclip_tools/nodes.py | Deterministic nodes for search/sql/filter/assembly + citation URL helpers. |
| crossbar_llm/paperclip_tools/prompts.py | System prompts for router/synthesizer/sql-synth/depth evaluator. |
| crossbar_llm/paperclip_tools/schemas.py | Router/depth schemas + state + citation/context models. |
| crossbar_llm/paperclip_tools/structured_output.py | Structured-output helper with JSON fallback. |
| crossbar_llm/paperclip_tools/tools.py | Never-raise adapter wrappers + SQL guardrails. |
| crossbar_llm/paperclip_tools/usage.py | Token usage aggregation + optional logging. |
| crossbar_llm/paperclip_tools/tests/init.py | Test package marker. |
| crossbar_llm/paperclip_tools/tests/conftest.py | Pytest-asyncio mode enforcement + marker registration. |
| crossbar_llm/paperclip_tools/tests/pytest.ini | Scoped pytest config for Paperclip suite. |
| crossbar_llm/paperclip_tools/tests/test_live.py | Live-gated integration/smoke tests against real Paperclip server. |
Suppressed comments (3)
crossbar_llm/paperclip_tools/usage.py:65
- Docstring still points at the PubTator3 logger name; it should match the Paperclip logger configured above.
Totals are logged per model name at INFO level on
`crossbar_llm.pubtator3.usage`.
"""
crossbar_llm/paperclip_tools/usage.py:75
- Log message prefix says "pubtator3" which makes Paperclip usage lines ambiguous in aggregated logs.
"pubtator3 llm usage model=%s input=%s output=%s reasoning=%s "
"cache_read=%s total=%s",
crossbar_llm/pubtator3_tools/agent.py:9
- Docstring references
agents.schemas/agents.usage, but the actual modules live undercrossbar_llm.pubtator3_tools.*, so these pointers are wrong.
Pydantic schemas live in `agents.schemas`; token-usage helpers live in
`agents.usage`. The names are re-exported here for backwards compatibility
with code that imported them from this module directly.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| @@ -0,0 +1,90 @@ | |||
| """Token-usage capture / logging helpers for the PubTator3 graph. | |||
| from langchain_core.callbacks import get_usage_metadata_callback | ||
|
|
||
|
|
||
| _usage_logger = logging.getLogger("crossbar_llm.pubtator3.usage") |
Comment on lines
+235
to
+241
| decision = PaperclipRouterDecision( | ||
| question_type="keyword_search", | ||
| source="pmc", | ||
| search_query=state["question"], | ||
| map_question=state["question"], | ||
| rationale=f"router error fallback: {e}", | ||
| ) |
Comment on lines
+342
to
+352
| __all__ = [ | ||
| "_add_warning", | ||
| "_message_content_to_text", | ||
| "_extract_json_object", | ||
| "_ainvoke_structured_with_json_fallback", | ||
| "_is_confident_match", | ||
| "resolve_node", | ||
| "partner_discovery_node", | ||
| "search_node", | ||
| "export_node", | ||
| ] |
Comment on lines
+3
to
+5
| This module defines `build_graph`, which wires the standalone nodes from | ||
| `agents.nodes` together with three inline LLM-bound nodes (router, | ||
| synthesizer, depth evaluator) that close over the chat model and prompts. |
paperclip_tools/usage.py was a verbatim copy of the PubTator3 helper, so both packages logged to `crossbar_llm.pubtator3.usage` with a "pubtator3" message prefix. Paperclip now logs under its own namespace, so the two can be filtered and configured independently. The router's error fallback pinned source="pmc". That was a leftover from when the MCP path required an explicit -s; REST made broad search the documented default. Narrowing the search on the failure path is the wrong direction, so it now falls back to broad. pubtator3_tools/nodes.py listed `_message_content_to_text` and `_extract_json_object` in __all__ without defining them, so `import *` raised AttributeError. Also corrects module paths in docstrings left over from the earlier `agents/` layout. Both behavioural fixes have regression tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two independent literature-evidence agents, each in its own package alongside
agent_tools/. Purely additive — no existing file is modified.Structure
Each package holds its own client/adapter, LangGraph agent, prompts, schemas and tests. Neither imports the other — verified by asserting that importing one leaves the other absent from
sys.modules. The two small provider-agnostic helpers they both need (structured-output-with-JSON-fallback, token usage capture) are duplicated rather than shared, so a change to one agent cannot break the other.Contract
Both expose the same shape:
question_type == "out_of_scope"withfinal_answer is Nonemeans the agent declines — so a caller can tell "not my kind of question" apart from an answer or a failure. Useful for routing: KG questions (shortest path between X and Y) decline cleanly rather than inventing literature answers.build_graphtakes an already-builtchat_model, which keeps the graph independent of model configuration and is the seam the tests inject fakes at. Each package'sllm.pyis the convenience path for callers, building that model throughagent_tools.llm_factory.Not wired up
Nothing calls the agents yet — wiring them into the API is a separate change.
Dependencies
Runtime:
aiolimiter>=1.2.1,langchain-mcp-adapters>=0.3.2. Tests:pytest-asyncio>=1.4.0,pytest-httpx>=0.36.2.Resolved against this branch's lockfile on Python 3.12.11 with no change to existing pins (langchain-core 1.5.0, langgraph 1.2.6, httpx 0.28.1, pydantic 2.13.4) — 4 lines added.
Tests
172 offline tests. Network tests are marked
liveand skip without credentials;-m "not live"excludes them explicitly. The rootpytest.ini(testpaths = crossbar_llm/tests) is untouched, so a plainpytestrun is unaffected.Known constraints
mapstep at 100 operations/day per API key, shared across all users of a deployment. The agents run abstracts-only by default for this reason.mapis currently returning a server-side error (ERR: map: 'tpm_used') upstream; abstracts-only is unaffected. Reported to their support along with four other reproducible issues.