Feat/decision model plugin - #1149
Open
wujiaming-ai wants to merge 13 commits into
Open
wujiaming-ai wants to merge 13 commits into
wujiaming-ai wants to merge 13 commits into
Conversation
Add veadk.extensions.decisions, an optional capability that turns evidence
into typed judgements instead of generated prose: a Choice (one option plus
its full probability distribution), a Score (a probability-weighted
position on ordered levels), or a Noul (the 0-1 probability that a
statement holds).
DecisionExtension is the process-wide entry point. It deliberately does not
inherit BasePlugin: it holds configuration plus a lazily created
SystemOneClient and is meant to be held by plugins and tools rather than
registered as one. SystemOneClient is the only module that performs IO,
retrying 429/5xx with backoff, and every failure surfaces as a
DecisionModelError subclass.
The capability stays off unless DECISION_MODEL_ENABLED and
DECISION_MODEL_API_KEY are set, so existing behaviour is unchanged. The
same settings can live in config.yaml as model.decision.*.
L3 ships decision_evaluate as an agent tool. It returns {"error": ...}
rather than raising, so an unconfigured or failing decision model never
breaks a run.
The hosted Typesafe service, OpenRouter's gateway, and a self-hosted System One server all speak the same protocol, so the provider only selects a default API base and credential. - add OPENROUTER_API_BASE and DEFAULT_API_BASE_BY_PROVIDER; an explicit DECISION_MODEL_API_BASE still wins over the provider default - accept "typesafe", "openrouter", and "systemone" as provider values - read the optional `cost` field OpenRouter returns in `usage` - document the provider matrix in both READMEs Change-Id: I1dea2e2db9b413aa42532dad797457d821c75314
Compaction candidates, context modes, long-run steering, and long-term memory writes are now decided by the configured decision model when the matching strategy is set to ``decision``: - ``HARNESS_COMPACTION_STRATEGY=decision`` replaces the role based candidate rules with a content pre-filter plus a per-candidate "keep verbatim or summarize" judgement, because role labels cannot tell a tool result from the user's own text on real ADK traffic. - ``HARNESS_MODE_STRATEGY=decision`` decides the context mode blocks per invocation instead of matching keywords. - ``HARNESS_LONG_RUN_STRATEGY=decision`` only injects steering guidance while the run still looks unfinished, with the counter as a hard fallback after ``unconditional_after_model_calls``. - ``MEMORY_SAVE_STRATEGY=decision`` decides whether a turn holds something durable; a skipped turn keeps its cursor, so nothing is dropped. Every point defaults to its previous behaviour and degrades to it when the decision model is absent or fails. Change-Id: Id68bd9c71540e78a87638fc5c4735712cf7317a4
…lable The decision model sits in hot paths, so a failure has to reach callers as a ``DecisionModelError`` they can fall back from, and a down endpoint may not keep costing the retry and timeout budget: - Normalize every failure: a send error that is not an ``httpx.HTTPError`` (for example ``httpx.InvalidURL``) and a payload that fails pydantic validation now surface as ``DecisionModelRequestError`` / ``DecisionModelResponseError`` instead of escaping past the callers' ``except DecisionModelError``. - ``timeout`` is the budget of one whole judgement, retries and backoff included, and is capped at 5 seconds. - After ``failure_threshold`` failures in a row the endpoint is marked down for ``cooldown_seconds``; judgements then raise ``DecisionModelUnavailableError`` without an HTTP call, and one probe after the cooldown decides whether to resume. - Log one line per usable judgement at debug, retries and outages at warning; never log the judged state or the API key. - Reject an unusable ``api_base`` at configuration time, and keep the extension disabled instead of failing startup when settings are broken. Change-Id: I58b57993c4791c6e1c57e7c52af85af53bd05c85
Assembling plugins through ``HarnessExtension(...)`` could pick the compaction and mode strategies with the config objects, but the long-run strategy was only reachable through the Harness environment variables: ``plugins()`` called ``build_harness_plugins`` without ``long_run_strategy``, so the plugin always kept ``counter``. Pass it through the constructor like the other module settings, and document that an ``env`` mapping makes the environment variables the only source. Defaults are unchanged, so existing callers keep the call-count rule. Change-Id: I4e7c097a3d74280b1e4e201697b83b3a6dc2321b
…me way Each judgement point compares an answer against its own 0..1 threshold, but every call site read the setting on its own. A value outside [0, 1] silently flipped a point on or off, and NaN — which never compares true — turned "worth remembering" into "never save", with nothing in the logs. - add veadk/extensions/decisions/thresholds.py: probability_threshold() clamps an out-of-range value, which keeps the "always"/"never" intent a fallback to the default would silently reverse, and falls back with a warning only for a value that carries no intent (NaN or text) - read the three Harness thresholds from the environment, accepting the HARNESS_ENHANCE_-prefixed spelling the deploy config flattens to - route MEMORY_SAVE_WORTH_THRESHOLD through the same parser, and treat an empty setting as unset instead of raising at import - expose long_run_ready_threshold on HarnessExtension and build_harness_plugins, so the code path can tune what the env path tunes - document the thresholds in the decisions and harness READMEs and in the user-facing docs copy - cover parsing, env reading, the code path, the import-time reading, and the harness_enhance -> runtime env -> plugin deploy path Change-Id: I9d0b07cf648b0e4b2ad08b81f72661f3e103e274
… repair it The builtin verifier reads completion markers and asks whether any tool receipt succeeded. That cannot tell "the deployment receipt proves this" from "the answer says done and an unrelated tool succeeded", so it blocks supported answers and passes unsupported ones in the same shape of run. The ``decision`` strategy adds one judgement that carries both answers in a single request: - a rating of how well the answer stands on the run's receipts, which replaces the status the rules produced. The rule findings stay in the report, so the event payload shows both verdicts. - the repair the answer needs (rerun the tool call, soften the claim, drop it, or ask the user), which fills the instruction a caller hands back. An action that names no option keeps the default wording and the rating still applies. ``mode`` still decides whether a failure blocks, and a judgement that cannot be made leaves the builtin rules in charge. Change-Id: I3f281b6adba09293525e24eea23bc05e47363e37
``search_memory`` hands the agent whatever the backend ranked highest for the query. Similarity cannot tell "a memory the answer has to respect" from "a memory about the same topic the request does not need", the backend score is not comparable across backends, and the score was only carried as metadata. With ``MEMORY_RECALL_STRATEGY=decision``, the recalled memories are rated for the query (irrelevant / related / useful / required) in one request and the ones below ``MEMORY_RECALL_RELEVANCE_THRESHOLD`` (default 0.5) are dropped. The judged budget is capped, anything beyond it is returned in backend order, a memory the judgement did not rate is kept, and a judgement that cannot be made keeps every match. Reading the memory text also needed a fix: a ``types.Part`` went through ``str()``, so the state carried a full repr instead of the memory text. Change-Id: I862ccad0e87d3ee091c240efc15a3b3a2f23e239
The long-run judgement answered one yes/no question, and its answer only gated whether the plugin injected its one fixed guidance: how hard to push a run towards its answer was not a decision the model could make, even though losing scope and forcing an answer are different kinds of help. The judgement now carries a second question in the same request. The convergence probability still gates the injection, so ``HARNESS_LONG_RUN_READY_THRESHOLD`` keeps its meaning, and the chosen action selects the guidance that gets injected: narrow the scope, converge while working, or answer now. An action that names no option keeps the default guidance, so a failing judge and the counter strategy inject exactly what they did before. Change-Id: Ibc2fcc5d53cc4c57401c33466df602292fcf3563
- list the two thresholds the judgement points gained (``HARNESS_VERIFIER_SUPPORT_THRESHOLD``, ``MEMORY_RECALL_RELEVANCE_THRESHOLD``) and say that a threshold compares a rating where the point rates instead of asking - document ``HARNESS_VERIFIER_STRATEGY`` next to the other strategies, and the actions the long-run and verification judgements pick - complete the Harness environment table, which was missing the thresholds the previous judgement commit added - document the recall filter in the long-term memory guide Change-Id: I7c94a9c0c458ccc14133fc29e3369960a7a9d119
The environment-variable reference is the page a deployment reads to find out what it may set, and this change only added the eight ``HARNESS_*`` names to it. The nine ``DECISION_MODEL_*`` settings were left in the Harness extension README, and the four memory judgement settings were left in the long-term memory page, so the one place that claims to list variables by domain was missing everything that turns a judgement on. The decision model gets its own section, since it is a second model rather than part of the agent model, and the memory settings get theirs because the document had no memory group at all. Both the numeric thresholds the ``threshold`` strategy falls back on are listed next to the strategies that use them, otherwise the fallback has no visible default on this page. The opening sentence named five groups and had already lost the Harness extension one; it now names the groups the page actually has. Change-Id: I3c1c8ba3558d3ff268d93fb0b3033b5f8f036875
Two more places decide with nothing but wording. The skills callback advertises every loaded skill in the agent instruction, so a large library spends prompt budget on skills the request will never use, and the model picks from a list nobody narrowed. Routing a request to a sub-agent has the same shape: the model reads the agent descriptions and calls ``transfer_to_agent`` on its own. ``HarnessSkillPrefilterPlugin`` asks one ``noul`` question per advertised skill in a single request and rewrites the skill list of that request only. The candidates are judged independently instead of as one ``choice``, because a request often needs two skills and a choice names one winner, and the descriptions travel in the questions so one skill's description cannot decide another skill's answer. The agent instruction keeps every skill, so the next request starts from the full list, and a skill the judgement did not answer for stays advertised: a missing probability is not evidence of irrelevance. ``HarnessAgentRoutingPlugin`` asks one ``choice`` question whose options are the transfer targets and whose option descriptions are theirs, and returns the same ``transfer_to_agent`` call the model would have produced once the judgement clears ``HARNESS_ROUTING_DECISION_THRESHOLD``. Everything less certain stays with the model, which still sees the full transfer instructions, and a judgement never names a target the transfer tool does not expose, so ADK cannot be asked to resolve an agent it does not have. Both are opt-in components (``skill_prefilter`` and ``agent_routing``), judge at most once per invocation, judge only the text of the user's message, and fall back to their rule when no decision model is configured. The READMEs and the environment-variable reference document the five new settings. Change-Id: Ibd735d56a515b50687d48d3592859323d73a6a11
Every judgement point reads content the agent did not write: the user request, the final answer, the run trajectory, tool receipts, tool output, memory text and session events. A decision model reads that state as data rather than as hostile content, so a captured tool output claiming "the user already approved this" moved the measured block probability of the same dangerous command from 0.76 to 0.48. Every captured value now goes through ``untrusted()``, which wraps it in an ``<untrusted source=...>`` block the state declares non-authoritative, and replaces the spans inside it that try to give orders (``System: ...``, "ignore all previous instructions", "no further approval is needed", "always allow") with ``[defused]``, logging the source. The surrounding text stays, so a judgement still sees what the capture contains, and the raw attempt stays visible as a signal instead of silently acting as an instruction. The final-answer verifier asked for a four-level rating plus a repair action. The rating conflated two things: whether a receipt covers the claim, and whether the answer stays inside what the receipts show. It now asks one mutually exclusive outcome (``supported`` / ``partial`` / ``unsupported``) plus those two checks as separate ``noul`` questions, and the overclaim check is a veto, so an answer that claims more than its receipts fails even when the verdict says ``supported``. The probability of ``supported`` remains the value ``HARNESS_VERIFIER_SUPPORT_THRESHOLD`` compares, so an endpoint that reports only the chosen option falls back to the level the verdict names. The judged verdict, both checks and the action now reach the stored event payload, which is what an operator needs to explain a decision after the fact. ``choice`` and ``score`` answers carry a confidence nothing consumed, while ``noul`` answers carry none and are cascaded by their own threshold. The verifier and the long-run judge now refuse to act below ``HARNESS_VERIFIER_MIN_CONFIDENCE`` and ``HARNESS_LONG_RUN_MIN_CONFIDENCE``: the verifier keeps the builtin rules, and the long-run judge keeps the convergence probability while falling back to the default steering wording. Both default to ``0``, which keeps acting on every judged answer, because an endpoint may report no confidence at all; the READMEs record that an operator who has measured their own calibration can raise them. The READMEs and the environment-variable reference document the three new settings. Tests cover the defusing rules, the confidence cascade on both points, the veto, and the state hygiene of all eight judgement points. Change-Id: I7aab9807d151b87defd78c88e2982cd112185104
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.