diff --git a/content/docs/configuration/librechat_yaml/object_structure/classification.mdx b/content/docs/configuration/librechat_yaml/object_structure/classification.mdx new file mode 100644 index 000000000..e9fd9a022 --- /dev/null +++ b/content/docs/configuration/librechat_yaml/object_structure/classification.mdx @@ -0,0 +1,336 @@ +--- +title: "Classification Object Structure" +icon: SlidersHorizontal +--- + +## Overview + +The `classification` configuration connects LibreChat to a classifier: a model that +answers small typed questions about text and returns probabilities, rather than +writing prose. Features use it where a decision has to be made before or around a +model call, such as picking which tools a turn needs. + +Two question shapes are available: + +- **boolean** asks whether something holds and returns the probability of yes. +- **choice** picks one option from a defined set and returns the full distribution. + +Classification is disabled by default. When it is off, or when no provider is +configured, every feature built on it behaves exactly as it does without it. + +Every enabled capability sends conversation text to the configured host. Tool +selection sends the latest request and the deferred tools' descriptions; the memory +gate sends the last few messages. Point `baseURL` at a host you are allowed to send +that data to. + +Classifier usage is recorded as transactions with the context `classification`, so it +appears in token usage and draws on the balance like any other model call. TypeSafe's +Jev is priced at $0.042 per million input tokens with free output; a model with no +known price is charged at the default rate. + +## Example + +```yaml filename="Classification Object Structure" +classification: + enabled: true + provider: http + providers: + http: + baseURL: https://classifier.example.com/v1/classify + apiKeyEnv: CLASSIFIER_API_KEY + timeoutMs: 4000 + maxRetries: 2 + + toolSelection: + enabled: true + shortlist: 5 + needsToolThreshold: 0.15 + lowConfidenceExtra: 3 + + memoryGate: + enabled: true + threshold: 0.5 +``` + +## Configuration + +### Subkeys + + + +## providers + +Each provider reads its API key from an environment variable. The key is never +written in `librechat.yaml`; only the name of the variable is. + + + +### Presets + +The hosts that serve a classification model over HTTP differ only in their URL, +their model name and how they wrap the body. That is configuration rather than +behaviour, so each one ships as a preset: a set of defaults you can name, and +override field by field. + + + +Naming a preset is usually the whole configuration: + +```yaml filename="A preset, with nothing to add" +classification: + enabled: true + provider: typesafe + providers: + typesafe: + apiKeyEnv: TYPESAFE_API_KEY +``` + +`cloudflare` is the exception, because its URL carries your account id. Without +one it stays off and says so rather than guessing: + +```yaml filename="Cloudflare" +classification: + enabled: true + provider: cloudflare + providers: + cloudflare: + baseURL: https://api.cloudflare.com/client/v4/accounts//ai/run + apiKeyEnv: CLOUDFLARE_API_TOKEN +``` + +### A host with no preset + +A host nobody has written a preset for needs no code either, only its shape: + +```yaml filename="Any other host" +classification: + enabled: true + provider: inhouse + providers: + inhouse: + baseURL: https://classify.internal/v1/run + model: your-model + dialect: systemone + requestKey: input + responseKey: result + apiKeyEnv: INHOUSE_CLASSIFIER_KEY +``` + + + A provider entry rejects keys it does not recognise, so `requestkey` fails at + config load instead of being dropped and surfacing later as a missing setting. + + +### What can answer these questions + +This expects a classification model, not a chat model. The difference is not the +protocol but what comes back: a classifier returns a probability it was trained to +calibrate, and every threshold on this page reads one. A chat model asked to report +its own confidence returns a number that looks the same and means much less. + +That narrows the field considerably. Two model families answer caller-defined +questions with calibrated probabilities: + + + +Laya ships as weights, so it needs a small service in front of it. It implements +the same three question shapes, so `dialect: systemone` describes it and no code +is involved: + +```yaml filename="A self-hosted classifier" +classification: + enabled: true + provider: laya + providers: + laya: + baseURL: http://laya.internal:8080/classify + dialect: systemone + apiKeyEnv: CLASSIFIER_API_KEY +``` + +Three things that look like they should work and do not: + +- **Chat completion endpoints**, including OpenAI, Anthropic and the general + OpenRouter catalog. No provider here speaks chat completions, and nothing reads + logprobs. +- **Moderation endpoints** such as OpenAI's or Mistral's. They return scores over a + fixed taxonomy and cannot answer a question you define. +- **Zero-shot classification by embedding similarity**, such as Jina's classifier or + a hosted NLI model. These do take caller-defined labels, but the scores are + similarities and nobody claims they are calibrated. They will work in the + mechanical sense and quietly make the thresholds meaningless. + + + If a provider cannot measure confidence it should report it as absent rather than + as a number. Tool selection treats absent confidence as unconfident and widens the + shortlist, which is the safe direction; a fabricated `0` or `1.0` defeats that. + + +## toolSelection + +An agent holding a large MCP catalog pays for every tool's argument schema on every +turn. Deferring those schemas solves the cost, but the model then spends a +`tool_search` round trip discovering the tool it needs, on the turn it needs it. + +Tool selection ranks the deferred tools against the request before the model runs +and surfaces the few it is likely to need, so their schemas arrive with the first +call. It requires tools that are already deferred, whether by the per-tool toggle in +the agent builder or by [`mcpSettings.deferSchemaChars`](/docs/configuration/librechat_yaml/object_structure/mcp_settings). + +Selection only ever adds. A tool it passes over stays listed by name in the +`tool_search` description and remains one search away, so a wrong prediction costs a +round trip rather than an answer. + + + +## memoryGate + +The memory model runs after every turn, and most turns ask it for nothing. The +gate asks one yes/no question first and skips that call when the answer is no. + +By default the question is whether the user's newest message asks the assistant to +remember, update or forget something. That matches the default memory +instructions, which only store what the user asks to keep: a preference mentioned +in passing is skipped by both, while "please remember...", "forget where I live" +and a short yes to an offer to remember all go through. If your +`memory.instructions` also keep facts the user did not ask to keep, replace the +question to match (see [Customizing the questions](#customizing-the-questions)). + +It sits in front of the automatic memory agent, so it requires +[`memory`](/docs/configuration/librechat_yaml/object_structure/memory) to be +configured with `memory.agent.enabled: true`. Without automatic extraction there is +no call to skip, and the gate never runs. + +A gate that cannot reach its provider answers yes, so an outage never causes a +memory to be lost. + +The gate judges only the user's newest message. The few messages before it are sent +along as context, so "yes, remember that" is read against the offer it answers, but +an earlier request is not judged again on the next turn. + + + +`categorize` and `detectUpdates` ride in the request the gate already makes, so +they cost no additional round trip. + +Each capability can set its own `timeoutMs` because one ceiling does not suit +both. Ranking a large catalog is a request of several thousand tokens; the gate +asks one short question about the recent turn. The result reaches the memory model as a +suggestion it may ignore, not as an override: + +```text + +This request most likely belongs under `work_context`. It looks +like a change to what is already stored there, not a new fact. +Ignore this if it does not fit what the user actually said. + +``` + + + `whenTrue` and `whenFalse` are named that way because YAML reads bare `true:` and + `false:` keys as booleans rather than strings. + + +## Customizing the questions + +Every question ships with default wording, and an operator can replace it without +touching code. A support deployment can tell the ranker what its tools are for, and +a deployment with its own idea of what is worth remembering can say so: + +```yaml filename="Custom question wording" +classification: + toolSelection: + instructions: Pick the tool for this support ticket. + guidance: Prefer billing tools for anything about an invoice. + memoryGate: + instructions: Does `latest` mention a dietary requirement? + whenTrue: An allergy or a standing preference. + whenFalse: Anything about one meal only. +``` + +A deployment whose `memory.instructions` keep anything worth remembering, not only +what the user asks to keep, can ask about that instead: + +```yaml filename="Gate on anything worth remembering" +classification: + memoryGate: + instructions: Does `latest` say something about the user that would still matter weeks from now? + whenTrue: A lasting preference, or a fact about who they are or what they work on. + whenFalse: Small talk, or a detail that only matters inside this task. +``` + +## Notes on accuracy + +A classifier returns probabilities, and the thresholds above decide what to do with +them. Two properties are worth understanding before enabling either capability: + +- Thresholds are domain specific. Start with the defaults, observe your own traffic, + and adjust rather than assuming the defaults transfer. +- Probabilities are only useful when the provider produces calibrated ones. A + provider that cannot measure confidence reports it as absent, and tool selection + treats that as unconfident and widens the shortlist instead of narrowing it. +- Low confidence usually means two candidates are close, not that the answer is + poor. A catalog holding several tools that genuinely could serve the same request + will report low confidence often, which is the case `lowConfidenceExtra` exists + for: surfacing both costs one extra schema, while surfacing the wrong one alone + costs a `tool_search` round trip. Raise `lowConfidenceBelow` if your catalog has + many near-duplicates, lower it if its tools are sharply distinct. diff --git a/content/docs/configuration/librechat_yaml/object_structure/mcp_settings.mdx b/content/docs/configuration/librechat_yaml/object_structure/mcp_settings.mdx index da823e629..f7380b3b2 100644 --- a/content/docs/configuration/librechat_yaml/object_structure/mcp_settings.mdx +++ b/content/docs/configuration/librechat_yaml/object_structure/mcp_settings.mdx @@ -24,6 +24,10 @@ mcpSettings: - "host.docker.internal:8080" # Permit one private host on one port - "10.0.0.5:8000" # Permit one private IP on one port + # Withhold an MCP tool's argument schema from the model when its JSON is longer + # than this many characters. 0, the default, disables the rule. + deferSchemaChars: 4096 + # Optional MCP OAuth catalog-recovery tuning; defaults are shown. catalogRecovery: discoveryBackoffMs: [300000, 600000, 1200000, 1800000] @@ -45,6 +49,7 @@ mcpSettings: options={[ ['allowedDomains', 'Array of Strings', 'A list specifying allowed domains for MCP server connections.', 'When configured, only listed domains are allowed. When not configured, SSRF targets are blocked but all other domains are allowed.'], ['allowedAddresses', 'Array of Strings', 'An SSRF exemption list, scoped to private IP space. Hostname/IP + port pairs listed here bypass the default-deny SSRF block when `allowedDomains` is not configured.', 'Use when you want default SSRF protection AND specific internal MCP servers, without flipping `allowedDomains` into strict-whitelist mode.'], + ['deferSchemaChars', 'Number', 'Argument-schema size, in characters of its JSON, above which an MCP tool starts deferred: the model sees only its name and loads the schema through `tool_search` when it needs it. Needs the agents `deferred_tools` capability. A per-tool `defer_loading` toggle still takes precedence in either direction.', '0, which disables the rule and leaves every tool exactly as its own options configure it.'], ['catalogRecovery', 'Object', 'Optional bounds for passive MCP discovery, OAuth reauthentication recovery, and cross-replica authorization fencing.', 'Uses the defaults shown in the example.'], ]} /> @@ -197,3 +202,29 @@ A hostname entry trusts whatever IP that hostname resolves to at runtime on the - [MCP Servers Configuration](/docs/configuration/librechat_yaml/object_structure/mcp_servers) - [MCP Features](/docs/features/mcp) - [Actions allowedAddresses](/docs/configuration/librechat_yaml/object_structure/actions#allowedaddresses) (similar concept for Actions) + +## deferSchemaChars + +A small number of MCP tools can account for most of a catalog's size. A tool whose +argument schema is very large costs those tokens on every turn, even when the +conversation has nothing to do with it, while its description may be only a line. + +Setting `deferSchemaChars` withholds the argument schema of any tool whose JSON is +longer than that many characters. Characters track the tokens the rule saves more +closely than bytes would. The model still sees the tool's name in the `tool_search` +listing and loads the schema when it decides to use it, so nothing becomes +unreachable. The rule only applies when the agents `deferred_tools` capability is +enabled; with it off, every tool loads in full. + +Precedence, highest first: + +1. A per-tool `defer_loading` toggle in the agent builder, in either direction. +2. `mcpSettings.deferSchemaChars`, when the tool has no explicit toggle. +3. Otherwise the tool is loaded in full. + +Because an explicit per-tool choice always wins, the rule is safe to apply +instance-wide without overriding tools an author deliberately pinned open. + +In the agent builder, a tool deferred by this rule shows its defer toggle as on, +with a tooltip saying it was deferred automatically. Switching it off keeps that +tool loaded for the agent; switching it back on returns the tool to the rule. diff --git a/content/docs/configuration/librechat_yaml/object_structure/memory.mdx b/content/docs/configuration/librechat_yaml/object_structure/memory.mdx index 5fd04c908..69b5b18fc 100644 --- a/content/docs/configuration/librechat_yaml/object_structure/memory.mdx +++ b/content/docs/configuration/librechat_yaml/object_structure/memory.mdx @@ -370,3 +370,4 @@ memory: - Agent configuration allows customization of automatic memory processing behavior - When disabled, all memory features are turned off regardless of other settings - The message window size affects how much recent context is considered for memory updates +- [`classification.memoryGate`](/docs/configuration/librechat_yaml/object_structure/classification#memorygate) can skip the automatic memory agent on turns that do not ask to remember, update or forget anything, and suggest which valid key a request concerns diff --git a/content/docs/configuration/librechat_yaml/object_structure/meta.json b/content/docs/configuration/librechat_yaml/object_structure/meta.json index aa15aff93..bf10d6b5b 100644 --- a/content/docs/configuration/librechat_yaml/object_structure/meta.json +++ b/content/docs/configuration/librechat_yaml/object_structure/meta.json @@ -26,6 +26,7 @@ "mcp_settings", "skill_sync", "memory", + "classification", "summarization", "message_filter", "web_search", diff --git a/content/docs/configuration/librechat_yaml/object_structure/shared_endpoint_settings.mdx b/content/docs/configuration/librechat_yaml/object_structure/shared_endpoint_settings.mdx index 169472939..fc8b58353 100644 --- a/content/docs/configuration/librechat_yaml/object_structure/shared_endpoint_settings.mdx +++ b/content/docs/configuration/librechat_yaml/object_structure/shared_endpoint_settings.mdx @@ -490,18 +490,21 @@ When Langfuse tracing is enabled, the bounded reasoning snapshot is recorded as [ 'maxToolResultChars', 'Number', - 'Limits the maximum number of characters in tool call results sent to the model. Must be a positive number.', + 'Limits the number of characters of a tool call result kept in the conversation. Must be a positive number.', 'maxToolResultChars: 50000', ], ]} /> -**Default:** No limit +**Default:** 30% of the model's context window, at most 400,000 characters **Notes:** -- Helps prevent excessively large tool outputs from consuming too many tokens -- Applies to all tool call results for the endpoint +- A longer result is cut before the model sees it, with a marker telling the model it was cut +- The cut result is what gets stored, so every later turn re-sends the shorter version +- Applies to all tool call results for the endpoint, including MCP tools +- Set under `all` for every endpoint, or on one endpoint to override `all` +- Too low a cap costs calls instead of saving tokens: a model that cannot see the part of a result it needs searches again in smaller pieces. For tools that return whole pages, such as browser tools, start around 28,800 rather than a few thousand **Example:**