From 5ae298b163afc10a808c1bdde74a6cb885e20f17 Mon Sep 17 00:00:00 2001 From: Maxim Svistunov Date: Thu, 10 Sep 2026 17:03:52 +0200 Subject: [PATCH 1/4] LCORE-3386: align prompt guardrails spec with the shipped shield-based design The prompt guardrails spec described a standalone `guardrails:` config section with separate detectors and rules, a `src/guardrails/` package, a `DetectorBackend` protocol and a structured `ScreeningItem` payload. The detector framework that shipped under LCORE-3389 (PR #2580) took a different route: Granite Guardian is a shield type in the existing `shields:` list, configured through `GraniteGuardianShieldConfiguration`, `GraniteGuardianConfig` and `RiskDefinition`, and evaluated by an `AbstractSafetyCapability`. The spec was not updated at the time, so the tickets implementing the input and output points were being reviewed against a design that no longer matched the code. Rewrite the What, Requirements, Architecture, acceptance test surface, aspect-specific concerns and implementation suggestions sections to describe the shipped design: the shield's risk selection by point, the Granite Guardian 4.1 judge prompt, logprob-based scoring against the per-risk threshold, the two evaluation paths (run_shield_moderation_v2 before RAG on the Responses-based endpoints, agent capabilities on the agent-based endpoints), and the fail-closed error handling as implemented. Keep the requirements that still apply and name the ticket that covers each open one: concurrent evaluation and per-risk latency (LCORE-3390), advisory risks via a new `blocking` flag and streaming checkpoints (LCORE-3391), the validation-error metric that currently has no callers (LCORE-4089), and skipping RAG, the main LLM call and the topic-summary call for input-blocked requests on /v1/query and /v1/streaming_query (LCORE-4090). R6 is extended to cover topic-summary calls and RAG documents in the response, because the in-agent path currently returns both for blocked queries. Move capabilities of the original design that the shipped configuration does not provide into a new "Deferred from the original design" section: the separate config section and backend protocol, the openai_moderations and llama_stack_shields backends, out-of-the-box risk ids, the boolean verdict without a threshold, fail-open and refusal-shaped detector failures, api_key_path, the concurrent input execution mode and a global violation message. Record model selection and guardian token accounting as open questions, note that the `enable_thinking` docstring points at a `ModerationConfig.thinking_enabled` setting that does not exist, and add a changelog row explaining the revision. --- .../prompt-guardrails/prompt-guardrails.md | 652 +++++++++--------- 1 file changed, 340 insertions(+), 312 deletions(-) diff --git a/docs/design/prompt-guardrails/prompt-guardrails.md b/docs/design/prompt-guardrails/prompt-guardrails.md index 7b70d49e9..e07e6eb8d 100644 --- a/docs/design/prompt-guardrails/prompt-guardrails.md +++ b/docs/design/prompt-guardrails/prompt-guardrails.md @@ -2,361 +2,361 @@ | | | |--------------------|-------------------------------------------| -| **Date** | 2026-07-20 | +| **Date** | 2026-07-20 (revised 2026-09-10) | | **Component** | lightspeed-stack | | **Authors** | Maxim Svistunov | | **Feature** | [LCORE-230](https://redhat.atlassian.net/browse/LCORE-230) | +| **Epic** | [LCORE-3386](https://redhat.atlassian.net/browse/LCORE-3386) | | **Spike** | [LCORE-2657](https://redhat.atlassian.net/browse/LCORE-2657) | -| **Links** | [Spike doc](prompt-guardrails-spike.md), [OWASP LLM01](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) | +| **Links** | [Spike doc](prompt-guardrails-spike.md), [Shields guide](../../user_doc/shields_guide.md), [OWASP LLM01](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) | + +> **Revision note (2026-09-10).** The first version of this document proposed +> a standalone `guardrails:` configuration section and a `src/guardrails/` +> package with pluggable detector backends. The configuration that shipped +> under LCORE-3389 (PR #2580) instead builds guardrails as a **shield type** +> inside the existing shields framework. This revision describes that +> architecture. Capabilities of the original design that the shipped +> configuration does not provide are listed in +> [Deferred from the original design](#deferred-from-the-original-design); +> requirements that are still open name the ticket that covers them. ## What -An optional, config-driven guardrails layer owned by lightspeed-stack. -Deployers declare **detectors** (guardian-model endpoints reachable through -OpenAI-compatible APIs — Granite Guardian on vLLM/RHAIIS, any -`/v1/moderations` service, or, transitionally, OGX shields) and -**rules** (an out-of-the-box risk id or a custom risk definition, bound to -one or more guardrail **points**: `input`, `output`, `tool_content`, with a -blocking or advisory posture). The layer runs the applicable rules in -parallel at each point of the request lifecycle and blocks (or annotates) -requests whose content is flagged. +An optional, config-driven guardrails layer owned by lightspeed-stack and +built on its shields framework. A deployer adds a `granite_guardian` entry to +the top-level `shields:` list in `lightspeed-stack.yaml`. The entry names an +IBM Granite Guardian model reachable through an OpenAI-compatible API and +declares a list of **risks**. Each risk carries a custom risk definition, a +score threshold, a violation message, and the guardrail **points** where it +applies: `input`, `output`, or `tool`. At each point the applicable risks are +evaluated, and content that is flagged is blocked with that risk's violation +message. ## Why -Prompt injection is OWASP's #1 LLM risk. lightspeed-stack today moderates -only *input*, only through OGX shields — an OGX API surface upstream -has deleted in OGX 1.x — with no lightspeed-stack-side configuration, no -output or tool-content coverage, no Granite Guardian support, and no custom -risk definitions. Ask Red Hat's migration to Lightspeed Core -([LCORE-2253](https://redhat.atlassian.net/browse/LCORE-2253)) is blocked -on exactly those capabilities (they run parallel multi-risk Granite -Guardian screening with custom risks in production today). This feature -provides them generically, in a form that survives the planned OGX -phase-out. +Prompt injection is OWASP's #1 LLM risk. Before this feature, lightspeed-stack +moderated only *input*, and only through OGX shields -- an API surface that +OGX 1.x removed -- with no lightspeed-stack-side configuration, no output or +tool-content coverage, no Granite Guardian support, and no custom risk +definitions. Ask Red Hat's migration to Lightspeed Core +([LCORE-2253](https://redhat.atlassian.net/browse/LCORE-2253)) depends on +exactly those capabilities: they run parallel multi-risk Granite Guardian +screening with custom risks in production today. This feature provides them +generically, in a form that does not depend on OGX. ## Requirements +Requirements that are not yet met by merged code name the ticket that covers +them. + - **R1:** Guardrails are configured exclusively in the lightspeed-stack - config file under a top-level `guardrails:` section; absent config means - fully inert (no behavior change, no latency). -- **R2:** A rule can reference an out-of-the-box guardian risk (e.g. - `harm`, `jailbreak`, `answer_relevance`) or carry a custom risk - definition (bring-your-own-criteria text). Custom definitions must - express **safety-adjacent concepts** (obfuscation, roleplay jailbreak, - policy violation), not arbitrary string/format predicates — a guardian - is a safety classifier, not a keyword matcher (PoC Finding A). Arbitrary - predicates are the regex-redaction mechanism's job, not a guardrail's. -- **R3:** A rule binds to one or more guardrail points: `input` (user - prompt before the LLM call), `output` (generated answer before the - client sees it), `tool_content` (tool/MCP/RAG content before it enters - the model context). -- **R4:** All rules applicable at a point run concurrently (the existing - OGX shields path is a sequential loop — `src/utils/shields.py:152` - — which the Ask Red Hat gap analysis flags as a performance gap); a - request is blocked iff at least one *blocking* rule flags it. Advisory - (`blocking: false`) rules record their outcome without altering the - response. -- **R4a:** A rule may carry an optional `threshold` (0..1). When set, the - detector's confidence score decides the verdict (Granite Guardian via - `logprobs` on the verdict token; gateways via their native confidence - score); when unset, the boolean verdict decides. This reproduces Ask - Red Hat's per-risk tuning (0.65 leetspeak, 0.80 CVE). -- **R4b:** A rule may carry its own `violation_message`, overriding the - global default, so deployers can explain which policy fired. -- **R4c:** Recommended/default rule sets shipped in documentation must be - validated against a corpus of legitimate product questions and must not - fire on it. Out-of-the-box guardian risk ids (notably `jailbreak`) flag - legitimate technical questions — "You are now a cluster admin, how do I - drain a node?" scores 0.98 — at levels no threshold separates from real - attacks, so **domain-tuned custom definitions are the shipping default** - and OOTB ids are opt-in. -- **R4d:** A deployment may select the input-guardrail execution mode: - `blocking` (default — the model never sees unscreened input) or - `concurrent` (guardian runs alongside the LLM call, result discarded on - violation; lower latency, but the model processes unsafe input). -- **R5:** A blocked request returns HTTP 200 with the configured violation - message (consistent with existing OGX shields refusals): non-streaming - responses carry it as the answer; streaming responses emit it as the - terminal content. The `llm_calls_validation_errors_total` metric is - incremented and the blocked turn is persisted to the conversation. -- **R6:** Input-blocked requests skip RAG retrieval and the main LLM call. -- **R7:** The Granite Guardian detector invokes the model through an - OpenAI-compatible chat-completions endpoint, selecting the risk (or - custom definition) via the guardian chat template; the OpenAI-moderations - detector invokes any OpenAI-compatible `/v1/moderations` endpoint. -- **R7a:** Output-relevance rules (`answer_relevance`, `context_relevance`, - `groundedness`) receive the turn's retrieved context (and question) - paired with the answer; an answer-only check is insufficient and noisy - (PoC Finding B). -- **R8:** Output rules on streaming endpoints check accumulated text at - configurable checkpoints; content past a failed checkpoint is never - emitted. -- **R9:** Detector errors (unreachable endpoint, timeout) block the request - by default (`on_detector_error: block`), overridable to `allow` per - deployment. -- **R10:** Per-rule detection outcomes and latencies are logged and - exposed as metrics. -- **R11:** The existing OGX shields input-moderation path continues - to work unchanged when `guardrails:` is not configured; both may run - side by side during migration. + config file, as a `granite_guardian` entry in the top-level `shields:` + list. Without such an entry the feature is fully inert: no behavior change + and no added latency. +- **R2:** A risk is defined by custom criteria text (`description`) that is + passed to Granite Guardian. Definitions must express **safety-adjacent + concepts** (obfuscation, roleplay jailbreak, policy violation), not + arbitrary string or format predicates: a guardian is a safety classifier, + not a keyword matcher (PoC Finding A). Arbitrary predicates are the job of + the redaction shield. +- **R3:** A risk binds to one or more guardrail points: `input` (the user + prompt before the LLM call), `output` (the generated answer before the + client sees it), and `tool` (tool, MCP, or RAG content before it enters the + model context). +- **R4:** All risks applicable at a point are evaluated concurrently, with + per-risk latency logged. A request is blocked if at least one *blocking* + risk flags it. (Concurrency and latency logging: LCORE-3390. The shields + loop in `src/utils/shields.py` is sequential, which the Ask Red Hat gap + analysis flags as a performance gap.) +- **R4a:** Each risk has a `threshold` between 0 and 1 (default 0.65). The + verdict is decided by Granite Guardian's confidence score, derived from the + logprobs of its verdict token, which reproduces Ask Red Hat's per-risk + tuning (0.65 leetspeak, 0.80 CVE). +- **R4b:** Each risk carries its own `violation_message`, so deployers can + explain which policy fired. +- **R4c:** Recommended rule sets shipped in documentation must be validated + against a corpus of legitimate product questions and must not fire on it. + Generic phrasings of risks such as "jailbreak" flag legitimate technical + questions -- "You are now a cluster admin, how do I drain a node?" scores + 0.98 -- at levels no threshold separates from real attacks, so + **domain-tuned custom definitions are the shipping default** (LCORE-3394). +- **R4e:** A risk can be **advisory**: it records its outcome without + altering the response. Advisory risks are required for output relevance + checks. (Needs a `blocking` flag on `RiskDefinition`, default true: + LCORE-3391.) +- **R5:** A blocked request returns HTTP 200 with the violation message: + non-streaming responses carry it as the answer, streaming responses emit + it as the terminal content. The blocked turn is persisted to the + conversation and the `llm_calls_validation_errors_total` metric is + incremented. (The metric currently has no callers anywhere: LCORE-4089.) +- **R6:** A request blocked at the `input` point performs no RAG retrieval, + no main LLM call and no topic-summary call, and its response carries no RAG + chunks or referenced documents. (Met on `/v1/responses` and `/rlsapi`; not + yet on `/v1/query` and `/v1/streaming_query`: LCORE-4090.) +- **R7:** Granite Guardian is invoked through an OpenAI-compatible + chat-completions endpoint, using its client-side judge prompt with the + risk's criteria. +- **R7a:** Output relevance risks (answer relevance, context relevance, + groundedness) receive the turn's retrieved context and question alongside + the answer; an answer-only check is insufficient and noisy (PoC Finding B). + The shield's evaluation interface currently takes plain text, so the + payload for relevance checks is to be designed in LCORE-3391. +- **R8:** Output risks on streaming endpoints check accumulated text at + checkpoints; content past a failed checkpoint is never emitted + (LCORE-3391). +- **R9:** Guardian errors (unreachable endpoint, timeout, unparseable + verdict) fail closed: the request is not served. +- **R10:** Per-risk outcomes and latencies are logged and exposed as metrics + (LCORE-3390 for input, LCORE-3391 for output). +- **R11:** The other shields (`question_validity`, `redaction`) continue to + work unchanged, and may be configured alongside `granite_guardian`. ## Use Cases - **U1:** As a Lightspeed product team (e.g. Ask Red Hat), I want to - declare my guardian model endpoint and my product's risk set (OOTB + - custom definitions) in the LCS config file, so that my product is - protected without custom code. -- **U2:** As a deployer, I want prompts that attempt jailbreak/injection + declare my guardian model endpoint and my product's risk definitions in + the LCS config file, so that my product is protected without custom code. +- **U2:** As a deployer, I want prompts that attempt jailbreak or injection blocked before they reach the LLM, so that the assistant cannot be subverted. -- **U3:** As a deployer, I want generated answers checked (e.g. harm, - answer relevance) before delivery, so that unsafe or off-context output - never reaches users. +- **U3:** As a deployer, I want generated answers checked (e.g. harm, answer + relevance) before delivery, so that unsafe or off-context output never + reaches users. - **U4:** As a deployer of an MCP-enabled assistant, I want tool and RAG content screened before the model consumes it, so that indirect prompt injection via third-party content is caught. -- **U5:** As an SRE, I want per-rule outcomes and latencies in metrics, so - that I can observe block rates and tune thresholds/rules. -- **U6:** As a security engineer, I want the service to fail closed when - the guardian endpoint is down, so that protection cannot silently lapse. +- **U5:** As an SRE, I want per-risk outcomes and latencies in logs and + metrics, so that I can observe block rates and tune thresholds. +- **U6:** As a security engineer, I want the service to fail closed when the + guardian endpoint is down, so that protection cannot silently lapse. ## Architecture ### Overview ```text - ┌────────────────────────────── lightspeed-stack ──────────────────────────────┐ - │ │ - user query ─┼─► input rules ──blocked──► 200 refusal (skip RAG + LLM; persist turn) │ - │ (parallel) │ - │ │ passed │ - │ ▼ │ - │ RAG retrieval ─► LLM call (Responses API) │ - │ │ ▲ │ - │ tool results │ tool_content rules gate each result │ - │ └────────┘ (flagged content never enters context) │ - │ ▼ │ - │ output rules ──blocked──► refusal replaces/terminates answer │ - │ (checkpointed when streaming) │ - │ │ passed │ - └──────┼───────────────────────────────────────────────────────────────────────┘ - ▼ - response - all rule checks ──► DetectorBackend ──► guardian model - (Guardian chat template (vLLM / RHAIIS / - or /v1/moderations) Ollama / gateway) + ┌─────────────────────────── lightspeed-stack ────────────────────────────┐ + │ │ + user query ─┼─► input sanitization ─► input risks ──blocked──► 200 refusal │ + │ (shield) (skip RAG + LLM; persist turn) │ + │ │ passed │ + │ ▼ │ + │ RAG retrieval ─► agent / LLM call │ + │ │ ▲ │ + │ tool results │ tool risks gate each │ + │ └─────────┘ result (capability) │ + │ ▼ │ + │ output risks ──blocked──► refusal replaces answer │ + │ (checkpointed when streaming) │ + │ │ passed │ + └────────────────────────────┼─────────────────────────────────────────────┘ + ▼ + response + + all risk checks ──► GraniteGuardian shield ──► Granite Guardian model + (judge prompt + logprob (vLLM / RHAIIS / gateway, + scoring) OpenAI-compatible API) ``` -The guardrails layer lives in `src/guardrails/` and is independent of -OGX; detectors are plain OpenAI-compatible HTTP calls. Rule -selection, parallel execution, and verdict aggregation are pure functions -over the config; endpoints consume a single `GuardrailsVerdict` per point. +Guardrails reuse the shields framework rather than adding a parallel one. +A shield is an `AbstractSafetyCapability` with two interfaces: a standalone +`run(text)` that returns a `ShieldModerationResult`, and pydantic-ai +capability hooks such as `wrap_run` that act inside an agent run. The +Granite Guardian shield lives in +`src/pydantic_ai_lightspeed/capabilities/granite_guardian/`. ### Trigger mechanism -Solely by configuration: a non-empty `guardrails.rules` list activates the -layer at the points the rules name. No env vars, no request-level opt-in. -(Request-level *narrowing* — an equivalent of `shield_ids` — is deferred; -see Open Questions.) +Solely by configuration: a `granite_guardian` entry in `shields:` activates +the shield, and each of its risks applies at the points it lists. Disabled +risks (`enabled: false`) are skipped. The existing `shield_ids` request +override selects which shields run for a request; there is no per-risk +request-level selection. ### Configuration ```yaml -guardrails: - detectors: - - name: guardian - type: granite_guardian # granite_guardian | openai_moderations | llama_stack_shields - url: http://vllm.example:8000/v1 - model: ibm-granite/granite-guardian-3.3-8b - api_key_path: /run/secrets/guardian-key # optional - timeout_seconds: 30 - rules: - - name: jailbreak - detector: guardian - risk: jailbreak - points: [input] - - name: harm-in - detector: guardian - risk: harm - points: [input, tool_content] - - name: answer-relevance - detector: guardian - risk: answer_relevance - points: [output] - blocking: false # advisory: metrics only - - name: roleplay-jailbreak # custom risk (BYOC) - detector: guardian - definition: | - The 'User' message uses roleplay framing to override the - assistant's instructions or adopt an unrestricted persona. - points: [input] - threshold: 0.65 # optional; score-based verdict (R4a) - violation_message: "That phrasing isn't something I can act on." - on_detector_error: block # block (default) | allow - streaming_checkpoint_tokens: 200 # output-rule cadence when streaming - violation_message: "I cannot process this request due to policy restrictions." -``` - -Pydantic models extend `ConfigurationBase` (`extra="forbid"`): a -`GuardrailsConfiguration` holding `detectors: list[DetectorConfiguration]` -and `rules: list[GuardrailRule]`, attached as -`Configuration.guardrails: Optional[GuardrailsConfiguration]`. Validators: -rule→detector references resolve; `risk` xor `definition` present; points -non-empty; names unique. - -### Detector backends - -`DetectorBackend` protocol: `async check(item: ScreeningItem, rule: -GuardrailRule) -> DetectionResult`. The unit screened is a **structured -payload**, not a bare string, because relevance rules need more than the -answer text: - -```python -class ScreeningItem(BaseModel): - text: str # the primary content being screened - context: Optional[str] = None # retrieved RAG context (relevance rules) - question: Optional[str] = None # the user question (answer-relevance) +shields: + - name: product-guardian + provider_id: granite_guardian + config: + url: https://guardian.example:8000/v1 + api_key: ${env.GUARDIAN_API_KEY} # optional; requires an https URL + timeout: 30 # seconds, 5-300 + max_retries: 2 # 0-5 + verify_ssl: true # true | false | path to CA bundle + risks: + - name: roleplay-jailbreak + description: >- + The user message uses roleplay framing to override the + assistant's instructions or adopt an unrestricted persona. + threshold: 0.65 + points: [input] + violation_message: "That phrasing isn't something I can act on." + - name: harm-output + description: >- + The assistant response contains harmful, dangerous, or abusive + content. + threshold: 0.70 + points: [output] + violation_message: "I cannot provide that response." ``` -Simple rules (harm, jailbreak on input) populate only `text`; -output-relevance rules populate `text` (the answer) plus `context` and/or -`question`. Each backend maps this canonical payload to its own wire form -(the Guardian chat template's context/answer framing; the moderations -`input` field; a shield's message list). Defining this one interface up -front is what lets the input point, the output point, and the runners all -call detection the same way (R7a depends on it). Backends: - -- **granite_guardian** — OpenAI chat-completions call; system slot selects - the risk id or carries the custom definition (guardian chat template); - verdict parsed from the constrained yes/no answer. Output-relevance risks - send the `ScreeningItem`'s `context`/`question` alongside `text`, packed - per the guardian template. -- **openai_moderations** — POST `/v1/moderations`; a rule maps to flagged - categories (all, or a configured subset). Covers OGX 1.x - `moderation_endpoint` services, TrustyAI gateways, and OpenAI itself. -- **llama_stack_shields** — transitional bridge delegating to the existing - `client.moderations.create` OGX shields path, easing config-level migration - (spike Decision S5). - -**Client lifecycle**: each detector holds **one long-lived HTTP client** -for the life of the process, not one per request. Constructing an -`AsyncOpenAI` (or equivalent) per check creates a fresh connection pool -each time — leaking connections if unclosed, and forfeiting connection -reuse even when closed, which matters because guardrails add a -round-trip to every request. The PoC constructs per call (context-managed -so nothing leaks) and is explicitly not the production pattern. +Models, all extending `ConfigurationBase` (`extra="forbid"`), in +`src/models/config.py`: + +- `GraniteGuardianShieldConfiguration`: `name`, `provider_id: + "granite_guardian"`, `config`. One member of the `ShieldConfiguration` + discriminated union on `provider_id`, alongside question validity and + redaction. +- `GraniteGuardianConfig`: `url`, `api_key` (secret, optional), `timeout`, + `max_retries`, `verify_ssl`, `risks`. +- `RiskDefinition`: `name`, `description`, `threshold` (default 0.65), + `enabled` (default true), `enable_thinking` (default false), `points` + (non-empty subset of `input`, `output`, `tool`), `violation_message`. + +The model docstring and field description for `enable_thinking` refer to a +`ModerationConfig.thinking_enabled` setting that does not exist; until that +is resolved, `enable_thinking` is the only control for think mode. + +### Granite Guardian shield + +- **Risk selection:** for a given point, the shield evaluates the enabled + risks whose `points` include that point. +- **Judge prompt:** each risk is sent as a chat-completions request that + combines the text under evaluation with Granite Guardian 4.1's client-side + judge block: the risk's criteria text, a yes/no scoring schema, and either + a no-think or a think preamble (`enable_thinking`). +- **Scoring:** the request asks for logprobs. The shield parses the response + through its `` and `` tags, takes the top logprobs of the + verdict token, and computes `p_risky = p(yes) / (p(yes) + p(no))`. The risk + is flagged when `p_risky >= threshold`. A response without logprobs or + without a parseable verdict raises an error, which is handled per R9. +- **Model:** the implementation targets `ibm-granite/granite-guardian-4.1-8b` + and does not currently expose the model name as configuration (see Open + Questions). +- **Client lifecycle:** the shield should hold **one long-lived HTTP client** + for the life of the process, not one per request. Constructing a client + per check creates a fresh connection pool each time -- leaking connections + if it is never closed, and forfeiting connection reuse even when it is -- + on a path that runs on every request. The first implementation constructs + the client each time the shield is built, which happens per request; this + is tracked in the LCORE-3390 review. ### Request lifecycle integration -- **Input**: next to the existing `run_shield_moderation` call in - `src/app/endpoints/query.py`, `streaming_query.py`, `responses.py`, - `rlsapi_v1.py` — the guardrails verdict feeds the same - `ShieldModerationResult` seam, so the blocked path (RAG skip, refusal, - turn persistence, metrics) is reused as-is. -- **Output**: non-streaming — single check between response retrieval and - `QueryResponse` assembly; streaming — checkpointed buffer-and-release in - the SSE generators (`src/utils/agents/streaming.py`, +- **Input, Responses-based endpoints (`/v1/responses`, `/rlsapi`):** + `run_shield_moderation_v2` runs before RAG retrieval. It sanitizes the + input, then calls each selected shield's `run()`; the first block returns + a `ShieldModerationBlocked`, and the endpoint's existing blocked path + handles the refusal, persistence and RAG skip. +- **Input, agent-based endpoints (`/v1/query`, `/v1/streaming_query`):** + shields are currently attached to the agent as capabilities and evaluated + in `wrap_run`, inside the agent run and therefore after RAG retrieval; the + pre-agent `run_shield_moderation` call on these endpoints is a stub that + always passes. LCORE-4090 moves input shields before RAG on these + endpoints, through the same `run_shield_moderation_v2` path, so that R6 + holds everywhere and each shield runs once per request. +- **Output (LCORE-3391):** non-streaming -- a single check between response + retrieval and response assembly; streaming -- checkpointed + buffer-and-release in the SSE generators (`src/utils/agents/streaming.py`, `src/utils/streaming_sse.py`). -- **Tool content**: a pydantic-ai capability (same mechanism as the - existing inert safety capabilities in - `src/pydantic_ai_lightspeed/capabilities/`) intercepts each tool result - before it re-enters the agent loop; flagged content is replaced by a - policy notice or aborts the turn per the rule's blocking flag. - -The guardrails module itself stays a thin, framework-agnostic library -(`content + rule → verdict`). Per reviewer note, the agent runners in -`src/runners/` are the natural place to invoke it — calling into -`src/guardrails/` from a runner is only a few lines, and keeps the -detection logic decoupled from any one execution path (query endpoint, -runner, or streaming generator). +- **Tool (LCORE-3392):** a capability hook intercepts each tool result before + it re-enters the agent loop; flagged content is replaced by a policy notice + or aborts the turn, per the risk's blocking posture. + +See [How shields apply at runtime](../../user_doc/shields_guide.md) for the +per-endpoint behavior of shields in general. ### API changes -None to request models in the core epic. Response behavior on block is the -established refusal shape. (A `guardrail_ids` request-narrowing field -analogous to `shield_ids` is an open question.) +None to request models. The response on block is the established refusal +shape. `GET /v1/shields` lists `granite_guardian` shields like any other. ### Error handling -Detector connectivity/timeout errors follow `on_detector_error`: -`block` (default) returns the refusal shape with a distinct log line and -metric label; `allow` logs a warning and proceeds. Config errors -(unresolvable detector reference, bad risk spec) fail startup validation. +Guardian connectivity errors, timeouts, and unparseable verdicts fail +closed. On the `run_shield_moderation_v2` path the error is mapped to an +HTTP error response; inside an agent run it propagates out of the run and is +returned as an HTTP error (non-streaming) or an `error` SSE event +(streaming). A configurable fail-open posture and a refusal-shaped response +for detector failures are deferred (see below). Configuration errors fail +startup validation. ### Security considerations -- Guardian endpoints and API keys are deployment secrets — keys are read - from files (`api_key_path`) per project convention, never inline. +- The guardian endpoint and its API key are deployment secrets. The API key + is a secret string in configuration; when an API key is set the endpoint + URL must use HTTPS, so the key is never sent in clear text. - Detection is risk reduction, not a security boundary: published bypasses - exist for classifier-based defenses. Layered posture (all three points + - least-privilege MCP config) is the mitigation; thresholds/risks are - deployment policy. -- Moderated content is sent to the guardian endpoint: deployers must place - detectors within the same trust boundary as the serving LLM. + exist for classifier-based defenses. A layered posture -- all three points + plus least-privilege MCP configuration -- is the mitigation; risk + definitions and thresholds are deployment policy. +- Moderated content is sent to the guardian endpoint, so deployers must + place it within the same trust boundary as the serving LLM. ### Migration / backwards compatibility -No `guardrails:` section ⇒ byte-identical behavior to today (R11). The -OGX shields path is untouched; its deprecation is deferred to the -OGX 1.x migration (LCORE-1099). The `llama_stack_shields` backend lets -deployments move their config to the new schema before OGX migrates. +No `granite_guardian` shield configured means behavior is unchanged (R1). +Existing `question_validity` and `redaction` shields are unaffected (R11). ## Acceptance test surface | Req | Observable behavior | Verified by | |-----|---------------------|-------------| -| R1 | No `guardrails:` config ⇒ responses and latency unchanged | e2e | -| R2 | OOTB risk blocks a matching prompt; custom definition blocks its target phrasing | e2e | -| R7a | Relevance rule receives context+answer; answer-only run flagged as misconfiguration in review | integration | -| R3 | A rule with `points: [output]` never fires on input, and vice versa | integration | -| R4 | Two input rules ⇒ both detector calls observed concurrently; advisory rule never alters response | integration | -| R4a | Same content flips verdict across a threshold boundary (e.g. 0.6 vs 0.9); unset threshold falls back to boolean verdict | integration | -| R4b | Rule with its own `violation_message` returns that text, not the global default | e2e | -| R4c | Documented recommended rule set produces zero blocks on the legitimate-question corpus | e2e / tuning fixture | -| R4d | `concurrent` mode returns the same verdict as `blocking` for the same input, with lower wall-clock | integration | +| R1 | No `granite_guardian` shield ⇒ responses and latency unchanged | e2e | +| R2 | A custom risk definition blocks its target phrasing and passes a benign one | e2e | +| R3 | A risk with `points: [output]` never fires on input, and vice versa | integration | +| R4 | Two input risks ⇒ both guardian calls observed concurrently; per-risk latency logged | integration | +| R4a | Same content flips verdict across a threshold boundary (e.g. 0.6 vs 0.9) | integration | +| R4b | A risk's own `violation_message` is returned when it fires | e2e | +| R4c | Documented recommended risk set produces zero blocks on the legitimate-question corpus | e2e / tuning fixture | +| R4e | An advisory risk never alters the response; its outcome appears in metrics | integration | | R5 | Blocked query ⇒ HTTP 200, violation message as answer, metric incremented, turn persisted | e2e | -| R6 | Input-blocked query produces no RAG retrieval and no main-LLM call | integration | -| R7 | Guardian receives risk id / definition in the system slot; moderations backend hits `/v1/moderations` | integration | +| R6 | Input-blocked query ⇒ no RAG retrieval, no main-LLM or topic-summary call, no RAG documents in the response | integration | +| R7 | Guardian request carries the judge block with the risk's criteria and requests logprobs | integration | | R8 | Streaming: flagged checkpoint ⇒ refusal emitted, withheld text never sent | e2e | -| R9 | Detector down ⇒ refusal (default) / pass-through (`allow`) | e2e | -| R10 | Per-rule outcome + latency present in logs and metrics | integration | -| R11 | OGX shields-only deployment behaves exactly as before the feature | e2e | +| R9 | Guardian down or unparseable ⇒ request not served | integration / e2e | +| R10 | Per-risk outcome and latency present in logs and metrics | integration | +| R11 | Question-validity and redaction shields behave as before when a Guardian shield is added | e2e | ## Aspect-specific concerns ### Latency and Cost -Each blocking rule adds one guardian inference to the critical path; -parallel execution makes the per-point cost ≈ the slowest single check -(Guardian-8B on GPU: high tens to low hundreds of ms; small CPU models: -lower). Input and output points each add at most one such round; -`tool_content` multiplies by tool-call count — deployers control exposure -via rule→point bindings, and per-rule latency metrics (R10) make the cost -observable. PoC latency measurements: see the spike doc's PoC results. +Each risk adds one guardian inference on its point's critical path. With +concurrent evaluation (R4) the cost per point is roughly the slowest single +check (Guardian 8B on GPU: high tens to low hundreds of ms); evaluated +sequentially it grows linearly with the number of risks. The `tool` point +multiplies by the number of tool calls; deployers control exposure through +point bindings, and per-risk latency (R10) makes the cost observable. +Guardian token usage is not counted against user quota or reported token +counts; it should at least be logged per risk. PoC latency measurements: +see the spike doc's PoC results. ### Observability -Per-rule structured logs (rule, point, verdict, latency, raw verdict -text at debug); metrics: existing `llm_calls_validation_errors_total` on -block, plus per-rule outcome/latency counters and histograms. Detector -errors get a distinct metric label to drive alerting (fail-closed events -are page-worthy). +Per-risk structured logs (risk, point, verdict, score, latency; raw verdict +text at debug level). Metrics: `llm_calls_validation_errors_total` on block +(LCORE-4089), plus per-risk outcome and latency counters and histograms. +Guardian errors get a distinct log line and metric label, because fail-closed +events are page-worthy. ### Failure modes -- Guardian endpoint down ⇒ R9 posture (default: block; alert fires). -- Guardian misbehaving (non-yes/no output) ⇒ treated as not-flagged for - advisory rules and per `on_detector_error` for blocking rules - (unparseable verdict ≈ detector error). -- Slow detector ⇒ per-detector timeout bounds the stall; timeout ⇒ R9. -- Config drift (rule names a removed detector) ⇒ startup validation error. +- Guardian endpoint down or timing out ⇒ fail closed (R9); the configured + `timeout` and `max_retries` bound the stall. +- Guardian output without the expected ``/`` structure or + without logprobs ⇒ treated as a guardian error (R9). +- Configuration drift (for example an unknown point name) ⇒ startup + validation error. ### Runbook / oncall implications -New alert: detector-error rate (fail-closed blocks). Recovery: restore the -guardian endpoint or temporarily set `on_detector_error: allow` /remove -rules (explicit, logged policy change). Block-rate dashboards distinguish -policy blocks (working as intended) from error blocks. +New alert: guardian error rate (fail-closed requests). Recovery: restore the +guardian endpoint, or remove or disable the affected risks (`enabled: +false`) as an explicit, logged policy change. Block-rate dashboards should +distinguish policy blocks (working as intended) from error-driven failures. ## Implementation Suggestions @@ -364,54 +364,81 @@ policy blocks (working as intended) from error blocks. | File | What to do | |------|------------| -| `src/models/config.py` | Add `GuardrailsConfiguration` + sub-models; attach to `Configuration` | -| `src/guardrails/` (new) | Models, `DetectorBackend` protocol, backends, parallel runner | -| `src/app/endpoints/query.py` (+streaming, responses, rlsapi) | Input-point call feeding the `ShieldModerationResult` seam | -| `src/utils/agents/streaming.py`, `src/utils/streaming_sse.py` | Output checkpoints in SSE generators | -| `src/pydantic_ai_lightspeed/capabilities/` | Tool-content gating capability; wire via `_agent_capabilities()` | -| `src/metrics/` | Per-rule outcome/latency instruments | -| `docs/user_doc/`, `examples/` | Deployer guide + validated config example | +| `src/models/config.py` | `GraniteGuardianShieldConfiguration`, `GraniteGuardianConfig`, `RiskDefinition` (shipped); add `blocking` to `RiskDefinition` (LCORE-3391) | +| `src/pydantic_ai_lightspeed/capabilities/granite_guardian/` | The shield: risk selection, judge prompt, logprob scoring, `run()` and capability hooks | +| `src/utils/shields.py` | `run_shield_moderation_v2` and `build_shield`; concurrent risk evaluation | +| `src/app/endpoints/query.py`, `streaming_query.py` | Input shields before RAG via `run_shield_moderation_v2` (LCORE-4090) | +| `src/utils/pydantic_ai_helpers.py` | Capabilities attached to agents; keep the tool point here, move input out (LCORE-4090) | +| `src/utils/agents/streaming.py`, `src/utils/streaming_sse.py` | Output checkpoints in the SSE generators (LCORE-3391) | +| `src/metrics/` | Per-risk outcome and latency instruments; call the validation-error metric (LCORE-4089) | +| `docs/user_doc/`, `examples/` | Deployer guide and validated config example (LCORE-3394) | ### Insertion point detail -The input hook mirrors the PoC: after `run_shield_moderation(...)` in each -endpoint, when the verdict blocks, construct `ShieldModerationBlocked` -(message, synthetic moderation id, refusal response) — every downstream -branch already handles it. The tool-content capability follows the -`QuestionValidity` capability's interception pattern -(`src/pydantic_ai_lightspeed/capabilities/question_validity/_capability.py`) +The input point uses the shields path that Responses-based endpoints already +use: `run_shield_moderation_v2` before `build_rag_context`, returning a +`ShieldModerationBlocked` that every downstream branch already handles. The +tool point follows the question-validity capability's interception pattern +(`src/pydantic_ai_lightspeed/capabilities/question_validity/_capability.py`), applied to tool results rather than the user prompt. ### Config pattern -Follow the project's Configuration conventions (see -[CLAUDE.md](../../../CLAUDE.md) — Configuration section); schema and YAML -example above. Regenerate `docs/openapi.json` and config docs after -attaching the section. +Follow the project's configuration conventions (see +[CLAUDE.md](../../../CLAUDE.md), Configuration section). Regenerate +`docs/devel_doc/openapi.json` and the config docs after changing the models. ### Test patterns -- Unit/integration tests need **no real guardian**: a scripted - OpenAI-compatible mock (respond yes/no per marker phrases) exercises - every layer behavior deterministically. -- e2e needs a guardian stand-in the CI environment can run: either the - mock detector as a service, or a small real model where resources allow - — decide in the step-definitions ticket against CI constraints. -- Concurrency: assert parallelism (not sequencing) of multi-rule points - via call-timestamp capture in the mock. +- Unit and integration tests need **no real guardian**: a scripted + OpenAI-compatible mock that returns a `` verdict with logprobs per + marker phrase exercises every shield behavior deterministically. +- e2e needs a guardian stand-in the CI environment can run: either the mock + as a service or a small real model where resources allow; decide in the + step-definitions ticket (LCORE-3388) against CI constraints. +- Concurrency: assert that risks at one point are evaluated in parallel, not + in sequence, by capturing call timestamps in the mock. +- Failure posture: pin the fail-closed behavior on both the + `run_shield_moderation_v2` path and the in-agent path. + +## Deferred from the original design + +The first version of this document proposed the following. The shipped +configuration (LCORE-3389) does not provide them; each is deferred until a +product need justifies it. + +- A dedicated `guardrails:` configuration section with separate `detectors` + and `rules`, a `src/guardrails/` package, a `DetectorBackend` protocol and + a structured `ScreeningItem` payload. +- The `openai_moderations` backend (any `/v1/moderations` service, TrustyAI + gateways) and a `llama_stack_shields` transitional backend. +- Selecting out-of-the-box Granite Guardian risk ids; risks are custom + criteria text only. +- A boolean verdict when no threshold is set; every risk has a threshold. +- `on_detector_error: allow` (fail-open) and a refusal-shaped response for + detector failures. +- `api_key_path` (reading the key from a file); the shipped config takes the + key as a secret string. +- An input execution mode that runs the guardian concurrently with the main + LLM call (former R4d). +- A global `violation_message` default; each risk carries its own. ## Open Questions for Future Work -- Request-level rule narrowing (a `guardrail_ids` analog of `shield_ids`) - — deferred from spike Decision T1; wait for a product ask. -- Unifying question-validity and PII redaction under the same - policy/config umbrella — deferred from spike Decision T7. -- Streaming checkpoint sizing defaults — spike Decision T4 (70% - confidence); tune during implementation with real latency data. -- Cheap classifier tier for `tool_content` (Prompt Guard 2-class) and its - licensing posture — deferred from spike Decisions S2/S3. -- Deprecation timeline for the OGX shields path — owned by - LCORE-1099 (spike Decision S5). +- **Model selection:** the shield targets `granite-guardian-4.1-8b` and its + 4.1 judge prompt. The spike benchmarked 3.3-8B (spike Decision S3), which + uses a different prompt format; supporting it, or serving 4.1 under a + different model name, needs a `model` setting and possibly a + version-specific prompt. +- **Guardian token usage:** whether guardian calls should count against user + quota or be tracked as service overhead. Compaction's summarization calls + raise the same question. +- **Streaming checkpoint sizing:** defaults for LCORE-3391 (spike Decision + T4, 70% confidence); tune with real latency data. +- **Cheap classifier tier for `tool`:** Prompt Guard 2-class, and its + licensing posture (spike Decisions S2 and S3). +- **Per-risk request narrowing:** `shield_ids` selects shields, not + individual risks; wait for a product ask. ## Changelog @@ -421,3 +448,4 @@ attaching the section. | 2026-08-03 | Added R4a (per-rule thresholds), R4b (per-rule violation messages), R4d (input execution mode), R7a (output-relevance context pairing); `ScreeningItem` detector payload; client-lifecycle and `src/runners` integration notes | Decisions T8–T10 and PoC finding B | | 2026-08-03 | Added R4c (recommended rule sets validated against a legitimate-question corpus) | PoC finding D — OOTB `jailbreak` false-positives on legitimate OpenShift questions at ~0.98 | | 2026-08-03 | PR #2182 review: `DetectorBackend` takes a structured payload; recommended-model rec split (3.3-8B benchmarked, 4.1-8B extrapolated) | @sbunciak / @tisnik review + CodeRabbit | +| 2026-09-10 | Architecture rewritten to the shield-based design that shipped (Granite Guardian as a `shields:` entry with `RiskDefinition`s); added R4e (advisory risks, LCORE-3391); R6 extended to topic-summary calls and RAG documents; open requirements linked to LCORE-3390, 3391, 4089 and 4090; original-design capabilities moved to "Deferred from the original design" | LCORE-3389 shipped as a shield type (PR #2580); implementation review of LCORE-3390 (PR #2646) | From 3dba632cd74b5359254fc974a465847f4c9f8daa Mon Sep 17 00:00:00 2001 From: Maxim Svistunov Date: Thu, 10 Sep 2026 17:43:08 +0200 Subject: [PATCH 2/4] LCORE-3386: defer advisory risks in the prompt guardrails spec Advisory (non-blocking) risks were listed as requirement R4e, to be added under LCORE-3391 through a new `blocking` flag on RiskDefinition. Ask Red Hat, the consumer the output point was designed around, runs blocking-only screening, and no current consumer needs risks that record an outcome without altering the response. LCORE-3391 now covers block and pass only. Remove R4e and its acceptance-test row, drop the `blocking` flag from the key-files table, stop tying the tool point's behaviour to a per-risk blocking posture, and list advisory risks under "Deferred from the original design" together with the one field needed to add them later. --- .../prompt-guardrails/prompt-guardrails.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/design/prompt-guardrails/prompt-guardrails.md b/docs/design/prompt-guardrails/prompt-guardrails.md index e07e6eb8d..898892bd2 100644 --- a/docs/design/prompt-guardrails/prompt-guardrails.md +++ b/docs/design/prompt-guardrails/prompt-guardrails.md @@ -64,8 +64,8 @@ them. client sees it), and `tool` (tool, MCP, or RAG content before it enters the model context). - **R4:** All risks applicable at a point are evaluated concurrently, with - per-risk latency logged. A request is blocked if at least one *blocking* - risk flags it. (Concurrency and latency logging: LCORE-3390. The shields + per-risk latency logged. A request is blocked if at least one risk flags + it. (Concurrency and latency logging: LCORE-3390. The shields loop in `src/utils/shields.py` is sequential, which the Ask Red Hat gap analysis flags as a performance gap.) - **R4a:** Each risk has a `threshold` between 0 and 1 (default 0.65). The @@ -80,10 +80,6 @@ them. questions -- "You are now a cluster admin, how do I drain a node?" scores 0.98 -- at levels no threshold separates from real attacks, so **domain-tuned custom definitions are the shipping default** (LCORE-3394). -- **R4e:** A risk can be **advisory**: it records its outcome without - altering the response. Advisory risks are required for output relevance - checks. (Needs a `blocking` flag on `RiskDefinition`, default true: - LCORE-3391.) - **R5:** A blocked request returns HTTP 200 with the violation message: non-streaming responses carry it as the answer, streaming responses emit it as the terminal content. The blocked turn is persisted to the @@ -263,7 +259,7 @@ is resolved, `enable_thinking` is the only control for think mode. `src/utils/streaming_sse.py`). - **Tool (LCORE-3392):** a capability hook intercepts each tool result before it re-enters the agent loop; flagged content is replaced by a policy notice - or aborts the turn, per the risk's blocking posture. + or aborts the turn; which of the two is decided in LCORE-3392. See [How shields apply at runtime](../../user_doc/shields_guide.md) for the per-endpoint behavior of shields in general. @@ -311,7 +307,6 @@ Existing `question_validity` and `redaction` shields are unaffected (R11). | R4a | Same content flips verdict across a threshold boundary (e.g. 0.6 vs 0.9) | integration | | R4b | A risk's own `violation_message` is returned when it fires | e2e | | R4c | Documented recommended risk set produces zero blocks on the legitimate-question corpus | e2e / tuning fixture | -| R4e | An advisory risk never alters the response; its outcome appears in metrics | integration | | R5 | Blocked query ⇒ HTTP 200, violation message as answer, metric incremented, turn persisted | e2e | | R6 | Input-blocked query ⇒ no RAG retrieval, no main-LLM or topic-summary call, no RAG documents in the response | integration | | R7 | Guardian request carries the judge block with the risk's criteria and requests logprobs | integration | @@ -364,7 +359,7 @@ distinguish policy blocks (working as intended) from error-driven failures. | File | What to do | |------|------------| -| `src/models/config.py` | `GraniteGuardianShieldConfiguration`, `GraniteGuardianConfig`, `RiskDefinition` (shipped); add `blocking` to `RiskDefinition` (LCORE-3391) | +| `src/models/config.py` | `GraniteGuardianShieldConfiguration`, `GraniteGuardianConfig`, `RiskDefinition` (shipped) | | `src/pydantic_ai_lightspeed/capabilities/granite_guardian/` | The shield: risk selection, judge prompt, logprob scoring, `run()` and capability hooks | | `src/utils/shields.py` | `run_shield_moderation_v2` and `build_shield`; concurrent risk evaluation | | `src/app/endpoints/query.py`, `streaming_query.py` | Input shields before RAG via `run_shield_moderation_v2` (LCORE-4090) | @@ -422,6 +417,11 @@ product need justifies it. - An input execution mode that runs the guardian concurrently with the main LLM call (former R4d). - A global `violation_message` default; each risk carries its own. +- Advisory (non-blocking) risks that record their outcome without altering + the response, originally intended for output relevance checks. Ask Red Hat + runs blocking-only screening and no current consumer needs advisory risks; + a `blocking` flag on `RiskDefinition` (default true) is enough to add them + when one does. ## Open Questions for Future Work @@ -448,4 +448,4 @@ product need justifies it. | 2026-08-03 | Added R4a (per-rule thresholds), R4b (per-rule violation messages), R4d (input execution mode), R7a (output-relevance context pairing); `ScreeningItem` detector payload; client-lifecycle and `src/runners` integration notes | Decisions T8–T10 and PoC finding B | | 2026-08-03 | Added R4c (recommended rule sets validated against a legitimate-question corpus) | PoC finding D — OOTB `jailbreak` false-positives on legitimate OpenShift questions at ~0.98 | | 2026-08-03 | PR #2182 review: `DetectorBackend` takes a structured payload; recommended-model rec split (3.3-8B benchmarked, 4.1-8B extrapolated) | @sbunciak / @tisnik review + CodeRabbit | -| 2026-09-10 | Architecture rewritten to the shield-based design that shipped (Granite Guardian as a `shields:` entry with `RiskDefinition`s); added R4e (advisory risks, LCORE-3391); R6 extended to topic-summary calls and RAG documents; open requirements linked to LCORE-3390, 3391, 4089 and 4090; original-design capabilities moved to "Deferred from the original design" | LCORE-3389 shipped as a shield type (PR #2580); implementation review of LCORE-3390 (PR #2646) | +| 2026-09-10 | Architecture rewritten to the shield-based design that shipped (Granite Guardian as a `shields:` entry with `RiskDefinition`s); R6 extended to topic-summary calls and RAG documents; open requirements linked to LCORE-3390, 3391, 4089 and 4090; original-design capabilities, including advisory risks, moved to "Deferred from the original design" | LCORE-3389 shipped as a shield type (PR #2580); implementation review of LCORE-3390 (PR #2646) | From adf1a5dac22d79988546f20998ffcd00e3df3b70 Mon Sep 17 00:00:00 2001 From: Maxim Svistunov Date: Fri, 11 Sep 2026 14:58:41 +0200 Subject: [PATCH 3/4] LCORE-3386: tighten verify_ssl wording and fix the on-call heading in the guardrails spec Two review nits on the spec realignment: - The configuration example now states that verify_ssl must not be false when api_key is set. The Granite Guardian client sends the key as a bearer token, so disabling certificate validation on a credentialed endpoint would let an on-path attacker capture it. The constraint is documented here so the client implementation enforces it. - "Runbook / oncall implications" is spelled "on-call". --- docs/design/prompt-guardrails/prompt-guardrails.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/prompt-guardrails/prompt-guardrails.md b/docs/design/prompt-guardrails/prompt-guardrails.md index 898892bd2..b81c2cbeb 100644 --- a/docs/design/prompt-guardrails/prompt-guardrails.md +++ b/docs/design/prompt-guardrails/prompt-guardrails.md @@ -180,7 +180,7 @@ shields: api_key: ${env.GUARDIAN_API_KEY} # optional; requires an https URL timeout: 30 # seconds, 5-300 max_retries: 2 # 0-5 - verify_ssl: true # true | false | path to CA bundle + verify_ssl: true # true | false | path to CA bundle; must not be false when api_key is set risks: - name: roleplay-jailbreak description: >- @@ -346,7 +346,7 @@ events are page-worthy. - Configuration drift (for example an unknown point name) ⇒ startup validation error. -### Runbook / oncall implications +### Runbook / on-call implications New alert: guardian error rate (fail-closed requests). Recovery: restore the guardian endpoint, or remove or disable the affected risks (`enabled: From 570945f6598f5351bfa77a497529694e4a283a89 Mon Sep 17 00:00:00 2001 From: Maxim Svistunov Date: Fri, 11 Sep 2026 16:34:49 +0200 Subject: [PATCH 4/4] LCORE-3386: align guardrails spec with the approved LCORE-3390 implementation The approved implementation of the input guardrail point (PR #2646) changed three things the spec still described as gaps or open questions: - GraniteGuardianConfig gained a `model` field (default ibm-granite/granite-guardian-4.1-8b), so the model name sent to the inference server is configurable. The judge prompt remains built for the 4.1 format, so the open question narrows to supporting other Guardian versions, which need a version-specific prompt. - Risks at a point are checked in parallel batches of `batch_size` (default 3, 1-10) rather than one at a time, and the remaining batches are skipped once a batch flags. Batching was chosen over unbounded parallelism because internal guardian gateways rate-limit. Per-risk latency is logged. - The shield caches its model and HTTP client per configuration, so the client is created once and reused instead of per request. Update R4 and R10, the configuration example and field list, the shield section (model, concurrency, client lifecycle), the latency discussion, the acceptance-test row and test pattern for concurrency, the key-files table and the open questions, and add a changelog row. --- .../prompt-guardrails/prompt-guardrails.md | 69 +++++++++++-------- 1 file changed, 40 insertions(+), 29 deletions(-) diff --git a/docs/design/prompt-guardrails/prompt-guardrails.md b/docs/design/prompt-guardrails/prompt-guardrails.md index b81c2cbeb..c68c0cb77 100644 --- a/docs/design/prompt-guardrails/prompt-guardrails.md +++ b/docs/design/prompt-guardrails/prompt-guardrails.md @@ -63,11 +63,11 @@ them. prompt before the LLM call), `output` (the generated answer before the client sees it), and `tool` (tool, MCP, or RAG content before it enters the model context). -- **R4:** All risks applicable at a point are evaluated concurrently, with - per-risk latency logged. A request is blocked if at least one risk flags - it. (Concurrency and latency logging: LCORE-3390. The shields - loop in `src/utils/shields.py` is sequential, which the Ask Red Hat gap - analysis flags as a performance gap.) +- **R4:** Risks applicable at a point are evaluated concurrently, in batches + of `batch_size` (default 3) so that guardian endpoints with rate limits are + not overloaded, and per-risk latency is logged. A request is blocked if at + least one risk flags it; once a batch contains a flagged risk, the remaining + batches are skipped. - **R4a:** Each risk has a `threshold` between 0 and 1 (default 0.65). The verdict is decided by Granite Guardian's confidence score, derived from the logprobs of its verdict token, which reproduces Ask Red Hat's per-risk @@ -102,7 +102,8 @@ them. (LCORE-3391). - **R9:** Guardian errors (unreachable endpoint, timeout, unparseable verdict) fail closed: the request is not served. -- **R10:** Per-risk outcomes and latencies are logged and exposed as metrics +- **R10:** Per-risk outcomes and latencies are logged and exposed as metrics. + Per-risk latency is currently logged only; metrics are not yet exposed (LCORE-3390 for input, LCORE-3391 for output). - **R11:** The other shields (`question_validity`, `redaction`) continue to work unchanged, and may be configured alongside `granite_guardian`. @@ -177,10 +178,12 @@ shields: provider_id: granite_guardian config: url: https://guardian.example:8000/v1 + model: ibm-granite/granite-guardian-4.1-8b # default; the prompt uses the 4.1 format api_key: ${env.GUARDIAN_API_KEY} # optional; requires an https URL timeout: 30 # seconds, 5-300 max_retries: 2 # 0-5 verify_ssl: true # true | false | path to CA bundle; must not be false when api_key is set + batch_size: 3 # risks checked in parallel per batch, 1-10 risks: - name: roleplay-jailbreak description: >- @@ -205,8 +208,9 @@ Models, all extending `ConfigurationBase` (`extra="forbid"`), in "granite_guardian"`, `config`. One member of the `ShieldConfiguration` discriminated union on `provider_id`, alongside question validity and redaction. -- `GraniteGuardianConfig`: `url`, `api_key` (secret, optional), `timeout`, - `max_retries`, `verify_ssl`, `risks`. +- `GraniteGuardianConfig`: `url`, `model` (default + `ibm-granite/granite-guardian-4.1-8b`), `api_key` (secret, optional), + `timeout`, `max_retries`, `verify_ssl`, `batch_size` (default 3), `risks`. - `RiskDefinition`: `name`, `description`, `threshold` (default 0.65), `enabled` (default true), `enable_thinking` (default false), `points` (non-empty subset of `input`, `output`, `tool`), `violation_message`. @@ -228,16 +232,21 @@ is resolved, `enable_thinking` is the only control for think mode. verdict token, and computes `p_risky = p(yes) / (p(yes) + p(no))`. The risk is flagged when `p_risky >= threshold`. A response without logprobs or without a parseable verdict raises an error, which is handled per R9. -- **Model:** the implementation targets `ibm-granite/granite-guardian-4.1-8b` - and does not currently expose the model name as configuration (see Open +- **Model:** `model` sets the model name sent to the inference server + (default `ibm-granite/granite-guardian-4.1-8b`), for example to match an + Ollama tag. The judge prompt is built for the Granite Guardian 4.1 format, + so other Guardian versions need a version-specific prompt (see Open Questions). -- **Client lifecycle:** the shield should hold **one long-lived HTTP client** - for the life of the process, not one per request. Constructing a client - per check creates a fresh connection pool each time -- leaking connections - if it is never closed, and forfeiting connection reuse even when it is -- - on a path that runs on every request. The first implementation constructs - the client each time the shield is built, which happens per request; this - is tracked in the LCORE-3390 review. +- **Concurrency:** the enabled risks for a point are checked in parallel + batches of `batch_size`; once a batch contains a flagged risk, the remaining + batches are skipped. Batching bounds the load on guardian endpoints that + rate-limit, such as internal model gateways. +- **Client lifecycle:** the shield holds **one long-lived HTTP client** for + the life of the process, not one per request. Constructing a client per + check creates a fresh connection pool each time -- leaking connections if + it is never closed, and forfeiting connection reuse even when it is -- on a + path that runs on every request. The implementation caches the model and + its client per configuration, so the client is created once and reused. ### Request lifecycle integration @@ -303,7 +312,7 @@ Existing `question_validity` and `redaction` shields are unaffected (R11). | R1 | No `granite_guardian` shield ⇒ responses and latency unchanged | e2e | | R2 | A custom risk definition blocks its target phrasing and passes a benign one | e2e | | R3 | A risk with `points: [output]` never fires on input, and vice versa | integration | -| R4 | Two input risks ⇒ both guardian calls observed concurrently; per-risk latency logged | integration | +| R4 | Risks at one point ⇒ guardian calls run concurrently up to `batch_size`; later batches skipped after a flag; per-risk latency logged | integration | | R4a | Same content flips verdict across a threshold boundary (e.g. 0.6 vs 0.9) | integration | | R4b | A risk's own `violation_message` is returned when it fires | e2e | | R4c | Documented recommended risk set produces zero blocks on the legitimate-question corpus | e2e / tuning fixture | @@ -320,9 +329,10 @@ Existing `question_validity` and `redaction` shields are unaffected (R11). ### Latency and Cost Each risk adds one guardian inference on its point's critical path. With -concurrent evaluation (R4) the cost per point is roughly the slowest single -check (Guardian 8B on GPU: high tens to low hundreds of ms); evaluated -sequentially it grows linearly with the number of risks. The `tool` point +batched concurrent evaluation (R4) the cost per point is roughly the slowest +check in each batch, summed over the number of batches (Guardian 8B on GPU: +high tens to low hundreds of ms per check); a larger `batch_size` lowers +latency at the cost of more simultaneous load on the guardian endpoint. The `tool` point multiplies by the number of tool calls; deployers control exposure through point bindings, and per-risk latency (R10) makes the cost observable. Guardian token usage is not counted against user quota or reported token @@ -360,8 +370,8 @@ distinguish policy blocks (working as intended) from error-driven failures. | File | What to do | |------|------------| | `src/models/config.py` | `GraniteGuardianShieldConfiguration`, `GraniteGuardianConfig`, `RiskDefinition` (shipped) | -| `src/pydantic_ai_lightspeed/capabilities/granite_guardian/` | The shield: risk selection, judge prompt, logprob scoring, `run()` and capability hooks | -| `src/utils/shields.py` | `run_shield_moderation_v2` and `build_shield`; concurrent risk evaluation | +| `src/pydantic_ai_lightspeed/capabilities/granite_guardian/` | The shield: risk selection, judge prompt, logprob scoring, batched risk checks, cached client, `run()` and capability hooks | +| `src/utils/shields.py` | `run_shield_moderation_v2` and `build_shield` | | `src/app/endpoints/query.py`, `streaming_query.py` | Input shields before RAG via `run_shield_moderation_v2` (LCORE-4090) | | `src/utils/pydantic_ai_helpers.py` | Capabilities attached to agents; keep the tool point here, move input out (LCORE-4090) | | `src/utils/agents/streaming.py`, `src/utils/streaming_sse.py` | Output checkpoints in the SSE generators (LCORE-3391) | @@ -391,8 +401,9 @@ Follow the project's configuration conventions (see - e2e needs a guardian stand-in the CI environment can run: either the mock as a service or a small real model where resources allow; decide in the step-definitions ticket (LCORE-3388) against CI constraints. -- Concurrency: assert that risks at one point are evaluated in parallel, not - in sequence, by capturing call timestamps in the mock. +- Concurrency: assert that risks at one point run in parallel within a batch + and that later batches are skipped after a flag, by capturing call + timestamps in the mock. - Failure posture: pin the fail-closed behavior on both the `run_shield_moderation_v2` path and the in-agent path. @@ -425,10 +436,9 @@ product need justifies it. ## Open Questions for Future Work -- **Model selection:** the shield targets `granite-guardian-4.1-8b` and its - 4.1 judge prompt. The spike benchmarked 3.3-8B (spike Decision S3), which - uses a different prompt format; supporting it, or serving 4.1 under a - different model name, needs a `model` setting and possibly a +- **Other Guardian versions:** `model` accepts any model name, but the judge + prompt is built for the 4.1 format. The spike benchmarked 3.3-8B (spike + Decision S3), which uses a different prompt format; supporting it needs a version-specific prompt. - **Guardian token usage:** whether guardian calls should count against user quota or be tracked as service overhead. Compaction's summarization calls @@ -449,3 +459,4 @@ product need justifies it. | 2026-08-03 | Added R4c (recommended rule sets validated against a legitimate-question corpus) | PoC finding D — OOTB `jailbreak` false-positives on legitimate OpenShift questions at ~0.98 | | 2026-08-03 | PR #2182 review: `DetectorBackend` takes a structured payload; recommended-model rec split (3.3-8B benchmarked, 4.1-8B extrapolated) | @sbunciak / @tisnik review + CodeRabbit | | 2026-09-10 | Architecture rewritten to the shield-based design that shipped (Granite Guardian as a `shields:` entry with `RiskDefinition`s); R6 extended to topic-summary calls and RAG documents; open requirements linked to LCORE-3390, 3391, 4089 and 4090; original-design capabilities, including advisory risks, moved to "Deferred from the original design" | LCORE-3389 shipped as a shield type (PR #2580); implementation review of LCORE-3390 (PR #2646) | +| 2026-09-11 | Aligned with the LCORE-3390 implementation (PR #2646): configurable `model`, risks checked in parallel batches (`batch_size`), one cached client per configuration, per-risk latency logged | Implementation review of LCORE-3390 |