Skip to content

Commit 28362d2

Browse files
luohua13claude
andauthored
docs(ai): add garak LLM security scanning on Workbench (#848)
* docs(ai): add garak LLM security scanning on Workbench Prebuilt Workbench image with garak 0.17.0 and its offline detector models, datasets and NLTK corpora, so an isolated cluster can scan a published inference service without reaching PyPI or Hugging Face. Covers obtaining the image, importing the WorkspaceKind the console needs to offer it, preparing the scan configuration, running a scan and reading the reports, plus adapting the chat template for non-Qwen models and using an LLM judge detector. Verified against Qwen3.5-0.8B on vLLM: 21 probes in 775s, reports written to the Workspace volume, detectors loading from the image with the container offline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): address review on garak scanning article - drop the verified-versions line from Environment - any registry the cluster can pull from, not specifically the platform Private Registry - remove the Image contents section - condense image retrieval to the address plus the push commands, and keep section anchors only where another section links to them Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): drop the offline tar note from the Image section Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): drop the WorkspaceKind label and podMetadata notes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): make the chat endpoint the recommended scan path The shipped sample posts to /v1/completions because the chat endpoint of the verification service did not respond, but chat is the endpoint most services expose and it lets the server apply the model's own chat template. Present both plugins sections, chat first, and keep the template-adaptation section for the completions fallback only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): scan the chat endpoint only, add application-layer scanning The chat endpoint is what services expose for instruct models and what applications actually call, and the server applies the model's own chat template. Scanning through /v1/completions needed a hand-built template per model, and the rest generator drops all but the last turn, which silently degrades the multi-turn probes; drop that path and the template adaptation section with it. Fix the thinking-mode setting: chat_template_kwargs has to be nested under extra_body, otherwise the OpenAI client rejects it as an unexpected keyword argument. Add a section on what to scan: the model endpoint measures the model, while the risk that matters lives in the application in front of it, so cover passing the production system prompt and pointing the rest generator at the application's own API. Results table re-measured through the chat endpoint (21 probes, 701s), plus a note on the report corruption that can break HTML generation on long parallel runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): state the image address instead of registry CLI commands Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): any reachable OpenAI-compatible endpoint, not just cluster DNS The target does not have to be the in-cluster predictor Service: a gateway, an ingress or any other endpoint serving /v1/chat/completions works, so describe it as a base URL and mention the API key case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): write reports next to the config, drop the results section - reports go to ~/garak/report via an absolute reporting.report_dir; a relative path lands under ~/.local/share/garak, which is awkward to open from the file browser - remove the verification-results section and the report-corruption note - refresh the smoke test output for the chat generator Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): condense the image upgrade section Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ai): run the scan in the foreground Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d7d451c commit 28362d2

1 file changed

Lines changed: 455 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)