Skip to content

[FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR/guidance and smith & Ensure parameter’s completeness in test case generation - #39

Open
dhl123 wants to merge 9 commits into
mainfrom
hl/promptfoo_config_auto

Conversation

@dhl123

@dhl123 dhl123 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

Closes: #36 #40 #41

Changes

  • Auto-generate Promptfoo config: new smith --flag generate_promptfoo_config stage that creates/updates a Promptfoo redteam configuration from guidance and system variables, using a customizable template (PROMPTFOO_CONFIG_TEMPLATE).
  • Policy Explorer UI bridge: added explorer_server.py to serve an interactive HTML view of the policy alongside IR data for visual inspection. It also supports reseting smith configuration based on selected guidance.
  • LLM-based tool classification for promptfoo cases: during test generation, promptfoo cases are now classified to a target tool name via a single LLM call against the MCP tool definitions, removing the hardcoded "Promptfoo" placeholder. This steps aims to make test translation apply the same tool-name mismatch check to all cases uniformly. Also, when LLM calls several tools in one turn to make the decision, it avoids the promptfoo cases are assigned with the first tool call name blindly, causing false positives. For example, employee agent usually calls list_employee before every add_employee prompt. So a malicious promptfoo case will be assigned a wrong label list_employee
  • Include target tool parameters in the LLM prompt so generated test cases include concrete parameter values: src/smith/test_generation/case_generation.py. For detailed reasons please see [BUG]: Test case generation produces vague prompts missing required tool parameters #41
  • src/smith/test_generation/generate_promptfoo_config.py — Append tool parameter info to testGenerationInstructions in generated configs, same for [BUG]: Test case generation produces vague prompts missing required tool parameters #41
  • Refactored get_tool_definitions() helper to deduplicate MCP tool extraction across CLI flags.
  • Session config for IR mode: When working with IR-generated specs via the Policy Explorer UI, selected tool names are written to session_config.json; during test generation, translate_case filters out cases targeting tools not in that selection, keeping the test suite scoped to the tools under review.

Additional UI to read from raw guidance file.

  • New Guidance Classifier feature, src/smith/tools/classify_guidance_lines.py uses LLM classifier mapping each guidance line → the MCP tool call(s) it governs, referencing tool_definitions.json.
  • src/smith/tools/guidance_classifier_server.py, guidance_classifier.html serves the UI, independent from the IR explore html page and server.
  • save a copy of produced files: src/smith/tools/save_snapshot.py , src/smith/cli.py
  • scripts/clean_generated.sh now also empties assets/policy.rego.
  • test_generation/test_generation.md adds promptfoo config reminder
  • Uploads smith outputs grouped by tools.

Cpex agent demo

  • Updated the agent.py file to include three additional endpoints: /chat, /extract_tool_call, /extract_tool_definition (specifically for this agent example, since its tool definitions are inside of the agent, not server.py).
  • Added supports in smith to allow extract tool details from /extract_tool_definition endpoint.
  • Bypass cpex proxy as for now when extracting tool call values, since the proxy is not deployed for now.
  • Updated guidance.txt based on case understanding in https://github.com/praxis-proxy/demos/tree/main/demos/cpex.

Checks

  • make ci passes (lint, Rego lint, license headers, build smoke)
  • make test passes (policy scorecard — needed if policy behavior changed)
  • CHANGELOG.md updated under ## [Unreleased] (if user-facing)
  • Commits are signed off for the DCO (git commit -s)

Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@dhl123 dhl123 added the enhancement New feature or request label Jul 29, 2026
@dhl123 dhl123 changed the title [FEATURE]: support auto generation/updation of promptfoo config files [FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR and smith Jul 29, 2026
dhl123 added 2 commits July 30, 2026 13:56
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
…nslation, replacing the previous special-case bypass

Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@dhl123 dhl123 changed the title [FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR and smith [FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR and smith & Ensure parameter’s completeness in test case generation Aug 3, 2026
…s, IR constrains target tool scope, updated two agent examples with new cases

Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@dhl123
dhl123 force-pushed the hl/promptfoo_config_auto branch from 7458ae3 to c1393e0 Compare August 7, 2026 17:15
dhl123 added 2 commits August 7, 2026 13:15
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@dhl123 dhl123 changed the title [FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR and smith & Ensure parameter’s completeness in test case generation [FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR/guidance and smith & Ensure parameter’s completeness in test case generation Aug 10, 2026
dhl123 added 2 commits August 10, 2026 16:43
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@dhl123
dhl123 marked this pull request as ready for review August 11, 2026 15:46
@dhl123
dhl123 requested a review from araujof as a code owner August 11, 2026 15:46
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

1 participant