[FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR/guidance and smith & Ensure parameter’s completeness in test case generation

.env_template

@@ -30,6 +30,7 @@ TARGET_AGENT_PATH=examples/your_mcp_server/
 GUIDANCE_FILE=examples/your_mcp_server/smith/guidance.txt
 SYSTEM_VAR_FILE=examples/your_mcp_server/smith/system_vars.json
 PROMPTFOO_CONFIG_FILE=examples/your_mcp_server/smith/promptfooconfig.yaml
+PROMPTFOO_CONFIG_TEMPLATE=references/promptfoo_config_template.yaml
 PROMPTFOO_OUTPUT_FILE=examples/your_mcp_server/smith/redteam1.yaml
 
 ## MCP settings: for get_mcp_parameters when generating the policy
@@ -60,6 +61,9 @@ TEST_RESULT_PATH=scorecard_summary.txt
 TEST_FAILURES_PATH=score_test_failures.txt
 CROSS_VALIDATE_OUTPUT=references/cross_validate_report.json
 
+# Session configuration (written by the explorer UI, read by the CLI)
+SESSION_CONFIG_FILE=references/session_config.json
+
 # Test case generation configuration
 TEST_CASE_TEMPLATE=references/test_case_template.json
 TEST_GENERATION_PATH=src/smith/test_generation/

CHANGELOG.md

@@ -24,6 +24,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - ARES cases now inherit their parent test case's confidence score, verdict, and predicted label in the evaluation report instead of showing a blank confidence. An ARES case is a jailbreak-transformed variant of a parent `disallow` case, so it carries the parent's `ValidationResult` (matched by the parent's `user_input`) rather than being independently re-validated.
 - Updated SKILL.md documentation to reflect "all test deny" instead of "all test failed".
 - Updated car-price and call-for-papers examples' Promptfoo configuration with missing system variables.
+- `opa_policy_creation.md` Step 7: changed `cp` to `mv` for the generated policy file.
+- Include target tool parameters in the LLM prompt so generated test cases include concrete parameter values: src/smith/test_generation/case_generation.py
 
 ### Added
 
@@ -37,6 +39,11 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Integrated Promptfoo policy plugin for generating malicious test cases from guidances, with translation support for string-typed variables.
 - `ATTACK_TOOLS` environment variable to select which red-teaming tools to run (`ares`, `promptfoo`, `ares,promptfoo`, or `none`).
 - Clean-up bash script (`scripts/clean_generated.sh`) to reset generated intermediates when switching examples.
+- Added an employee hub agent example. 
+- **Promptfoo config auto-generation** (`smith --flag generate_promptfoo_config`): generates or updates a Promptfoo redteam configuration file from guidance and system variables, with a customizable template (`PROMPTFOO_CONFIG_TEMPLATE`). Also appends tool parameter definitions to `testGenerationInstructions` so Promptfoo generates prompts that include concrete values for all required parameters.
+- LLM-based tool classification for promptfoo cases: during test generation, promptfoo cases are now classified to a target tool name via a single LLM call against the MCP tool definitions, removing the hardcoded "Promptfoo" placeholder. This steps aims to make test translation apply the same tool-name mismatch check to all cases uniformly.
+- **Policy Explorer UI bridge**: `src/smith/tools/explorer_server.py` serves an interactive HTML view of the policy alongside IR (intermediate representation) data for visual inspection.
+- **Session config for IR and selected tools** (`SESSION_CONFIG_FILE`, default `references/session_config.json`): when working with IR-generated specs via the Policy Explorer UI, the explorer writes this file with `use_ir` and `selected_tools`. During test generation, `translate_case` filters out test cases whose target tool is not in `selected_tools`.
 
 ### Changed
 
@@ -48,6 +55,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Converted Promptfoo test cases now live in `references/test_cases/disallow/` (removed separate `promptfoo_malicious/` folder).
 - `test_case_translation` skips cases that already carry an `arguments` block (already translated), making translation re-runnable and avoiding a full-corpus re-translation when only newly-added bypass cases need it.
 - Reset `assets/policy.rego` to empty as a fresh starting point for policy creation.
+- `get_tool_definitions()` helper in `cli.py` to deduplicate MCP tool extraction across `test_generation`, `bypass_case_generation`, and `get_mcp_parameter` flags.
+
 
 ## [0.1.1] - 2026-06-29
 - Repackaged `scripts/` into an installable `smith` Python package using a `src/`

CLAUDE.md

@@ -31,7 +31,8 @@ make test            # policy scorecard: starts OPA in Docker + runs the package
 
 # CLI pipeline stages (run from anywhere once installed; reads paths from .env)
 smith --flag get_mcp_parameter      # auto-extract MCP tool defs -> <TARGET_AGENT_PATH>/smith/tool_definitions.json
-smith --flag test_generation        # full test-case generation pipeline
+smith --flag test_generation        # full test-case generation pipeline (includes promptfoo tool classification)
+smith --flag generate_promptfoo_config  # generate/update promptfoo redteam config from guidance
 smith --flag test_case_evaluation   # classify + validate labels + HTML report
 smith --flag test_case_translation  # resolve tool calls via agent /extract_tool_call
 smith --flag policy_testing         # run policy against all test cases (needs OPA server)
@@ -42,6 +43,8 @@ smith --flag cross_validate         # LLM cross-check failed cases → reference
 smith --flag apply_cross_validate   # apply approved label corrections from cross_validate report
 smith --flag policy_validation --policy_path <file.rego>      # validate a rego file
 smith --flag policy_validation_fix --policy_path <file.rego>  # validate and auto-fix
+smith --flag open_explorer          # launch the Policy Explorer UI (browse specs, select guidance)
+smith --flag classify_guidance      # launch the Guidance Classifier UI (map each guidance line -> tool call, combine + reset inputs)
 
 # Policy-testing OPA server (root Makefile; the packaged harness in
 # src/smith/policy_testing/ is what `smith --flag policy_testing` and `make test` invoke)
@@ -74,6 +77,8 @@ Target-agent selection is driven by a small set of vars: `TARGET_AGENT_PATH`, `G
 
 Key model vars: `MODEL_SONNET` (the LLM used across pipelines), `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `TEMP`, `TOP_P`.
 
+Session/IR config: `SESSION_CONFIG_FILE` (default `references/session_config.json`) — written by the Policy Explorer UI with `use_ir` and `selected_tools`; consumed by `translate_case` to filter test cases to only the tools selected in the explorer.
+
 ## Per-target-agent inputs
 
 Each target agent under `examples/<agent>/` carries its Smith inputs in a `smith/` subfolder:
@@ -92,9 +97,10 @@ The generated policy may **only** reference data available from tool arguments o
 ## src/smith/ package map
 
 - `policy_generation/` — MCP tool extraction (`extract_tools.py`) and rego validation (`validate_policy.py`).
-- `test_generation/` — generation pipeline stages run in order by the `test_generation` flag: `decompose` → `grey_condition` → `variable_extraction` → `case_generation` → `attack` (ARES) → `attack_promptfoo` → `convert_test_case`. Also `extract_tool_args.py` for translation.
+- `test_generation/` — generation pipeline stages run in order by the `test_generation` flag: `decompose` → `grey_condition` → `variable_extraction` → `case_generation` → `attack` (ARES) → `attack_promptfoo` → `classify_promptfoo_tool` → `convert_test_case`. Also `extract_tool_args.py` for translation and `generate_promptfoo_config.py` for config generation.
 - `test_case_evaluation/` — three-tier label validation: `tier1_rules.py` (pattern match) → `tier2_semantic.py` (embeddings + NLI) → `tier3_llm_judge.py` (LLM), plus `classify_guidance.py` and `visualization/build_report.py`.
 - `policy_agent/` — refinement engine: `red_feedback/` (DBSCAN clustering of failed cases, tuned by `CLUSTER_EPS`/`CLUSTER_MIN_SAMPLES`), `policy_analysis/regal/` (Regal), `reduce_improve/` (graph + LLM dedup), `policy_evaluation/`.
+- `tools/` — developer utilities: `explorer_server.py` (policy explorer UI bridge, `policy_explorer.html`), `guidance_classifier_server.py` + `classify_guidance_lines.py` + `guidance_classifier.html` (upstream guidance-line → tool-call classifier UI), `license_headers.py`.
 - `tests/integration/` — the `make test` scorecard harness (bash + curl against the OPA server).
 
 ## Refinement workflow (the SKILL.md contract)

README.md

@@ -57,7 +57,7 @@ python -m venv .venv
 source .venv/bin/activate
 ```
 
-**2. ARES** (red-teaming framework). Installs into `src/smith/test_generation/ares/` with its own `.venv`, which is the layout the test-generation pipeline expects (`src/smith/test_generation/attack.py` invokes `ares/.venv/bin/ares`):
+**2. ARES** (optional red-teaming framework). Installs into `src/smith/test_generation/ares/` with its own `.venv`, which is the layout the test-generation pipeline expects (`src/smith/test_generation/attack.py` invokes `ares/.venv/bin/ares`):
 
 ```bash
 cd src/smith/test_generation/ares
@@ -77,7 +77,7 @@ cd ../../../../
 source .venv/bin/activate
 ```
 
-**3. Promptfoo** (red-teaming framework).
+**3. Promptfoo** (optional red-teaming framework).
 
 ```bash
 npm install -g promptfoo
@@ -171,22 +171,22 @@ Smith operates as an agent skill with a CLI backend. The AI agent reads instruct
 │                                     Smith                                      │
 │                                                                                │
 │  SKILL.md ──→ Orchestration ──→ smith CLI                                      │
-│                                    │                                           │
-│              ┌─────────────────────┼──────────────────────┬─────────┐          │
-│              ▼                     ▼                      ▼         ▼          │
-│         Policy              Test Case Gen              Policy     Policy       │
-│         Creation                 │                     Testing   Refinement    │
-│              │        ┌──────────┼──────────┬────────┐    │         │          │
+│                                       │                                        │
+│              ┌────────────────────────┼───────────────────┬─────────┐          │
+│              ▼                        ▼                   ▼         ▼          │
+│         Policy              Test Case Generatio        Policy     Policy       │
+│         Creation                      │                Testing   Refinement    │
+│              │        ┌──────────┬────┼─────┬────────┐    │         │          │
 │              ▼        ▼          ▼          ▼        ▼    └────⇄────┘          │
 │         OPA Policy Legitimate  ARES    Promptfoo  Bypass                       │
 │         (.rego)        │        │          │        │                          │
-│                        └────────┴────┬─────┴────────┘                          │
-│                                      ▼                                         │
+│                        └────────┴─────┬────┴────────┘                          │
+│                                       ▼                                        │
 │                              Test Case Evaluation                              │
 └────────────────────────────────────────────────────────────────────────────────┘
 ```
 
-Bypass cases are generated from the current policy (a `Policy Creation` output) rather than the guidance alone, so they close the loop back into `Test Case Gen`.
+Bypass cases are generated from the current policy (a `Policy Creation` output) rather than the guidance alone, so they close the loop back into `Test Case Generation`.
 
 
 ## Core Concepts
@@ -213,6 +213,16 @@ The agent follows `test_generation/test_generation.md`, which first asks which k
 - **Policy-bypass cases** — adversarial cases that target divergences between the guidance and the **current policy** (requires an existing, non-empty policy).
 - **Both.**
 
+#### Promptfoo config auto-generation
+
+If you use Promptfoo for red-teaming, you can auto-generate the `promptfooconfig.yaml` instead of writing it manually:
+
+```bash
+smith --flag generate_promptfoo_config
+```
+
+This generates `purpose`, `contexts`, and `policy` text from your guidance and system variables, and appends tool parameter definitions to `testGenerationInstructions` so Promptfoo generates prompts with concrete values for all required parameters. Generation is LLM + deterministic — always review the output before running red-team tests.
+
 #### Guidance-targeted generation
 
 ```bash
@@ -344,9 +354,11 @@ smith/
 │   ├── policy_agent/        # OPA policy analysis and refinement
 │   ├── policy_generation/   # MCP tool extraction and policy generation
 │   ├── test_generation/     # Test case generation and translation pipeline
+│   │   ├── classify_promptfoo_tool.py  # LLM-based tool classification for promptfoo cases
+│   │   └── generate_promptfoo_config.py # Auto-generate promptfoo redteam config
 │   ├── test_case_evaluation/ # Label validation and report generation
 │   ├── policy_testing/      # OPA scorecard harness (score_card.sh, coverage)
-│   └── tools/               # Repo tooling (e.g. license headers)
+│   └── tools/               # Developer utilities (explorer UI, license headers)
 ├── tests/                   # Placeholder for the test suite (TODO)
 ├── test_generation/         # Test generation skill markdown file
 ├── .env_template            # Environment template

docs/content/docs/configuration.md

@@ -68,6 +68,7 @@ The sections below list these alongside the optional variables in context. Anyth
 | `ATTACK_TOOLS` | Comma-separated list of red-teaming tools to run during test generation. Valid values: `ares`, `promptfoo`, `ares,promptfoo`, `none` | `ares,promptfoo` |
 | `ARES_HOME` | Absolute path to the ARES installation directory | **Required when `ATTACK_TOOLS` includes `ares`** — no default |
 | `PROMPTFOO_CONFIG_FILE` | Promptfoo red-team config path | `examples/your_mcp_server/smith/promptfooconfig.yaml` |
+| `PROMPTFOO_CONFIG_TEMPLATE` | Path to a custom Jinja/YAML template used by `generate_promptfoo_config` to scaffold the config file | *(built-in default template)* |
 | `PROMPTFOO_OUTPUT_FILE` | Promptfoo generated output path | `examples/your_mcp_server/smith/redteam1.yaml` |
 | `OLLAMA_BASE_URL` | Base URL for promptfoo's native ollama provider (no `/v1` suffix). Distinct from the agent's `INFERENCE_BASE_URL` | `http://localhost:11434` |
 
@@ -80,6 +81,12 @@ The sections below list these alongside the optional variables in context. Anyth
 | `TEST_OUTPUT_DIR` | Where the scorecard harness writes results | `references/scorecard/` |
 | `CROSS_VALIDATE_OUTPUT` | Cross-validation report output path | `references/cross_validate_report.json` |
 
+## Session Config (Policy Explorer / IR)
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `SESSION_CONFIG_FILE` | Path to the session config JSON written by the Policy Explorer UI. Contains `use_ir` (boolean) and `selected_tools` (list of tool names). During translation, `translate_case` filters out test cases whose target tool is not in `selected_tools`. | `references/session_config.json` |
+
 ## Test Case Generation
 
 | Variable | Description | Default |

docs/content/docs/promptfoo-config.md

@@ -280,6 +280,29 @@ defaultTest:
     transformVars: '{ ...vars, sessionId: context.uuid }'
 ```
 
+## Auto-Generating the Configuration
+
+Instead of writing the config manually, you can generate it from your existing guidance and system variables:
+
+```bash
+smith --flag generate_promptfoo_config
+```
+
+This reads your `GUIDANCE_FILE` and `SYSTEM_VAR_FILE`, then produces (or updates) the `promptfooconfig.yaml` at the path specified by `PROMPTFOO_CONFIG_FILE`.
+
+**What it does:**
+
+- Populates `redteam.purpose` from the guidance description.
+- Creates a `contexts` entry for each role/profile found in `system_vars.json`.
+- Writes the policy `plugins` text from the guidance rules.
+- Appends **tool parameter definitions** to `testGenerationInstructions`, so Promptfoo generates prompts that include concrete values for all required parameters (e.g., required fields, allowed enum values).
+
+**Custom templates:** Set the `PROMPTFOO_CONFIG_TEMPLATE` environment variable to point to your own Jinja/YAML template if you need a different structure. When unset, the built-in default template is used.
+
+You can re-run the command after changing guidance or system variables to keep the config in sync. The generated config is a valid starting point; you can still hand-edit it afterward.
+
+> **Note:** Generation is LLM + deterministic — the LLM generates `purpose` and `contexts`, while deterministic logic fills `vars`, policy text, and tool parameters. Always review the generated config before running red-team tests, as the LLM-generated fields may contain inaccuracies.
+
 ## Common Pitfalls
 
 1. **Missing system variables in contexts** — Every variable in `system_vars.json` should appear in each context's `vars`. Otherwise, translated test cases will be incomplete.