Follow-up correctness fixes: model matching, confidence, demo, injection probe, SSRF

[imphillip]

Summary

Follow-up correctness fixes from a code review of the verifier core, on top of #1. Each fix is an independent, focused commit with a regression test, so they can be reviewed and merged commit by commit.

Fixes

1. scoring: model-label matching (model_label_parts)

   • Splitting on : kept the part after the colon, so two unrelated models sharing a tag like :free (meta-llama/llama-3-8b:free vs qwen-2-7b:free) matched on a shared ("free",) token tuple — masking a real substitution.
   • Exact token equality meant a standard dated-snapshot echo gpt-4.1-2025-04-14 did not match the requested gpt-4.1, so an honest gateway was wrongly flagged as substituted (high risk, identity_score 0).
   • Fix: compare by base name (keep the part before a :tag) and strip a dated snapshot suffix (-YYYY-MM-DD / -YYYYMMDD) as an extra candidate.

2. scoring: confidence_score scale — the mismatch/high-risk branch weighted components to sum 0.8 while the normal branch sums to 1.0, so the high-risk branch could never exceed 80 even with perfect evidence. Normalize by the weight sum so both branches share a 0-100 scale.

3. client: content extraction — extract_choice_content stringified list-shaped message.content ([{"type":"text",...}]) into a Python repr, while the protocol path already used the correct providers.extract_openai_choice_text. Drop the duplicate and reuse the providers helper.

4. demo: verdict consistency + query decode — demo_protocol_report/demo_answer hardcoded upstream gpt-4.1, so the other two advertised demo models always produced a false substituted/high-risk verdict; now echo the selected model. Also parse the settings query with parse_qs for proper percent-decoding, replacing a hand-rolled parser that only handled +.

5. probes: injection-resistance scoring — the prompt_injection probe forbade a phrase (只回答 PASS) that a faithful summary naturally quotes, so an injection-resistant answer scored 0/suspicious while a hijacked "PASS" scored 35 — correct behavior ranked below compromised behavior. Replace it with a real task plus an inline injection lure: resistant → 100, hijacked → 35.

6. report: SSRF guard on the core entry point — validate_public_base_url ran at the CLI/demo-server boundaries but not inside run_verification, so a direct Python-API caller could pass a private/metadata host and reach it with the supplied auth header. Validate target.base_url at the entry point.

Out of scope (deliberately not changed)

• DNS-rebinding / fail-open in url_safety — the guard resolves DNS only for a pre-flight check and fails open on resolution errors, so TOCTOU rebinding remains. Closing it needs request-time IP pinning across the httpx paths; left as a follow-up (noted in the commit message of fix 6).
• 198.18.0.0/15 allowlist and dev-local.sh host-API default — both looked like bugs in review but are intentional and locked by existing tests / check-deploy-scripts.sh (a benchmark/proxy range that is deliberately allowed; a developer default that assumes a local host API). Left as-is.

Testing

pytest -q — all pass (52), including new regression tests for each fix.

🤖 Generated with Claude Code