Skip to content

fix: repair broken error handling, deprecated model, and Windows crashes - #1

Open
Pyhroff wants to merge 1 commit into
masterfrom
fix/error-handling-and-deprecated-model
Open

fix: repair broken error handling, deprecated model, and Windows crashes#1
Pyhroff wants to merge 1 commit into
masterfrom
fix/error-handling-and-deprecated-model

Conversation

@Pyhroff

@Pyhroff Pyhroff commented Sep 7, 2026

Copy link
Copy Markdown
Owner

What changed

  • TAP and Crescendo now catch exceptions around API calls instead of crashing the whole run on a transient failure (rate limit, timeout, etc.) — matches PAIR's existing error handling.
  • Fixed ci/sweep silently folding errored goals into the ASR denominator, which could misreport a failed scan as 0% ASR ("model resisted") instead of flagging that the scan never actually ran. ci now fails the gate on any errored goal, and sweep shows an explicit error count instead of a false RESILIENT verdict.
  • Replaced the deprecated/retired Groq model id (llama-3.3-70b-versatile) with a currently valid one everywhere it was hardcoded (.env.example, config.py, CLI option defaults, docstrings).
  • Forced UTF-8 stdout/stderr on Windows so rich's box-drawing/arrow characters don't crash on the default cp1252 console codepage.
  • Fixed defend's instruction_override regex missing stacked modifiers (e.g. "ignore all previous instructions" wasn't matched — only single-modifier phrasing was).
  • Fixed history crashing on sqlite3.Row.get() (Row doesn't support .get(), only dict does).
  • Fixed TAP's scan-result formatting — garbled 24n d3 now reads 24 nodes (d3).
  • Added a category-filter error message listing valid categories instead of a bare "no behaviors matched."
  • Added a regression test for the errored-goals-must-fail-the-ci-gate behavior.

Why

Ran a full manual audit of every CLI command against the live Groq API. Several of these were only exposed once real API calls were actually made (a rate-limit error during a live Crescendo run crashed it uncaught, which is what led to auditing and fixing TAP's identical gap before it could do the same). The ASR/errored-goals issue is the most consequential fix — as shipped, a ci gate could silently PASS when the scan never actually completed, which defeats the point of a CI safety gate.

Testing

  • Full test suite: 62/62 passing (61 existing + 1 new regression test)
  • Every CLI command (scan with pair/tap/crescendo, sweep, ci, defend, history, report, serve, gcg) exercised against the live Groq API and/or a local HF model, not just imported
  • Confirmed the exact bug scenarios before and after each fix (e.g. reproduced the regex false-negative, the sqlite3.Row crash, and the ERROR-displayed-as-RESILIENT sweep output, then verified each was resolved)

- TAP and Crescendo now catch exceptions around API calls instead of
  crashing the whole run on a transient failure (rate limit, timeout, etc.)
- Fix ci/sweep silently folding errored goals into the ASR denominator,
  which could misreport a failed scan as 0% ASR (model resisted) instead
  of flagging that the scan never actually ran; ci now fails the gate and
  sweep now shows an explicit error count instead of a false RESILIENT verdict
- Replace deprecated/retired Groq model id (llama-3.3-70b-versatile) with
  a currently valid one everywhere it was hardcoded
- Force UTF-8 stdout/stderr on Windows so rich's box-drawing/arrow
  characters don't crash on the default cp1252 console codepage
- Fix defend's instruction-override regex missing stacked modifiers
  (e.g. "ignore all previous instructions")
- Fix history command crashing on sqlite3.Row.get()
- Fix TAP scan-result formatting (garbled '24n d3' -> '24 nodes (d3)')
- Add category-filter error message listing valid categories
- Add regression test for the errored-goals-must-fail-ci-gate behavior
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant