Skip to content

FIX: record UNDETERMINED when the objective scorer returns no score - #3169

Open
Utkarsh Bahuguna (u7k4rs6) wants to merge 1 commit into
microsoft:mainfrom
u7k4rs6:fix/unscored-undetermined
Open

Utkarsh Bahuguna (u7k4rs6) wants to merge 1 commit into
microsoft:mainfrom
u7k4rs6:fix/unscored-undetermined

Conversation

@u7k4rs6

Copy link
Copy Markdown
Contributor

Fixes #3112.

When an objective scorer is set and there's a response but the scorer returns no score, the outcome is now UNDETERMINED with "Objective scorer returned no score", instead of FAILURE. That applies to PromptSendingAttack, MultiPromptSendingAttack, RedTeamingAttack and ChunkedRequestAttack.

What doesn't change:

  • a real false score is still FAILURE
  • blocked responses get a false score from the scorer, so they stay FAILURE
  • no response at all stays FAILURE

New test goes through execute_async with a real text-only SubStringScorer and reads the stored result back:

  • each of prompt sending, multi prompt sending and red teaming against text, blocked and image responses (image is the unscored one)
  • chunked request with and without an objective score

The 4 unscored cases fail on main, and the 7 text/blocked ones pass on both, as guards. Three existing tests pinned the old FAILURE for "scorer returned nothing", so I updated them. tests/unit/executor, scenario, analytics and backend: 6275 passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: attacks record FAILURE when the objective scorer returns no score

1 participant