Skip to content

readme: state OpenAI's audit finding precisely - #14

Merged
mnkprs merged 1 commit into
mainfrom
openai-precision
Aug 30, 2026
Merged

readme: state OpenAI's audit finding precisely#14
mnkprs merged 1 commit into
mainfrom
openai-precision

Conversation

@mnkprs

@mnkprs mnkprs commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Why

§1 summarized OpenAI's Verified retirement as "at least 59.4% of those problems had tests that reject functionally correct submissions" and dated the post 23 February. Checked against the post itself: 59.4% is "material issues in test design and/or problem description" (35.5% narrow tests, 18.8% wide tests, 5.1% miscellaneous), the page shows no publication date, and the recommendation is broader than SWE-bench Pro alone.

What changed

  • §1: the OpenAI bullet now carries the audit's actual method (138 hardest problems, ≥6 engineers per case) and the narrow/wide/miscellaneous breakdown, notes that this taxonomy is the report's two axes, softens the date to "In February 2026", and lists the full recommendation (Pro, private uncontaminated benchmarks, expert grading).
  • §1: "left a 59% flaw rate on the hard subset" → "left material issues in 59% of the hard subset".

…escription issues; narrow/wide breakdown)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mnkprs
mnkprs merged commit b6588c1 into main Aug 30, 2026
2 checks passed
@mnkprs
mnkprs deleted the openai-precision branch August 30, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant