Fix rules that never fired, drop mention rules, add Jules - #2
Merged
Merged
Conversation
Measured against 5,169 pull requests in Coderbuds' production data: +172 Jules, +9 Claude Code, +4 Cursor and +4 Codex attributions, and 13 false Codex attributions removed (9 of them Dependabot bumps of @openai/codex). - Jules (new): signature, task link, jules-<id> branches. - Codex: drop bare "codex/" / "codex-" title and description mentions, which matched dependency bumps and `.codex/` paths; add the Codex cloud task link and the "[codex]" title prefix. - Claude Code: `claude/` branches; co-author trailers matched without regard to case, since git trailers are case-insensitive. - Cursor: `cursor/` background-agent branches. - Aider, WindSurf, Replit: fix rules that could never match (regex without `regex: true`, keys the matcher never reads) and drop rules that fired on a tool's name rather than its signature. - scripts/validate_rules.py + fixtures/cases.yml, run in CI: rejects unread keys and unflagged regexes, and checks real PR shapes against the tool they must or must not be attributed to. - README: drop accuracy figures the data does not support, say plainly that no marker does not mean a person wrote it, fix the schema example that taught the unread keys. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Evidence
Old vs new rules run through Coderbuds' production matcher over 5,169 pull requests (180 days, demo repos excluded; description/title/branch rules only):
@openai/codex.codex/config.toml, "docs: add Codex MCP setup", two stacked PRs whose base was acodex/branch)Rule changes
jules.google.com/task/link,jules-<id>branches. 172 PRs in one tracked repo, none attributed before.codex//codex-mentions in title/description (they matched dependency bumps and config paths, never an agent's own PR). Added[Codex Task](https://chatgpt.com/codex/cloud/tasks/…)link and[codex]title prefix. Kept branch patterns and label.^claude/branches (Claude Code web/desktop worktrees);Co-Authored-Bymatched case-insensitively (git writesCo-authored-by).^cursor/background-agent branches.regex: true, so compared as literal^aider[-/]and never matched. Aider'saider:prefix is anchored to a commit-message line start (it matched "raider:"); dropped the bare "aider" / "WindSurf" / "Replit AI" mention rules.bot_authorsusedpattern/locationandlabelsusedpattern, none of which the matcher reads. Nowusername/email/name.regex: true(dead; the footer rule already covers it).Guard rails
scripts/validate_rules.py+.github/workflows/validate.yml: rejects keys the matcher never reads, regex-looking patterns withoutregex: true, invalid regexes and#(the production delimiter). Run againstmainit reports every bug above.fixtures/cases.yml: 17 real PR shapes with the tool each must — or must not — be attributed to. All pass under both the Python reference matcher and Coderbuds' PHPCheckYamlRulesAction.README
Dropped the 98–100% accuracy badge and per-tool accuracy column — on our own data Codex CLI recall was 0/10, because it signs nothing. Added: no marker ≠ human-written. Fixed the contributing schema example, which taught the unread
bot_authorskeys. Brand spelled Coderbuds.Not in this package
The app still scores any match as "Definitive 100" regardless of a marker's confidence; that's
CheckYamlRulesActionin the Coderbuds app, not these rules.🤖 Generated with Claude Code