feat(assessors): add lint config coverage assessor#516
Merged
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds the ChangesLint configuration coverage
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Repository
participant LintConfigCoverageAssessor
participant CIWorkflows
participant Finding
Repository->>LintConfigCoverageAssessor: provide language and configuration files
LintConfigCoverageAssessor->>CIWorkflows: extract lint commands
CIWorkflows-->>LintConfigCoverageAssessor: return detected tools
LintConfigCoverageAssessor->>Finding: produce score, evidence, and remediation
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/agentready/assessors/code_quality.py`:
- Around line 1661-1665: Update the CI tool-detection logic around
_extract_ci_run_commands and the candidate loop to extract commands from GitLab
script fields and CircleCI run.command fields, then match candidates only as
explicit executable tokens using token boundaries rather than substring checks.
Preserve all_tools updates only for genuine command invocations and apply the
same behavior to the related logic around lines 1688–1700.
- Around line 1303-1329: Update the assessor scoring around n_covered to use
calculate_proportional_score() so proportional results follow the documented
integer contract. Replace the direct Finding(...) constructions in both the
passing and failing branches with Finding.create_pass() and
Finding.create_fail(), preserving the existing measured values, threshold,
evidence, remediation, and error details.
- Around line 1342-1374: Replace binary Python tool-presence detection in the
assessor with enabled rule-family parsing: empty configs must receive no
coverage, and inactive linters-settings keys must not count as enabled linters.
Update the ESLint handling to require plugins/configs to be referenced by active
configuration rather than merely installed. In
src/agentready/assessors/code_quality.py lines 1342-1374, 1399-1411, and
1463-1496, adjust the relevant configuration parsing and tool-detection logic;
in tests/unit/test_assessors_code_quality.py lines 479-488, add explicit Ruff
rules plus an empty-config negative case, and in lines 650-667 assert unused
installed plugins receive no category credit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 2ef1dc6d-9da9-4acd-8784-5479d06e7680
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock,!uv.lock
📒 Files selected for processing (5)
docs/attributes.mdsrc/agentready/assessors/__init__.pysrc/agentready/assessors/code_quality.pysrc/agentready/data/default-weights.yamltests/unit/test_assessors_code_quality.py
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/attributes.md`:
- Around line 894-916: Synchronize the Go, Python, and JavaScript/TypeScript
tool catalogs in docs/attributes.md with the assessor’s recognized tools, adding
entries such as pycodestyle, flake8-bandit, unused, nilnil, gocritic,
eslint:all, airbnb-base, and plugin:import in their appropriate categories.
Prefer generating the tables from the assessor catalogs if supported; otherwise
document every recognized tool so the published scoring expectations remain
complete.
In `@src/agentready/assessors/code_quality.py`:
- Around line 1418-1443: Update the ESLint config processing in the JSON/YAML
loop and `_extract_eslint_tools` to parse `.eslintrc.yaml` and `.eslintrc.yml`
with `yaml.safe_load` instead of treating them as JSON or using text matching.
Inspect only active `extends`, `plugins`, and `rules` entries so comments,
disabled examples, and unrelated text cannot count as tool coverage; preserve
the existing structured behavior for JSON configs.
- Around line 1295-1301: Update the tool-matching logic in the catalog coverage
loop to require exact normalized tool-ID matches, removing the prefix-based
startswith behavior that lets tools such as ruff-format match ruff. Preserve
support for explicitly defined aliases only, and continue recording categories
and tools for valid matches.
- Around line 1398-1404: Guard the parsed configuration root before calling
.get() in the linter configuration reader: require parsed to be a mapping and
otherwise replace it with an empty mapping or return the existing skipped/error
finding. Apply the same protection to the package.json, tsconfig, and pre-commit
readers so scalar or list roots cannot crash their assessors, while preserving
normal object-based parsing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 4e25b025-fc40-4c28-b631-ff668c4a22ce
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock,!uv.lock
📒 Files selected for processing (5)
docs/attributes.mdsrc/agentready/assessors/__init__.pysrc/agentready/assessors/code_quality.pysrc/agentready/data/default-weights.yamltests/unit/test_assessors_code_quality.py
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (1)
src/agentready/assessors/code_quality.py (1)
1303-1330: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winStill bypasses
Finding.create_pass()/Finding.create_fail(); score still doesn't match the documented 33/67 contract.
calculate_proportional_score(1, 3)= 33.33…,(2, 3)= 66.67…, not the33/67documented indocs/attributes.md. This was flagged before and marked unresolved (no "Addressed" note, unlike sibling fixes in this file). Both branches still constructFinding(...)directly instead of the required factory methods.🔧 Proposed fix (verify exact `Finding.create_pass`/`create_fail` signature first)
- if n_covered == 3: - return Finding( - attribute=self.attribute, - status="pass", - score=score, - measured_value="3/3 categories", - threshold="all 3 categories (correctness, standards, security)", - evidence=evidence, - remediation=None, - error_message=None, - ) - - missing = [c for c in self.CATEGORIES if c not in covered] - return Finding( - attribute=self.attribute, - status="fail", - score=score, - measured_value=f"{n_covered}/3 categories", - threshold="all 3 categories (correctness, standards, security)", - evidence=evidence, - remediation=self._create_remediation(missing, primary), - error_message=None, - ) + if n_covered == 3: + return Finding.create_pass( + attribute=self.attribute, + score=score, + measured_value="3/3 categories", + threshold="all 3 categories (correctness, standards, security)", + evidence=evidence, + ) + + missing = [c for c in self.CATEGORIES if c not in covered] + return Finding.create_fail( + attribute=self.attribute, + score=score, + measured_value=f"{n_covered}/3 categories", + threshold="all 3 categories (correctness, standards, security)", + evidence=evidence, + remediation=self._create_remediation(missing, primary), + )As per coding guidelines, "Create findings using
Finding.create_pass()orFinding.create_fail()with appropriate evidence and details/remediation parameters."#!/bin/bash rg -n -A20 'def create_pass|def create_fail' src/agentready/models/finding.py🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/agentready/assessors/code_quality.py` around lines 1303 - 1330, Update the pass/fail branches in the assessor method to use Finding.create_pass() and Finding.create_fail() with the existing evidence, threshold, and remediation details. Replace calculate_proportional_score for this three-category result with the documented integer scores: 0, 33, 67, or 100 based on n_covered, and verify the factory signatures in Finding before wiring the arguments.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/agentready/assessors/code_quality.py`:
- Around line 1502-1516: Update _collect_js_tools() to validate that tsconfig’s
compilerOptions value is a dictionary before calling get on it. Treat non-object
values such as lists, strings, or null as malformed and skip that configuration
gracefully, preserving the existing behavior for valid compilerOptions and
preventing assessor crashes.
- Around line 1632-1649: Harden _collect_precommit_hooks against malformed YAML
structures by validating that repos is a list, each repo_entry is a dict, hooks
is a list, and each hook is a dict before accessing .get or iterating. Preserve
collecting lowercase nonempty hook ids and return the tools set while skipping
invalid entries instead of raising AttributeError.
- Around line 1351-1360: Update the setup.cfg parsing branch in the assessor to
add a section to tools only when it contains at least one configuration entry,
matching the existing pyproject.toml empty-section behavior. Preserve the
current normalization and error handling, and add a regression test alongside
test_empty_pyproject_tool_section_not_counted for an empty setup.cfg tool
section.
- Around line 1691-1696: Update the tool-matching regex in the CI scan loop
around _extract_ci_run_commands so detection works for catalog entries beginning
with “@”. Replace the leading word-boundary assumption with a delimiter check
that still prevents substring matches, while preserving whole-name matching for
ordinary tools and scoped package names such as
`@typescript-eslint/recommended-type-checked`.
---
Duplicate comments:
In `@src/agentready/assessors/code_quality.py`:
- Around line 1303-1330: Update the pass/fail branches in the assessor method to
use Finding.create_pass() and Finding.create_fail() with the existing evidence,
threshold, and remediation details. Replace calculate_proportional_score for
this three-category result with the documented integer scores: 0, 33, 67, or 100
based on n_covered, and verify the factory signatures in Finding before wiring
the arguments.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: e45f11d6-43bf-4024-92d1-a31d5c57725a
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock,!uv.lock
📒 Files selected for processing (5)
docs/attributes.mdsrc/agentready/assessors/__init__.pysrc/agentready/assessors/code_quality.pysrc/agentready/data/default-weights.yamltests/unit/test_assessors_code_quality.py
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (1)
src/agentready/assessors/code_quality.py (1)
1303-1330: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winStill bypasses
Finding.create_pass()/create_fail(), and score isn't rounded to match the documented 33/67/100 contract.Both branches construct
Finding(...)directly instead of using the required factory methods, andcalculate_proportional_score(n_covered, 3)yields33.333.../66.666...rather than the integer33/67documented in docs/attributes.md. This mirrors a still-open prior review comment on this exact code.🔧 Proposed fix (verify exact factory signature before applying)
n_covered = len(covered) - score = self.calculate_proportional_score(n_covered, 3) + score = round(self.calculate_proportional_score(n_covered, 3)) evidence = self._build_evidence(covered, tools_by_cat, tools) if n_covered == 3: - return Finding( - attribute=self.attribute, - status="pass", - score=score, - measured_value="3/3 categories", - threshold="all 3 categories (correctness, standards, security)", - evidence=evidence, - remediation=None, - error_message=None, - ) + return Finding.create_pass( + self.attribute, + score=score, + measured_value="3/3 categories", + threshold="all 3 categories (correctness, standards, security)", + evidence=evidence, + ) missing = [c for c in self.CATEGORIES if c not in covered] - return Finding( - attribute=self.attribute, - status="fail", - score=score, - measured_value=f"{n_covered}/3 categories", - threshold="all 3 categories (correctness, standards, security)", - evidence=evidence, - remediation=self._create_remediation(missing, primary), - error_message=None, - ) + return Finding.create_fail( + self.attribute, + score=score, + measured_value=f"{n_covered}/3 categories", + threshold="all 3 categories (correctness, standards, security)", + evidence=evidence, + remediation=self._create_remediation(missing, primary), + )As per coding guidelines, "Create findings using
Finding.create_pass()orFinding.create_fail()with appropriate evidence and details/remediation parameters."#!/bin/bash rg -n -A 20 'def create_pass|def create_fail' src/agentready/models/finding.py🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/agentready/assessors/code_quality.py` around lines 1303 - 1330, Update the finding construction in the assessor method around calculate_proportional_score to use Finding.create_pass() for the 3/3 branch and Finding.create_fail() for the missing-category branch, matching their existing signatures and preserving the current evidence, remediation, and error details. Round the proportional score to the documented integer contract so coverage yields 33, 67, or 100 rather than fractional values.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/attributes.md`:
- Around line 890-908: Update the documented source list to include
`.travis.yml` alongside the other CI workflow files, and add `.pylintrc` to the
Python configuration paths in the Tool Catalog. Keep the existing categories and
tool listings unchanged.
In `@src/agentready/assessors/__init__.py`:
- Line 89: Update the Tier 2 Critical comment in the assessor documentation to
list all 2%-weight exceptions: lint_config_coverage, gitignore_completeness, and
pattern_references, while preserving the stated 9-attribute 3% weighting.
In `@src/agentready/assessors/code_quality.py`:
- Around line 1400-1413: Update the linters handling in _collect_go_tools so
parsed.get("linters") is validated as a dictionary before calling .get("enable")
or .get("enable-all"); treat any non-dict value, including non-empty lists and
strings, as malformed and skip that configuration without raising. Preserve
normal enable-list and enable-all processing for valid dictionaries and ensure
the assessor degrades gracefully.
In `@src/agentready/data/default-weights.yaml`:
- Around line 55-65: Update the top-of-file summary in default-weights.yaml to
match the current Tier 2 configuration: report 29 total attributes, and describe
Tier 2 as 27% total across 10 attributes without claiming they are uniformly
weighted at 3% each. Leave the individual weight entries unchanged.
---
Duplicate comments:
In `@src/agentready/assessors/code_quality.py`:
- Around line 1303-1330: Update the finding construction in the assessor method
around calculate_proportional_score to use Finding.create_pass() for the 3/3
branch and Finding.create_fail() for the missing-category branch, matching their
existing signatures and preserving the current evidence, remediation, and error
details. Round the proportional score to the documented integer contract so
coverage yields 33, 67, or 100 rather than fractional values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 44c2f79e-aa26-48fb-8942-5c3424ace512
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock,!uv.lock
📒 Files selected for processing (5)
docs/attributes.mdsrc/agentready/assessors/__init__.pysrc/agentready/assessors/code_quality.pysrc/agentready/data/default-weights.yamltests/unit/test_assessors_code_quality.py
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/agentready/assessors/code_quality.py`:
- Around line 1220-1242: Update the _JS_TOOLS mapping to include a
JavaScript-native correctness tool, such as the core ESLint recommended
configuration, so JavaScript repositories can earn the correctness signal
without TypeScript tooling. Add or update the relevant full-pass test to cover a
JavaScript-only configuration and verify it reaches 3/3 with remediation
pointing to the JavaScript tool rather than `@typescript-eslint`.
- Around line 1706-1713: Update _extract_ci_run_commands and the
candidate-detection logic so only commands that actually invoke a tool count
toward coverage. Return an empty command set for successfully parsed YAML
without run/script commands, and exclude installation commands, echo statements,
comments, job names, and other inert textual mentions while preserving valid
tool invocations and `@-prefixed` names.
- Around line 1589-1600: Update the JSON-handling branch in the tool-extraction
method to return without text matching when is_json is true and json.loads fails
or yields a non-dict. Preserve the existing _extract_eslint_tools_from_dict path
for valid JSON objects, while retaining substring matching only for non-JSON
configuration formats.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 49e95da5-dd1f-4d30-a928-62c0c0b619b5
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
docs/attributes.mdsrc/agentready/assessors/__init__.pysrc/agentready/assessors/code_quality.pysrc/agentready/data/default-weights.yamltests/unit/test_assessors_code_quality.py
Binary lint detection doesn't distinguish a formatting-only config from one running correctness, standards, and security tools — both score the same. Agents generating code in a formatting-only repo get no automated signal about logic errors or security anti-patterns. New LintConfigCoverageAssessor (lint_config_coverage, T2, 2%) parses lint configs and CI workflows to categorize detected tools into three buckets (correctness, standards, security), scoring 33/67/100 for 1/2/3 categories covered. Supports Go (.golangci.yml), Python (pyproject.toml, ruff.toml, .flake8, etc.), and JS/TS (ESLint, tsconfig). CI workflow files are also scanned. Closes ambient-code#511 Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
📈 Test Coverage Report
Coverage calculated from unit tests only |
jwm4
self-requested a review
July 15, 2026 14:41
jwm4
requested changes
Jul 15, 2026
jwm4
left a comment
jwm4
left a comment
Contributor
There was a problem hiding this comment.
Solid PR. The assessor is well-structured, follows existing codebase patterns correctly, and the test coverage is thorough. I tested the latest revision against real repositories and the results look right:
| Repo | Language | Result | Details |
|---|---|---|---|
| golangci/golangci-lint | Go | PASS 100 | All three categories detected from .golangci.yml |
| pydantic/pydantic | Python | FAIL 67 | Correctness + standards via mypy/ruff, missing security |
| fastapi/fastapi | Python | FAIL 67 | Same pattern as pydantic |
| expressjs/express | JS | FAIL 0 | Correctly scores zero (no lint categories configured) |
The latest push addresses the issues from the earlier CodeRabbit reviews well, particularly the CI workflow false positive fix (returning empty when YAML parses fine but has no run:/script: steps) and the defensive type guards throughout the config parsers.
One remaining item: the tool catalogs in docs/attributes.md are missing several tools that the assessor's code recognizes. For example, pycodestyle, flake8-bandit, unused, nilnil, eslint:all, airbnb-base, and plugin:import appear in the code but not in the docs tables. Per project conventions, these should be kept in sync.
All 47 unit tests pass and linting is clean.
-- Bill Murdock, with assistance from Claude Code
Contributor
Author
|
Hi @jwm4, thanks for looking at the PR! AFAIK these should be addressed unused, nilnil -> line 898, Go Correctness
|
jwm4
approved these changes
Jul 16, 2026
jwm4
left a comment
jwm4
left a comment
Contributor
There was a problem hiding this comment.
Oh, sorry! I think I mixed up what was in the original PR vs what was in the updated PR. I think this is good then. I will merge now.
Contributor
|
🎉 This PR is included in version 2.50.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
15 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add
LintConfigCoverageAssessorto evaluate lint config depth across threecategories: correctness, standards, and security. A formatting-only config
now scores differently from one running staticcheck, revive, and gosec.
Type of Change
Related Issues
Fixes #511
Changes Made
LintConfigCoverageAssessor(lint_config_coverage, Tier 2, 2% weight).golangci.yml), Python (pyproject.toml,ruff.toml,.flake8, etc.), and JS/TS (ESLint,tsconfig.json) lint configsdocs/attributes.mdupdated with tool catalog and remediation examplesdefault-weights.yamlupdated (funded by −1%gitignore_completeness, −1%pattern_references)Testing
pytest)Manual test: assessed
golangci/golangci-lintrepo →lint_config_coveragePASS 100/100 ✅Checklist
Additional Notes