diff --git a/CHANGELOG.md b/CHANGELOG.md index fecbcda..7ccac65 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ Notable changes to Agent Code Guard are recorded here. ### Fixed +- Known per-file syntax and provider failures now produce blocking structured + incomplete results while preserving independent LOC, Markdown, and unaffected + syntax evidence; completed output remains schema- and byte-compatible. - Valid C# that uses `async` as an expression identifier or named-argument name now receives a narrow, coordinate-preserving parser compatibility retry while unknown, ambiguous, and malformed syntax still fails closed. diff --git a/README.md b/README.md index 33652e7..ec0905d 100644 --- a/README.md +++ b/README.md @@ -136,12 +136,14 @@ hook-assisted process. warranted; normally exit `1`. - **Completed FAIL** — blocks normal completion until fixed or an explicitly authorized exception applies; exit `2` with a completed report on stdout. +- **INCOMPLETE** — one or more requested syntax analyses are unavailable; + independent completed findings remain authoritative, but the run exits `3`. - An argparse usage or invalid-choice error exits `2`, writes usage/error text to stderr, and produces no completed report. -- Other Code Guard tool, configuration, scope, or provider errors exit `3`. +- Other Code Guard tool, configuration, scope, or unexpected errors exit `3`. `--ci` makes REVIEW nonblocking at the process level by changing its exit to -`0`; it does not hide the findings or change FAIL and tool-error exits. +`0`; it does not hide findings or change FAIL, INCOMPLETE, and tool-error exits. **Never game a metric.** Do not create artificial helpers, files, abstractions, formatting, exclusions, or policy changes merely to lower a @@ -173,6 +175,12 @@ result, scope, required policies, guards, ordering, and actionable findings. Named modes require `--json`. See [Usage](https://github.com/stef-k/agent-code-guard/blob/main/docs/usage.md) for the schema and option contract. +Known per-file syntax or provider failures instead produce `overall: +"incomplete"`, ordered structured `unavailable` records, and completed guard +evidence. Syntax guards identify unavailable paths; LOC and Markdown remain +complete. Full, debug, and compact JSON retain unavailable records unchanged, +and completed runs retain their existing schema. + ### Common scope commands ```bash @@ -198,7 +206,8 @@ Markdown guards apply to `.md` files. Generic `.h` files are not syntax-dispatched; `.markdown` is not enabled; Vue template and style regions are not executable syntax input; and unsupported artifacts are inapplicable. Malformed applicable syntax or a required provider -failure is a fail-closed tool error. See [Language support](https://github.com/stef-k/agent-code-guard/blob/main/docs/language-support.md). +failure is unavailable evidence, distinct from both an inapplicable file and a +completed FAIL finding. See [Language support](https://github.com/stef-k/agent-code-guard/blob/main/docs/language-support.md). ### Skill integration diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 63d4b00..9a81d53 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -230,9 +230,12 @@ and decision relationships. Tree-sitter nodes never cross the extraction boundar Tree-sitter 0.26.0 and tree-sitter-language-pack 1.14.3 are the pinned initial provider. Python 3.10+ and a compatible platform wheel/native build are required only when syntax analysis is invoked. Unsupported ordinary artifacts are -inapplicable; a supported artifact with malformed syntax or an unavailable -provider/grammar is a deterministic analysis error suitable for the existing -runner exit-3 boundary. +inapplicable. The strict `analyze_files` seam raises for malformed syntax or an +unavailable provider/grammar. The runner's batch seam catches only those known +per-file failures, returns immutable completed facts plus ordered immutable +unavailable records, and continues independent files. Arbitrary reads, +extraction, configuration, guard, and programming failures retain the existing +abort boundary. An incomplete public result remains blocking with exit `3`. ## D23 — Second-wave languages preserve the production fact contract diff --git a/docs/language-support.md b/docs/language-support.md index 4cae8e8..3f1840f 100644 --- a/docs/language-support.md +++ b/docs/language-support.md @@ -59,10 +59,15 @@ documentation format, independently of syntax-language dispatch. ## Failure behavior Malformed applicable syntax and failures to load a required parser provider or -grammar are fail-closed tool errors (exit `3`). Code Guard does not produce a -partial heuristic result. Files with unsupported extensions are simply -inapplicable to syntax or Markdown guards; other applicable guards may still -consider them according to their own configured extension policy. +grammar are fail-closed unavailable evidence (exit `3`). The runner records the +affected path, embedded language, failure kind, and exact message, discards all +syntax facts for that file, and continues independent LOC, Markdown, and other +source-language analysis. It never manufactures partial or heuristic syntax +metrics. Direct syntax-pipeline callers remain strict and raise the original +error. Files with unsupported extensions are instead inapplicable to syntax or +Markdown guards; other applicable guards may still consider them according to +their own configured extension policy. A completed FAIL is different again: it +is an authoritative completed guard finding, not unavailable provider evidence. Operating-system, architecture, Python-version, and native-wheel availability are documented separately in [Platform support](platform-support.md). diff --git a/docs/usage.md b/docs/usage.md index e16537b..85da3cf 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -202,11 +202,13 @@ must not create a ratchet baseline. - `PASS` means no special action and exits `0`. - `REVIEW` means inspect findings and apply judgment; normal invocation exits `1`. - `FAIL` blocks normal completion and exits `2`. -- Tool, configuration, parser/provider, or scope errors exit `3`. +- `INCOMPLETE` retains independent completed evidence when known per-file + syntax/provider evidence is unavailable and exits `3`. +- Other tool, configuration, or scope errors exit `3` without a completed report. -`--ci` changes a REVIEW-only result to exit `0`. It does not suppress FAIL or -tool errors. REVIEW is not automatic refactoring, and metrics must never be -gamed. +`--ci` changes a REVIEW-only result to exit `0`. It does not suppress FAIL, +INCOMPLETE, or tool errors. REVIEW is not automatic refactoring, and metrics +must never be gamed. ## Human and JSON output @@ -253,6 +255,17 @@ Counts do not change aggregate state, findings, required policies, or exit codes. Tool errors retain their existing human or JSON error form and do not include a successful `scope` object. +When a known per-file syntax or provider failure occurs, the headline is +`INCOMPLETE`, followed by ordered unavailable context and the incomplete syntax +guard identifiers before ordinary findings. JSON uses `overall: "incomplete"`, +adds the authoritative completed aggregate as `completedOverall`, and includes +ordered top-level `unavailable` records containing `path`, embedded `language`, +`kind`, and the exact provider message. `scope.unavailable` overlaps +`analyzed`, so `analyzed + inapplicable == selected` remains true. Every guard +adds `complete` only on incomplete runs; incomplete syntax guards also add +ordered `unavailablePaths`. Guard states, findings, and `requiredPolicies` +continue to describe only completed evidence. + Choose a completed-analysis serialization mode explicitly when needed: ```bash @@ -269,6 +282,11 @@ but their normalized state is `pass`. `debug` is an explicit name for the full output and is byte-for-byte identical to bare `--json` for the same completed analysis. +For incomplete output, full and debug retain identical unavailable records; +compact filters only ordinary passing findings and also retains those records +unchanged. All three JSON modes, human output, normal invocation, and `--ci` +exit `3`. + Both named modes require `--json` and apply only to completed analysis output. They do not change analysis, scope, policies, ordering, aggregate or guard states, exit codes, or error shapes and channels. Version JSON supports only diff --git a/src/agent_code_guard/analysis/errors.py b/src/agent_code_guard/analysis/errors.py index 96e008e..c8c02f5 100644 --- a/src/agent_code_guard/analysis/errors.py +++ b/src/agent_code_guard/analysis/errors.py @@ -4,6 +4,10 @@ class AnalysisError(RuntimeError): """Base error suitable for Code Guard's existing exit-3 boundary.""" + def __init__(self, message: str, *, language: str | None = None) -> None: + super().__init__(message) + self.language = language + class ProviderUnavailableError(AnalysisError): """The configured parser provider or a required grammar is unavailable.""" diff --git a/src/agent_code_guard/analysis/pipeline.py b/src/agent_code_guard/analysis/pipeline.py index c29f19f..793c6eb 100644 --- a/src/agent_code_guard/analysis/pipeline.py +++ b/src/agent_code_guard/analysis/pipeline.py @@ -2,11 +2,12 @@ from __future__ import annotations +from dataclasses import dataclass from pathlib import Path from .adapters import extract_facts from .csharp_compat import corrected_csharp_root -from .errors import SyntaxAnalysisError +from .errors import ProviderUnavailableError, SyntaxAnalysisError from .facts import AnalysisFacts, FileFacts from .provider import ParserProvider, TreeSitterProvider from .regions import executable_regions, is_applicable @@ -15,31 +16,71 @@ def analyze_files(files: tuple[Path, ...] | list[Path], provider: ParserProvider | None = None) -> AnalysisFacts: """Analyze only applicable entries from the already-resolved caller scope.""" active_provider = provider or TreeSitterProvider() + results = [_analyze_file(Path(path), active_provider) for path in files if is_applicable(Path(path))] + return AnalysisFacts(tuple(results)) + + +@dataclass(frozen=True) +class UnavailableAnalysis: + path: Path + language: str + kind: str + message: str + + +@dataclass(frozen=True) +class BatchAnalysis: + facts: AnalysisFacts + unavailable: tuple[UnavailableAnalysis, ...] + + +def analyze_files_for_runner( + files: tuple[Path, ...] | list[Path], provider: ParserProvider | None = None, +) -> BatchAnalysis: + """Analyze selected files independently while retaining only known unavailable evidence.""" + active_provider = provider or TreeSitterProvider() results: list[FileFacts] = [] - for path in files: - path = Path(path) + unavailable: list[UnavailableAnalysis] = [] + for value in files: + path = Path(value) if not is_applicable(path): continue - callables = [] - controls = [] - decisions = [] - regions = executable_regions(path, active_provider) - for region in regions: - tree = active_provider.parse(region.language, region.source) - root = tree.root_node - if tree.root_node.has_error: - root = ( - corrected_csharp_root(active_provider, region.source, tree) - if region.language == "csharp" - else None + try: + results.append(_analyze_file(path, active_provider)) + except (SyntaxAnalysisError, ProviderUnavailableError) as exc: + if exc.language is None: + raise + kind = "syntax" if isinstance(exc, SyntaxAnalysisError) else "provider" + unavailable.append(UnavailableAnalysis(path, exc.language, kind, str(exc))) + return BatchAnalysis(AnalysisFacts(tuple(results)), tuple(unavailable)) + + +def _analyze_file(path: Path, provider: ParserProvider) -> FileFacts: + callables = [] + controls = [] + decisions = [] + regions = executable_regions(path, provider) + for region in regions: + try: + tree = provider.parse(region.language, region.source) + except (SyntaxAnalysisError, ProviderUnavailableError) as exc: + if exc.language is None: + raise type(exc)(str(exc), language=region.language) from exc + raise + root = tree.root_node + if tree.root_node.has_error: + root = ( + corrected_csharp_root(provider, region.source, tree) + if region.language == "csharp" + else None + ) + if root is None: + raise SyntaxAnalysisError( + f"unable to parse {path}: embedded {region.language} syntax tree contains errors", + language=region.language, ) - if root is None: - raise SyntaxAnalysisError( - f"unable to parse {path}: embedded {region.language} syntax tree contains errors" - ) - region_callables, region_controls, region_decisions = extract_facts(root, region) - callables.extend(region_callables) - controls.extend(region_controls) - decisions.extend(region_decisions) - results.append(FileFacts(path, tuple(callables), tuple(controls), tuple(decisions), len(regions))) - return AnalysisFacts(tuple(results)) + region_callables, region_controls, region_decisions = extract_facts(root, region) + callables.extend(region_callables) + controls.extend(region_controls) + decisions.extend(region_decisions) + return FileFacts(path, tuple(callables), tuple(controls), tuple(decisions), len(regions)) diff --git a/src/agent_code_guard/analysis/provider.py b/src/agent_code_guard/analysis/provider.py index 792296e..eba493f 100644 --- a/src/agent_code_guard/analysis/provider.py +++ b/src/agent_code_guard/analysis/provider.py @@ -33,13 +33,15 @@ def parse(self, language: str, source: bytes): except (ImportError, LookupError, OSError, RuntimeError) as exc: raise ProviderUnavailableError( f"syntax provider unavailable for supported language {language!r}: {exc}; " - "reinstall Agent Code Guard" + "reinstall Agent Code Guard", + language=language, ) from exc self._parsers[language] = parser try: return parser.parse(source) - except Exception as exc: + except RuntimeError as exc: raise ProviderUnavailableError( f"syntax provider failed for supported language {language!r}: {exc}; " - "verify the Agent Code Guard installation" + "verify the Agent Code Guard installation", + language=language, ) from exc diff --git a/src/agent_code_guard/analysis/regions.py b/src/agent_code_guard/analysis/regions.py index 3998597..23a718e 100644 --- a/src/agent_code_guard/analysis/regions.py +++ b/src/agent_code_guard/analysis/regions.py @@ -65,7 +65,9 @@ def executable_regions(path: Path, provider: ParserProvider) -> tuple[Executable def _vue_regions(path: Path, source: bytes, provider: ParserProvider) -> tuple[ExecutableRegion, ...]: root = provider.parse("vue", source).root_node if root.has_error: - raise SyntaxAnalysisError(f"unable to parse {path}: Vue container syntax tree contains errors") + raise SyntaxAnalysisError( + f"unable to parse {path}: Vue container syntax tree contains errors", language="vue", + ) regions: list[ExecutableRegion] = [] for element in root.named_children: if element.type != "script_element": @@ -73,7 +75,9 @@ def _vue_regions(path: Path, source: bytes, provider: ParserProvider) -> tuple[E start_tag = next(child for child in element.named_children if child.type == "start_tag") attributes = _attributes(start_tag, source) if "src" in attributes: - raise SyntaxAnalysisError(f"unable to analyze {path}: external Vue script regions are unsupported") + raise SyntaxAnalysisError( + f"unable to analyze {path}: external Vue script regions are unsupported", language="vue", + ) language = _script_language(path, attributes.get("lang")) raw_text = next((child for child in element.named_children if child.type == "raw_text"), None) if raw_text is not None: @@ -106,4 +110,6 @@ def _script_language(path: Path, value: str | None) -> str: return "javascript" if value in {"ts", "typescript"}: return "typescript" - raise SyntaxAnalysisError(f"unable to analyze {path}: unsupported Vue script language: {value}") + raise SyntaxAnalysisError( + f"unable to analyze {path}: unsupported Vue script language: {value}", language="vue", + ) diff --git a/src/agent_code_guard/code_guard.py b/src/agent_code_guard/code_guard.py index 70e07d7..3f3b021 100644 --- a/src/agent_code_guard/code_guard.py +++ b/src/agent_code_guard/code_guard.py @@ -17,6 +17,7 @@ from .human_output import format_completed_analysis from . import loc_baseline from .result_model import GuardResult, aggregate_state, required_policies +from .reporting import reporting_path from .skill_distribution import export_skill, skill_path as installed_skill_path DISTRIBUTION_NAME = "agent-code-guard" @@ -250,13 +251,31 @@ class ScopeSummary: analyzed: int inapplicable: int excluded: int + unavailable: int | None = None def to_json(self) -> dict[str, int]: - return { + data = { "selected": self.selected, "analyzed": self.analyzed, "inapplicable": self.inapplicable, - "excluded": self.excluded, + } + if self.unavailable is not None: + data["unavailable"] = self.unavailable + data["excluded"] = self.excluded + return data + + +@dataclass(frozen=True) +class UnavailableEntry: + path: str + language: str + kind: str + message: str + + def to_json(self) -> dict[str, str]: + return { + "path": self.path, "language": self.language, + "kind": self.kind, "message": self.message, } @@ -264,6 +283,8 @@ def to_json(self) -> dict[str, int]: class CompletedAnalysis: results: list[GuardResult] scope: ScopeSummary + unavailable: tuple[UnavailableEntry, ...] = () + incomplete_guard_ids: tuple[str, ...] = () def payload( @@ -273,12 +294,28 @@ def payload( if isinstance(analysis, list): analysis = CompletedAnalysis(analysis, ScopeSummary(0, 0, 0, 0)) results = analysis.results + completed_overall = aggregate_state(results) + incomplete = bool(analysis.unavailable) data = { - "overall": aggregate_state(results), + "overall": "incomplete" if incomplete else completed_overall, "scope": analysis.scope.to_json(), "requiredPolicies": required_policies(results), "guards": {result.guard_id: result.to_json() for result in results}, } + if incomplete: + data = { + "overall": "incomplete", + "completedOverall": completed_overall, + "scope": data["scope"], + "unavailable": [entry.to_json() for entry in analysis.unavailable], + "requiredPolicies": data["requiredPolicies"], + "guards": data["guards"], + } + unavailable_paths = [entry.path for entry in analysis.unavailable] + for guard_id, guard in data["guards"].items(): + guard["complete"] = guard_id not in analysis.incomplete_guard_ids + if not guard["complete"]: + guard["unavailablePaths"] = unavailable_paths if json_mode == "compact": for guard in data["guards"].values(): guard["findings"] = [ @@ -321,7 +358,8 @@ def run_analysis( if needs_analysis: analysis = import_module("agent_code_guard.analysis.pipeline") analyzed_files.update(path for path in scope.files if analysis.is_applicable(path)) - facts = analysis.analyze_files(scope.files) + batch = analysis.analyze_files_for_runner(scope.files) + facts = batch.facts if callable_size_config.enabled: results.append(callable_size.run(scope.root, callable_size_config, facts)) if nesting_config.enabled: @@ -345,9 +383,27 @@ def run_analysis( results.append(markdown_section_size.run(scope.root, markdown_section_config, _empty_markdown_facts())) selected = len(scope.files) analyzed = len(analyzed_files) + unavailable = tuple( + UnavailableEntry( + reporting_path(item.path, scope.root), item.language, item.kind, item.message, + ) + for item in (batch.unavailable if needs_analysis else ()) + ) + incomplete_guard_ids = tuple( + guard_id for guard_id, enabled in ( + ("callableSize", callable_size_config.enabled), + ("nesting", nesting_config.enabled), + ("complexity", complexity_config.enabled), + ) if enabled and unavailable + ) return CompletedAnalysis( results, - ScopeSummary(selected, analyzed, selected - analyzed, len(scope.excluded_files)), + ScopeSummary( + selected, analyzed, selected - analyzed, len(scope.excluded_files), + len({entry.path for entry in unavailable}) if unavailable else None, + ), + unavailable, + incomplete_guard_ids, ) @@ -362,6 +418,8 @@ def print_text(data: dict[str, object]) -> None: def exit_code(overall: str, ci: bool) -> int: + if overall == "incomplete": + return 3 if overall == "fail": return 2 if overall == "review" and not ci: diff --git a/src/agent_code_guard/human_output.py b/src/agent_code_guard/human_output.py index 5843fcb..c3c57e1 100644 --- a/src/agent_code_guard/human_output.py +++ b/src/agent_code_guard/human_output.py @@ -106,10 +106,25 @@ def _markdown_lines(data: dict[str, object]) -> list[str]: def format_completed_analysis(data: dict[str, object]) -> str: """Return the complete human report for an existing completed payload.""" scope = data["scope"] - lines = [ - f"{str(data['overall']).upper()}: {scope['selected']} selected; {scope['analyzed']} analyzed; " - f"{scope['inapplicable']} inapplicable; {scope['excluded']} excluded." - ] + if data["overall"] == "incomplete": + lines = [ + f"INCOMPLETE: {scope['selected']} selected; {scope['analyzed']} analyzed; " + f"{scope['inapplicable']} inapplicable; {scope['unavailable']} unavailable; " + f"{scope['excluded']} excluded. Completed findings: {str(data['completedOverall']).upper()}." + ] + lines.extend( + f"UNAVAILABLE: {item['path']} [{item['language']} {item['kind']}] - {item['message']}" + for item in data["unavailable"] + ) + incomplete_guards = [ + guard_id for guard_id, result in data["guards"].items() if not result["complete"] + ] + lines.append(f"Incomplete guards: {', '.join(incomplete_guards)}.") + else: + lines = [ + f"{str(data['overall']).upper()}: {scope['selected']} selected; {scope['analyzed']} analyzed; " + f"{scope['inapplicable']} inapplicable; {scope['excluded']} excluded." + ] lines.extend(_loc_lines(data)) lines.extend(_callable_size_lines(data)) lines.extend(_nesting_lines(data)) diff --git a/tests/helpers.py b/tests/helpers.py index 315941e..6bd5864 100644 --- a/tests/helpers.py +++ b/tests/helpers.py @@ -51,3 +51,11 @@ def read_json(self, result: subprocess.CompletedProcess[str]) -> dict[str, objec def findings(self, result: subprocess.CompletedProcess[str]) -> list[dict[str, object]]: return self.read_json(result)["guards"]["loc"]["findings"] + + def assert_syntax_unavailable(self, result: subprocess.CompletedProcess[str], path: str) -> None: + data = self.read_json(result) + self.assertEqual(result.returncode, 3) + self.assertEqual(data["overall"], "incomplete") + self.assertEqual(data["scope"]["unavailable"], 1) + self.assertEqual(data["unavailable"][0]["path"], path) + self.assertEqual(data["unavailable"][0]["kind"], "syntax") diff --git a/tests/test_callable_size.py b/tests/test_callable_size.py index 34672dd..57887c6 100644 --- a/tests/test_callable_size.py +++ b/tests/test_callable_size.py @@ -11,6 +11,7 @@ from helpers import CodeGuardTestCase, write_config from agent_code_guard.analysis import analyze_files +from agent_code_guard.analysis.pipeline import analyze_files_for_runner from agent_code_guard.code_guard import run_guards from agent_code_guard.file_selection import ResolvedScope from agent_code_guard.guards import callable_size @@ -153,7 +154,10 @@ def test_disabled_does_not_import_analysis_and_enabled_builds_facts_once(self) - "cyclomaticComplexity": {"enabled": False}, }) enabled_args = SimpleNamespace(**vars(args(config)), warn=None, fail=None, include=[], exclude=[], count_blank_lines=False, ignore_comment_lines=False) - with patch("agent_code_guard.analysis.pipeline.analyze_files", wraps=analyze_files) as analyze: + with patch( + "agent_code_guard.analysis.pipeline.analyze_files_for_runner", + wraps=analyze_files_for_runner, + ) as analyze: results = run_guards(scope, enabled_args) self.assertEqual(analyze.call_count, 1) self.assertIn("callableSize", [result.guard_id for result in results]) @@ -220,8 +224,8 @@ def test_enabled_malformed_source_is_exit_three(self) -> None: source.write_text("def broken(:\n pass\n", encoding="utf-8") config = write_config(root, {"enabled": False}, guards={"callableSize": {"enabled": True, "reviewAt": 80}}) result = self.run_guard(root, str(source), "--config", str(config), "--json") - self.assertEqual(result.returncode, 3) - self.assertIn("syntax tree contains errors", self.read_json(result)["error"]) + self.assert_syntax_unavailable(result, "broken.py") + self.assertIn("syntax tree contains errors", self.read_json(result)["unavailable"][0]["message"]) if __name__ == "__main__": diff --git a/tests/test_csharp_contextual_keywords.py b/tests/test_csharp_contextual_keywords.py index d1413cf..5850e57 100644 --- a/tests/test_csharp_contextual_keywords.py +++ b/tests/test_csharp_contextual_keywords.py @@ -101,8 +101,7 @@ def test_async_identifiers_and_named_arguments_preserve_callable_measurements(se "class UnsupportedRole { void Run() { async: return; } }\n", ) unsupported_result = self.run_guard(root, str(unsupported_role), "--json") - self.assertEqual(unsupported_result.returncode, 3) - self.assertEqual(set(self.read_json(unsupported_result)), {"error"}) + self.assert_syntax_unavailable(unsupported_result, "UnsupportedRole.cs") if __name__ == "__main__": diff --git a/tests/test_dart_named_constructor_identity.py b/tests/test_dart_named_constructor_identity.py index bcfe5b0..f6d8c17 100644 --- a/tests/test_dart_named_constructor_identity.py +++ b/tests/test_dart_named_constructor_identity.py @@ -224,8 +224,8 @@ def test_public_command_rejects_malformed_named_constructor(self) -> None: root = Path(temp) (root / "broken.dart").write_text("class C { C.named( { }", encoding="utf-8") result = self.run_guard(root, "broken.dart", "--json") - self.assertEqual(result.returncode, 3) - self.assertIn("syntax tree contains errors", self.read_json(result)["error"]) + self.assert_syntax_unavailable(result, "broken.dart") + self.assertIn("syntax tree contains errors", self.read_json(result)["unavailable"][0]["message"]) if __name__ == "__main__": diff --git a/tests/test_go_receiver_callback_identity.py b/tests/test_go_receiver_callback_identity.py index 5e1b9fa..946faa8 100644 --- a/tests/test_go_receiver_callback_identity.py +++ b/tests/test_go_receiver_callback_identity.py @@ -201,8 +201,7 @@ def test_malformed_go_is_public_exit_three(self) -> None: root = Path(temp) path = write_source(root, "package sample\nfunc Run() { _ = func( { }\n") result = self.run_guard(root, str(path), "--json") - self.assertEqual(result.returncode, 3) - self.assertEqual(set(self.read_json(result)), {"error"}) + self.assert_syntax_unavailable(result, path.name) if __name__ == "__main__": diff --git a/tests/test_nesting.py b/tests/test_nesting.py index 724dd5e..2201c3f 100644 --- a/tests/test_nesting.py +++ b/tests/test_nesting.py @@ -1,10 +1,7 @@ from __future__ import annotations -from contextlib import redirect_stdout -import io import json import os -import sys import tempfile import unittest from pathlib import Path @@ -14,8 +11,8 @@ from helpers import CodeGuardTestCase, write_config from agent_code_guard.analysis import analyze_files -from agent_code_guard.analysis.errors import ProviderUnavailableError -from agent_code_guard.code_guard import main, run_guards +from agent_code_guard.analysis.pipeline import analyze_files_for_runner +from agent_code_guard.code_guard import run_guards from agent_code_guard.file_selection import ResolvedScope from agent_code_guard.guards import nesting @@ -194,7 +191,8 @@ def test_analysis_activation_matrix_builds_shared_facts_once(self) -> None: for guards, expected_calls, expected_ids in configurations: config = write_config(root, {"enabled": False}, guards=guards) with self.subTest(guards=guards), patch( - "agent_code_guard.analysis.pipeline.analyze_files", wraps=analyze_files + "agent_code_guard.analysis.pipeline.analyze_files_for_runner", + wraps=analyze_files_for_runner, ) as analyze: results = run_guards(scope, args(config)) self.assertEqual(analyze.call_count, expected_calls) @@ -215,33 +213,6 @@ def test_all_syntax_guards_disabled_do_not_import_analysis(self) -> None: loader.assert_not_called() self.assertEqual([result.guard_id for result in results], ["loc"]) - def test_missing_provider_propagates_as_tool_error(self) -> None: - with tempfile.TemporaryDirectory() as temp: - root = Path(temp) - source = root / "sample.py" - source.write_text("def sample():\n pass\n", encoding="utf-8") - scope = ResolvedScope(root, (source,)) - config = write_config(root, {"enabled": False}, guards={"nesting": {"enabled": True, "reviewAt": 4}}) - with patch( - "agent_code_guard.analysis.pipeline.analyze_files", - side_effect=ProviderUnavailableError("provider unavailable"), - ), self.assertRaisesRegex(ProviderUnavailableError, "provider unavailable"): - run_guards(scope, args(config)) - - def test_default_provider_failure_is_cli_exit_three(self) -> None: - with tempfile.TemporaryDirectory() as temp: - source = Path(temp) / "sample.py" - source.write_text("def sample():\n pass\n", encoding="utf-8") - output = io.StringIO() - with patch.object(sys, "argv", ["code-guard", str(source), "--json"]), patch( - "agent_code_guard.analysis.pipeline.analyze_files", - side_effect=ProviderUnavailableError("provider unavailable"), - ), redirect_stdout(output): - result = main() - self.assertEqual(result, 3) - self.assertEqual(json.loads(output.getvalue()), {"error": "provider unavailable"}) - - class NestingRunnerTests(CodeGuardTestCase): def test_review_ci_json_shape_human_output_and_external_path(self) -> None: with tempfile.TemporaryDirectory() as root_temp, tempfile.TemporaryDirectory() as source_temp: @@ -319,8 +290,8 @@ def test_enabled_malformed_source_is_exit_three_but_disabled_is_loc_only(self) - source.write_text("def broken(:\n pass\n", encoding="utf-8") enabled = write_config(root, {"enabled": False}, guards={"nesting": {"enabled": True, "reviewAt": 4}}) result = self.run_guard(root, str(source), "--config", str(enabled), "--json") - self.assertEqual(result.returncode, 3) - self.assertIn("syntax tree contains errors", self.read_json(result)["error"]) + self.assert_syntax_unavailable(result, "broken.py") + self.assertIn("syntax tree contains errors", self.read_json(result)["unavailable"][0]["message"]) disabled = write_config(root, {"enabled": True, "warnAt": 10, "failAt": 20}, guards={ "callableSize": {"enabled": False}, "nesting": {"enabled": False}, "cyclomaticComplexity": {"enabled": False}, diff --git a/tests/test_pattern_guard_complexity.py b/tests/test_pattern_guard_complexity.py index 695afd7..cc90c18 100644 --- a/tests/test_pattern_guard_complexity.py +++ b/tests/test_pattern_guard_complexity.py @@ -275,8 +275,7 @@ def test_malformed_python_and_csharp_are_public_exit_three(self) -> None: for name, source in sources.items(): with self.subTest(name=name): result = self.run_guard(root, str(write_source(root, name, source)), "--json") - self.assertEqual(result.returncode, 3) - self.assertEqual(set(self.read_json(result)), {"error"}) + self.assert_syntax_unavailable(result, name) if __name__ == "__main__": diff --git a/tests/test_php_complexity.py b/tests/test_php_complexity.py index 8a66d20..7196aef 100644 --- a/tests/test_php_complexity.py +++ b/tests/test_php_complexity.py @@ -236,8 +236,8 @@ def test_malformed_php_is_public_exit_three(self) -> None: self.assertEqual((first.returncode, second.returncode), (3, 3)) payload = self.read_json(first) self.assertEqual(payload, self.read_json(second)) - self.assertEqual(set(payload), {"error"}) - self.assertIn("syntax tree contains errors", payload["error"]) + self.assert_syntax_unavailable(first, path.name) + self.assertIn("syntax tree contains errors", payload["unavailable"][0]["message"]) def test_below_threshold_php_complexity_requires_no_policy(self) -> None: with tempfile.TemporaryDirectory() as temp: diff --git a/tests/test_provider_failure_isolation.py b/tests/test_provider_failure_isolation.py new file mode 100644 index 0000000..c291de2 --- /dev/null +++ b/tests/test_provider_failure_isolation.py @@ -0,0 +1,189 @@ +from __future__ import annotations + +import tempfile +from pathlib import Path + +from tests.helpers import CodeGuardTestCase, write_config + +from agent_code_guard.analysis.pipeline import analyze_files_for_runner +from agent_code_guard.analysis.provider import TreeSitterProvider + + +class ProviderFailureIsolationLifecycleTests(CodeGuardTestCase): + def create_mixed_selection(self, root: Path) -> Path: + (root / "valid.py").write_text( + "def classify(value):\n" + " if value:\n" + " return 1\n" + " return 0\n", + encoding="utf-8", + ) + (root / "broken.py").write_text( + "def broken(:\n value = 1\n value = 2\n value = 3\n", encoding="utf-8", + ) + (root / "guide.md").write_text("# Guide\n\nUseful text.\n", encoding="utf-8") + loc_config = {"warnAt": 3, "failAt": 20} + return write_config(root, loc_config, guards={ + "loc": loc_config, + "markdownDocumentSize": {"reviewAt": 2}, + "markdownSectionSize": {"reviewAt": 1}, + }) + + def test_mixed_public_run_preserves_independent_results(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + config = self.create_mixed_selection(root) + + result = self.run_guard( + root, "valid.py", "broken.py", "guide.md", "--config", str(config), "--json", + ) + + self.assertEqual(result.returncode, 3, result.stderr) + data = self.read_json(result) + self.assertEqual(data["overall"], "incomplete") + self.assertEqual(data["completedOverall"], "review") + self.assertEqual(data["scope"], { + "selected": 3, "analyzed": 3, "inapplicable": 0, "unavailable": 1, "excluded": 0, + }) + self.assertEqual( + [(item["path"], item["language"], item["kind"]) for item in data["unavailable"]], + [("broken.py", "python", "syntax")], + ) + self.assertIn("embedded python syntax tree contains errors", data["unavailable"][0]["message"]) + self.assertEqual( + data["requiredPolicies"], ["loc", "markdownDocumentSize", "markdownSectionSize"], + ) + self.assertEqual(data["guards"]["loc"]["complete"], True) + for guard_id in ("callableSize", "nesting", "complexity"): + self.assertEqual(data["guards"][guard_id]["complete"], False) + self.assertEqual(data["guards"][guard_id]["unavailablePaths"], ["broken.py"]) + for guard_id in ("markdownDocumentSize", "markdownSectionSize"): + self.assertEqual(data["guards"][guard_id]["complete"], True) + self.assertEqual( + {item["path"] for item in data["guards"]["loc"]["findings"]}, + {"valid.py", "broken.py"}, + ) + self.assertTrue(data["guards"]["complexity"]["findings"]) + self.assertEqual(data["guards"]["markdownDocumentSize"]["findings"][0]["path"], "guide.md") + self.assertEqual(data["guards"]["markdownSectionSize"]["findings"][0]["path"], "guide.md") + + def test_all_output_and_ci_modes_retain_unavailable_context_and_exit_three(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + config = self.create_mixed_selection(root) + base = ("valid.py", "broken.py", "guide.md", "--config", str(config)) + json_payloads = {} + for ci in (False, True): + ci_args = ("--ci",) if ci else () + human = self.run_guard(root, *base, *ci_args) + self.assertEqual(human.returncode, 3, human.stderr) + lines = human.stdout.splitlines() + self.assertEqual( + lines[0], + "INCOMPLETE: 3 selected; 3 analyzed; 0 inapplicable; 1 unavailable; " + "0 excluded. Completed findings: REVIEW.", + ) + self.assertTrue(lines[1].startswith("UNAVAILABLE: broken.py [python syntax] - ")) + self.assertEqual(lines[2], "Incomplete guards: callableSize, nesting, complexity.") + self.assertTrue(any(line.startswith("REVIEW: valid.py ") for line in lines[3:])) + self.assertTrue(any(line.startswith("REVIEW: broken.py ") for line in lines[3:])) + self.assertTrue(any(line.startswith("REVIEW: guide.md ") for line in lines[3:])) + self.assertTrue(any("section \"Guide\"" in line for line in lines[3:])) + self.assertIn( + "Required policies: loc, markdownDocumentSize, markdownSectionSize", lines, + ) + for mode, mode_args in ( + ("full", ("--json",)), + ("debug", ("--json", "--json-mode", "debug")), + ("compact", ("--json", "--json-mode", "compact")), + ): + result = self.run_guard(root, *base, *ci_args, *mode_args) + self.assertEqual(result.returncode, 3, result.stderr) + json_payloads[(ci, mode)] = self.read_json(result) + expected = json_payloads[(False, "full")]["unavailable"] + self.assertTrue(all(data["unavailable"] == expected for data in json_payloads.values())) + self.assertEqual(json_payloads[(False, "debug")], json_payloads[(False, "full")]) + self.assertEqual(json_payloads[(True, "debug")], json_payloads[(True, "full")]) + + def test_one_file_unavailable_selection_is_structured(self) -> None: + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + (root / "broken.py").write_text("def broken(:\n", encoding="utf-8") + result = self.run_guard(root, "broken.py", "--json") + data = self.read_json(result) + self.assertEqual(result.returncode, 3) + self.assertEqual(data["scope"], { + "selected": 1, "analyzed": 1, "inapplicable": 0, "unavailable": 1, "excluded": 0, + }) + self.assertEqual(data["unavailable"][0]["path"], "broken.py") + + +class ProviderFailureIsolationPipelineTests(CodeGuardTestCase): + def test_provider_failure_records_each_path_and_continues_other_languages(self) -> None: + from tree_sitter_language_pack import get_parser + + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + first = root / "first.py" + second = root / "second.py" + java = root / "Example.java" + first.write_text("value = 1\n", encoding="utf-8") + second.write_text("value = 2\n", encoding="utf-8") + java.write_text("class Example { int value() { return 1; } }\n", encoding="utf-8") + + def factory(language: str): + if language == "python": + raise LookupError("missing python grammar") + return get_parser(language) + + result = analyze_files_for_runner( + [first, second, java], TreeSitterProvider(factory), + ) + self.assertEqual([item.path for item in result.facts.files], [java]) + self.assertEqual( + [(item.path, item.language, item.kind) for item in result.unavailable], + [(first, "python", "provider"), (second, "python", "provider")], + ) + self.assertTrue(all(item.message == result.unavailable[0].message for item in result.unavailable)) + + def test_arbitrary_concrete_parser_exception_still_aborts(self) -> None: + defect = ZeroDivisionError("programming defect") + + class BrokenParser: + def parse(self, source: bytes): + raise defect + + with tempfile.TemporaryDirectory() as temp: + path = Path(temp) / "sample.py" + path.write_text("value = 1\n", encoding="utf-8") + with self.assertRaises(ZeroDivisionError) as caught: + analyze_files_for_runner( + [path], TreeSitterProvider(parser_factory=lambda _: BrokenParser()), + ) + self.assertIs(caught.exception, defect) + + def test_concrete_parser_runtime_failure_is_unavailable(self) -> None: + class UnavailableParser: + def parse(self, source: bytes): + raise RuntimeError("runtime unavailable") + + with tempfile.TemporaryDirectory() as temp: + path = Path(temp) / "sample.py" + path.write_text("value = 1\n", encoding="utf-8") + + result = analyze_files_for_runner( + [path], TreeSitterProvider(parser_factory=lambda _: UnavailableParser()), + ) + + self.assertEqual(result.facts.files, ()) + self.assertEqual( + [(item.path, item.language, item.kind) for item in result.unavailable], + [(path, "python", "provider")], + ) + self.assertIn("runtime unavailable", result.unavailable[0].message) + + +if __name__ == "__main__": + import unittest + + unittest.main() diff --git a/tests/test_swift_do_catch.py b/tests/test_swift_do_catch.py index 4e7df4d..be1778d 100644 --- a/tests/test_swift_do_catch.py +++ b/tests/test_swift_do_catch.py @@ -283,8 +283,7 @@ def test_malformed_do_catch_is_public_exit_three(self) -> None: root = Path(temp) path = write_source(root, "broken.swift", "func broken() { do { try work() } catch {\n") result = self.run_guard(root, str(path), "--json") - self.assertEqual(result.returncode, 3) - self.assertEqual(set(self.read_json(result)), {"error"}) + self.assert_syntax_unavailable(result, path.name) if __name__ == "__main__": diff --git a/tests/test_switch_complexity.py b/tests/test_switch_complexity.py index 69b9199..22fae67 100644 --- a/tests/test_switch_complexity.py +++ b/tests/test_switch_complexity.py @@ -276,8 +276,7 @@ def test_malformed_classic_switch_is_public_exit_three(self) -> None: root = Path(temp) path = write_source(root, ".js", "function broken(x) { switch (x) { case 1:", "broken") result = self.run_guard(root, str(path), "--json") - self.assertEqual(result.returncode, 3) - self.assertEqual(set(self.read_json(result)), {"error"}) + self.assert_syntax_unavailable(result, path.name) if __name__ == "__main__":