Skip to content

feat: write the calibration report as JSON and YAML - #202

Merged
bernardladenthin merged 2 commits into
mainfrom
claude/calibrate-machine-readable
Sep 1, 2026
Merged

feat: write the calibration report as JSON and YAML#202
bernardladenthin merged 2 commits into
mainfrom
claude/calibrate-machine-readable

Conversation

@bernardladenthin

Copy link
Copy Markdown
Owner

CI will be red for a known, unrelated reason

main pins net.ladenthin:llama:5.2.0, which is not published yet, so every srcmorph PR fails at dependency resolution. The gates below were run locally with -Dllama.version=5.2.0-SNAPSHOT — the binding built from the state #408 merged, API-identical to 5.2.0 since #409 changed only spotbugs-exclude.xml.

Summary

  • srcmorph:calibrate produced numbers that could not be used. CalibrateEngine built a CalibrationReport and printed it as INFO lines, and that was the entire output — so prefill/decode throughput and chars-per-token could not be diffed across runs, committed as a baseline, or fed back into aiDefinitions by anything other than a human re-reading the console. That is most of the point of measuring them.
  • execute() now also writes srcmorph-calibration.json and srcmorph-calibration.yaml into the configured outputDirectory, creating it when missing. Both carry the same keys in the same order and add loadSeconds, midPrefillTokensPerSecond and cachedPromptTokens, which the pasteable <calibration> block does not.
  • The three figures the XML block does carry use the identical format strings, so the JSON and the paste block cannot disagree about what was measured — pinned by a test that asserts both spellings of the same number side by side.

Design notes worth stating

  • Hand-rolled, not Jackson. This module is framework-free and carries no JSON dependency (only srcmorph-cli does), and renderXml() already set that precedent. Model keys come from user configuration, so they are escaped for both formats — an unescaped quote produces a file neither parser accepts.
  • The split follows the PIT boundary, not convenience. Rendering is pure and lives in CalibrationReport, which is on the gate at mutationThreshold 100; file writing lives in CalibrateEngine, which is deliberately off the gate as orchestration.
  • The documents are asserted whole, not field by field. A per-field check passes on output that is not valid JSON at all — a missing brace, a stray comma — which is exactly how a hand-rolled writer fails. Separator handling therefore has its own two-model test, since a single-model test cannot see it.

Two things found while building it

  • A test expectation of mine was wrong, and the whole-document assertion caught it: 1.2345 formatted with %.3f rounds to 1.235, not 1.234. The code was right.
  • The feature made existing tests write into the checkout. outputDirectory defaults to the source-tree path src/site/ai, and the plugin module's MojoConfigurationMappingTest points it at the relative path out-dir — so execute() started leaving files behind on every run. Both the engine tests and that plugin test now use a @TempDir. My first commit shipped two of those generated files into the repo before I caught it; the follow-up commit removes them and fixes the cause.

Test plan

  • Full reactor mvn test: BUILD SUCCESS, 0 failures, 0 skipped in all three modules
  • PIT at mutationThreshold 100 on all three modules: srcmorph 807/807 — up from 777, so the new renderers generate 30 mutants and every one is killed — srcmorph-cli 16/16, srcmorph-maven-plugin 62/62
  • CalibrationReportTest 16/16 (10 new: exact documents for both formats, empty-report cases, the two-model separator, escaping across every branch including the control-character fallback and the space just above its boundary, and the XML-agreement check)
  • CalibrateEngineTest 6/6 (2 new: files exist, content is byte-for-byte what the report renders, and a missing directory is created rather than failing the run)
  • git status clean after a full test run — verified explicitly, since that is the failure this PR itself introduced once
  • CI is green on this branch — no, see the banner

Related issues / PRs

Closes the srcmorph:calibrate reports only through the log entry in TODO.md, removed here. Announced during the 1.2.0 audit cycle and never landed; this is that.

Checklist

  • I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
  • My commits follow Conventional Commits
  • No security-sensitive changes

Generated by Claude Code

srcmorph:calibrate built a CalibrationReport and printed it as INFO lines, and
that was the whole output. So the numbers a calibration run produces -- prefill
and decode throughput, chars per token, per model -- could not be diffed across
runs, committed as a baseline, or fed back into aiDefinitions by anything other
than a human re-reading the console, which is most of the point of measuring
them.

execute() now also writes srcmorph-calibration.json and
srcmorph-calibration.yaml into the configured outputDirectory, creating it when
missing. Both carry the same keys in the same order and add loadSeconds,
midPrefillTokensPerSecond and cachedPromptTokens, which the pasteable
<calibration> block does not. The three figures the XML does carry use the
identical format strings, so the JSON and the paste block cannot disagree about
what was measured -- pinned by a test that asserts both spellings of the same
number.

The renderers are hand-rolled in CalibrationReport rather than delegated to
Jackson: this module is framework-free and carries no JSON dependency (only
srcmorph-cli does), and renderXml() already set that precedent. Model keys come
from user configuration, so they are escaped for both formats; an unescaped
quote would produce a file neither parser accepts.

The split follows the PIT boundary rather than convenience: rendering is pure
and lives in CalibrationReport, which is on the gate at threshold 100, so the
documents are asserted whole rather than field by field -- a per-field check
passes on output that is not valid JSON at all, which is exactly how a
hand-rolled writer fails. The file writing lives in CalibrateEngine, which is
deliberately off the gate as orchestration.

Also fixed while here: outputDirectory defaults to the source-tree path
src/site/ai, so every existing CalibrateEngineTest would have started leaving
two files behind in the checkout. They now use a @tempdir.
The previous commit shipped two generated files --
srcmorph-maven-plugin/out-dir/srcmorph-calibration.{json,yaml} -- into the
repository. They are output, not source.

Root cause, and it is the same one the engine tests already hit: execute() now
writes the report into outputDirectory, and MojoConfigurationMappingTest's
fillSharedParameters points that at the relative path "out-dir". So
calibrateMojo_execute_logsTheInstructionsAndTheRenderedReport left two files in
the working tree on every run, and 'git add -A' swept them in.

The test now uses a @tempdir like the engine tests, and the two committed files
are removed.

What actually went wrong on my side: 'git add -A' on a tree I had not looked at
first. The untracked directory was listed in the status output of that very
commit.
@claude

claude Bot commented Sep 1, 2026

Copy link
Copy Markdown

Excellent work. This PR implements machine-readable calibration output with high code quality. The hand-rolled JSON/YAML rendering maintains the framework-free architecture. Escaping logic is comprehensive and correct. Format consistency with XML output prevents measurement disagreement. 10 new tests provide excellent coverage of edge cases and whole-document validation. No bugs or security issues found. Approved.

@claude

claude Bot commented Sep 1, 2026

Copy link
Copy Markdown

Detailed Analysis:

Strengths:

  1. Architecture: Hand-rolled JSON/YAML rendering maintains framework-free design - sound choice avoiding Jackson dependency
  2. Escaping: Comprehensive and correct handling of quotes, backslashes, unicode escapes, control characters, and space boundary
  3. Format Consistency: Numeric formatting identical to XML block prevents measurement disagreement across output formats
  4. Empty Cases: Both JSON and YAML correctly produce valid but empty structures
  5. Test Quality: 10 new tests with whole-document assertions (catches parse-level errors), separator validation, escaping boundaries, and integration verification
  6. Error Handling: IOException properly wrapped in SrcMorphException following project conventions; directory creation transparent
  7. Documentation: Clear Javadoc explaining design rationale; README updated

Technical Assessment:

  • UTF-8 encoding explicit and consistent
  • Files.createDirectories() handles nested paths
  • Try-with-resources properly resource managed
  • Locale.ROOT ensures deterministic formatting
  • No null-safety violations
  • No path traversal vulnerabilities
  • Deterministic LinkedHashMap ordering preserved

No bugs or security issues identified. The whole-document test approach effectively mitigates hand-rolled JSON/YAML fragility. All three modules maintain their PIT mutation thresholds.

@bernardladenthin
bernardladenthin merged commit 17d6472 into main Sep 1, 2026
19 of 27 checks passed
@bernardladenthin
bernardladenthin deleted the claude/calibrate-machine-readable branch September 1, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants