feat: write the calibration report as JSON and YAML - #202
Merged
Conversation
srcmorph:calibrate built a CalibrationReport and printed it as INFO lines, and that was the whole output. So the numbers a calibration run produces -- prefill and decode throughput, chars per token, per model -- could not be diffed across runs, committed as a baseline, or fed back into aiDefinitions by anything other than a human re-reading the console, which is most of the point of measuring them. execute() now also writes srcmorph-calibration.json and srcmorph-calibration.yaml into the configured outputDirectory, creating it when missing. Both carry the same keys in the same order and add loadSeconds, midPrefillTokensPerSecond and cachedPromptTokens, which the pasteable <calibration> block does not. The three figures the XML does carry use the identical format strings, so the JSON and the paste block cannot disagree about what was measured -- pinned by a test that asserts both spellings of the same number. The renderers are hand-rolled in CalibrationReport rather than delegated to Jackson: this module is framework-free and carries no JSON dependency (only srcmorph-cli does), and renderXml() already set that precedent. Model keys come from user configuration, so they are escaped for both formats; an unescaped quote would produce a file neither parser accepts. The split follows the PIT boundary rather than convenience: rendering is pure and lives in CalibrationReport, which is on the gate at threshold 100, so the documents are asserted whole rather than field by field -- a per-field check passes on output that is not valid JSON at all, which is exactly how a hand-rolled writer fails. The file writing lives in CalibrateEngine, which is deliberately off the gate as orchestration. Also fixed while here: outputDirectory defaults to the source-tree path src/site/ai, so every existing CalibrateEngineTest would have started leaving two files behind in the checkout. They now use a @tempdir.
The previous commit shipped two generated files --
srcmorph-maven-plugin/out-dir/srcmorph-calibration.{json,yaml} -- into the
repository. They are output, not source.
Root cause, and it is the same one the engine tests already hit: execute() now
writes the report into outputDirectory, and MojoConfigurationMappingTest's
fillSharedParameters points that at the relative path "out-dir". So
calibrateMojo_execute_logsTheInstructionsAndTheRenderedReport left two files in
the working tree on every run, and 'git add -A' swept them in.
The test now uses a @tempdir like the engine tests, and the two committed files
are removed.
What actually went wrong on my side: 'git add -A' on a tree I had not looked at
first. The untracked directory was listed in the status output of that very
commit.
bernardladenthin
had a problem deploying
to
maven-central
September 1, 2026 14:05 — with
GitHub Actions
Failure
bernardladenthin
had a problem deploying
to
maven-central
September 1, 2026 14:05 — with
GitHub Actions
Failure
|
Excellent work. This PR implements machine-readable calibration output with high code quality. The hand-rolled JSON/YAML rendering maintains the framework-free architecture. Escaping logic is comprehensive and correct. Format consistency with XML output prevents measurement disagreement. 10 new tests provide excellent coverage of edge cases and whole-document validation. No bugs or security issues found. Approved. |
|
Detailed Analysis: Strengths:
Technical Assessment:
No bugs or security issues identified. The whole-document test approach effectively mitigates hand-rolled JSON/YAML fragility. All three modules maintain their PIT mutation thresholds. |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI will be red for a known, unrelated reason
mainpinsnet.ladenthin:llama:5.2.0, which is not published yet, so every srcmorph PR fails at dependency resolution. The gates below were run locally with-Dllama.version=5.2.0-SNAPSHOT— the binding built from the state #408 merged, API-identical to 5.2.0 since #409 changed onlyspotbugs-exclude.xml.Summary
srcmorph:calibrateproduced numbers that could not be used.CalibrateEnginebuilt aCalibrationReportand printed it asINFOlines, and that was the entire output — so prefill/decode throughput and chars-per-token could not be diffed across runs, committed as a baseline, or fed back intoaiDefinitionsby anything other than a human re-reading the console. That is most of the point of measuring them.execute()now also writessrcmorph-calibration.jsonandsrcmorph-calibration.yamlinto the configuredoutputDirectory, creating it when missing. Both carry the same keys in the same order and addloadSeconds,midPrefillTokensPerSecondandcachedPromptTokens, which the pasteable<calibration>block does not.Design notes worth stating
srcmorph-clidoes), andrenderXml()already set that precedent. Model keys come from user configuration, so they are escaped for both formats — an unescaped quote produces a file neither parser accepts.CalibrationReport, which is on the gate atmutationThreshold100; file writing lives inCalibrateEngine, which is deliberately off the gate as orchestration.Two things found while building it
1.2345formatted with%.3frounds to1.235, not1.234. The code was right.outputDirectorydefaults to the source-tree pathsrc/site/ai, and the plugin module'sMojoConfigurationMappingTestpoints it at the relative pathout-dir— soexecute()started leaving files behind on every run. Both the engine tests and that plugin test now use a@TempDir. My first commit shipped two of those generated files into the repo before I caught it; the follow-up commit removes them and fixes the cause.Test plan
mvn test:BUILD SUCCESS, 0 failures, 0 skipped in all three modulesmutationThreshold100 on all three modules:srcmorph807/807 — up from 777, so the new renderers generate 30 mutants and every one is killed —srcmorph-cli16/16,srcmorph-maven-plugin62/62CalibrationReportTest16/16 (10 new: exact documents for both formats, empty-report cases, the two-model separator, escaping across every branch including the control-character fallback and the space just above its boundary, and the XML-agreement check)CalibrateEngineTest6/6 (2 new: files exist, content is byte-for-byte what the report renders, and a missing directory is created rather than failing the run)git statusclean after a full test run — verified explicitly, since that is the failure this PR itself introduced onceRelated issues / PRs
Closes the
srcmorph:calibrate reports only through the logentry inTODO.md, removed here. Announced during the 1.2.0 audit cycle and never landed; this is that.Checklist
CONTRIBUTING.mdandCODE_OF_CONDUCT.mdGenerated by Claude Code