Measure the agent's real overhead, and write the policy Central needs - #18
Merged
Conversation
A single baseline-then-agent run cannot tell the agent's cost apart from cache, JIT, scheduling, and thermal drift moving the same way. The 2026-08-10 capture showed the agent faster than the baseline, which is the shape that mistake makes. Measure both modes over several rounds instead, restarting each JVM every round and flipping which mode runs first, then report the median of the per-round paired differences.
Claiming a gap is only credible next to the tools that already fill the space around it. Names OpenClover, the commercial test-impact products, APM tracing, and traffic replay, states what each answers, and lists the cases where one of them is the better answer.
The writer has always stamped schemaVersion, but the reader ignored it, so a v2 document fed to a v1 build would parse into a silently wrong result instead of an error. A report is written by one CI job and read by another, and the two do not upgrade at the same moment. A missing field still reads as version 1 — that is the only shape that has ever existed, and rejecting it would only break hand-written documents.
…eeds Maven Central is permanent, so the compatibility promises attached to a version have to exist before the first upload rather than after it. The policy doubles as the acceptance criteria for #4. GOVERNANCE records who decides what, how someone becomes a maintainer, and what happens to the project if we stop — the question a contest project fairly gets asked.
…d methods
The demo endpoint enters four instrumented methods, so measuring it
answers only what a request pays before any probe fires. Comparing two
depths on /auto/depth/{n} gives the per-entry cost, which is the number
someone deciding whether to run the agent actually needs.
Enables Discussions and points the issue chooser at Q&A and Show and tell, so "does this fit my case" stops arriving as a bug report. SUPPORT collects where to go and what a report we can act on contains. CITATION.cff so the project can be cited without guessing at the metadata.
The 2026-08-10 capture reported the agent as faster than the baseline, which is what a single-order run produces when drift points the same way. It is retired rather than reinterpreted. Measures four endpoint shapes over six alternating rounds each. At four instrumented entries the cost is inside the noise floor; at a thousand and above every round agrees, giving about 24 ns per instrumented method entry. Lists what the measurement does not cover — concurrency, GC, startup — because those are where a surprise would come from. Adds a roadmap, so the order we intend to work in is arguable in public.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Post-0.2.0 hardening in three areas: an overhead measurement that can be
quoted, the policy work Maven Central publication depends on, and the project
files an outside contributor looks for.
Overhead is now measurable, and measured
The 2026-08-10 capture ran the baseline once, then the agent once, and reported
the agent as faster than the baseline. That is the shape a single-order run
produces when drift points the same way. It is retired rather than
reinterpreted.
capture-performance-evidence.shnow measures both modes over several rounds,restarting each JVM every round and flipping which mode runs first, and reports
the median of the per-round paired differences.
That alone was not enough: the demo endpoint enters four instrumented methods,
too few to separate from noise.
/auto/depth/{n}in the MVC sample walks achosen number of instrumented method entries, so the cost can be measured as a
slope rather than at one point.
/auto/orders/1/auto/depth/0/auto/depth/1000/auto/depth/5000≈24 ns per instrumented method entry (≈50 ns cold, ≈20 ns warm), with the
fixed per-request cost inside the noise floor. The "rounds slower" column is
the honesty check — at four entries the agent lands on either side of the
baseline, so the −0.038 ms is noise and not a speed-up.
docs/15_performance_results.md lists what this
does not cover: concurrency, GC and allocation, startup cost.
A report from a newer schema is refused instead of misparsed
Found while writing the compatibility policy. The writer has always stamped
schemaVersion, butCoverageReportJson.readignored it, so a future v2document fed to a v1 build would parse into a silently wrong result. A report is
written by one CI job and read by another, and the two do not upgrade at the
same moment.
A document with no
schemaVersionstill reads as version 1 — that is theonly shape that has ever existed, and rejecting it would only break hand-written
documents.
Policy and project files
what a version promises, what counts as public API, how the report schema
evolves, and what happens when a release is broken. Central is permanent, so
this had to exist before the first upload. Doubles as the acceptance
criteria for Automate signed Maven publication for the reusable library modules #4.
OpenClover, the commercial test-impact products, APM tracing, and traffic
replay, including the cases where one of those is the better answer.
SUPPORT.md,
CITATION.cff.instead of collecting them as bug reports.
Korean versions of docs 15, 19, and 20 are included.
Checklist
./gradlew testpasses — 125 tests, 0 failures./gradlew buildpassesdocs/)THIRD_PARTY_NOTICES.md— no dependency changes.envfiles included