Skip to content

Measure the agent's real overhead, and write the policy Central needs - #18

Merged
lsmin3388 merged 9 commits into
mainfrom
chore/post-submission-hardening
Aug 25, 2026
Merged

Measure the agent's real overhead, and write the policy Central needs#18
lsmin3388 merged 9 commits into
mainfrom
chore/post-submission-hardening

Conversation

@lsmin3388

Copy link
Copy Markdown
Contributor

What changed

Post-0.2.0 hardening in three areas: an overhead measurement that can be
quoted, the policy work Maven Central publication depends on, and the project
files an outside contributor looks for.

Overhead is now measurable, and measured

The 2026-08-10 capture ran the baseline once, then the agent once, and reported
the agent as faster than the baseline. That is the shape a single-order run
produces when drift points the same way. It is retired rather than
reinterpreted.

capture-performance-evidence.sh now measures both modes over several rounds,
restarting each JVM every round and flipping which mode runs first, and reports
the median of the per-round paired differences.

That alone was not enough: the demo endpoint enters four instrumented methods,
too few to separate from noise. /auto/depth/{n} in the MVC sample walks a
chosen number of instrumented method entries, so the cost can be measured as a
slope rather than at one point.

Endpoint Instrumented entries Median Δ p50 Rounds the agent was slower
/auto/orders/1 ~4 −0.038 ms 2 / 6
/auto/depth/0 ~5 +0.010 ms 4 / 6
/auto/depth/1000 ~1005 +0.059 ms 6 / 6
/auto/depth/5000 ~5005 +0.138 ms 6 / 6

≈24 ns per instrumented method entry (≈50 ns cold, ≈20 ns warm), with the
fixed per-request cost inside the noise floor. The "rounds slower" column is
the honesty check — at four entries the agent lands on either side of the
baseline, so the −0.038 ms is noise and not a speed-up.
docs/15_performance_results.md lists what this
does not cover: concurrency, GC and allocation, startup cost.

A report from a newer schema is refused instead of misparsed

Found while writing the compatibility policy. The writer has always stamped
schemaVersion, but CoverageReportJson.read ignored it, so a future v2
document fed to a v1 build would parse into a silently wrong result. A report is
written by one CI job and read by another, and the two do not upgrade at the
same moment.

A document with no schemaVersion still reads as version 1 — that is the
only shape that has ever existed, and rejecting it would only break hand-written
documents.

Policy and project files

Korean versions of docs 15, 19, and 20 are included.

Checklist

  • ./gradlew test passes — 125 tests, 0 failures
  • ./gradlew build passes
  • New behavior is documented (README or docs/)
  • License or dependency changes are reflected in THIRD_PARTY_NOTICES.md — no dependency changes
  • No secrets, tokens, or .env files included

A single baseline-then-agent run cannot tell the agent's cost apart from
cache, JIT, scheduling, and thermal drift moving the same way. The
2026-08-10 capture showed the agent faster than the baseline, which is
the shape that mistake makes.

Measure both modes over several rounds instead, restarting each JVM every
round and flipping which mode runs first, then report the median of the
per-round paired differences.
Claiming a gap is only credible next to the tools that already fill the
space around it. Names OpenClover, the commercial test-impact products,
APM tracing, and traffic replay, states what each answers, and lists the
cases where one of them is the better answer.
The writer has always stamped schemaVersion, but the reader ignored it,
so a v2 document fed to a v1 build would parse into a silently wrong
result instead of an error. A report is written by one CI job and read by
another, and the two do not upgrade at the same moment.

A missing field still reads as version 1 — that is the only shape that
has ever existed, and rejecting it would only break hand-written
documents.
…eeds

Maven Central is permanent, so the compatibility promises attached to a
version have to exist before the first upload rather than after it. The
policy doubles as the acceptance criteria for #4.

GOVERNANCE records who decides what, how someone becomes a maintainer,
and what happens to the project if we stop — the question a contest
project fairly gets asked.
…d methods

The demo endpoint enters four instrumented methods, so measuring it
answers only what a request pays before any probe fires. Comparing two
depths on /auto/depth/{n} gives the per-entry cost, which is the number
someone deciding whether to run the agent actually needs.
Enables Discussions and points the issue chooser at Q&A and Show and
tell, so "does this fit my case" stops arriving as a bug report. SUPPORT
collects where to go and what a report we can act on contains.

CITATION.cff so the project can be cited without guessing at the
metadata.
The 2026-08-10 capture reported the agent as faster than the baseline,
which is what a single-order run produces when drift points the same way.
It is retired rather than reinterpreted.

Measures four endpoint shapes over six alternating rounds each. At four
instrumented entries the cost is inside the noise floor; at a thousand
and above every round agrees, giving about 24 ns per instrumented method
entry. Lists what the measurement does not cover — concurrency, GC,
startup — because those are where a surprise would come from.

Adds a roadmap, so the order we intend to work in is arguable in public.
@lsmin3388
lsmin3388 requested a review from TaeHuiKKIM as a code owner August 25, 2026 11:56
@lsmin3388
lsmin3388 merged commit 36c790f into main Aug 25, 2026
4 checks passed
@lsmin3388
lsmin3388 deleted the chore/post-submission-hardening branch August 25, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant