Repository navigation
docs(perf): measure 0.10.0 against 0.9.2 - #1015
Open
devin-ai-integration[bot] wants to merge 3 commits into
Open
devin-ai-integration[bot] wants to merge 3 commits into
devin-ai-integration[bot] wants to merge 3 commits into
Conversation
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Contributor
Author
There was a problem hiding this comment.
🔍 Devin Review: 3 flags
Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
5 of 6 tasks
…gainst 0.9.2 Co-Authored-By: jason.han <hanhuijun@gmail.com>
…failing-row handling Co-Authored-By: jason.han <hanhuijun@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The release-gate performance measurement of the 0.10.0 candidate against release 0.9.2. The candidate is
hotfix/0.10.0-develop-syncat82f0fac87(#1024):mainat the 0.10.0 release merge withdevelop's post-cut changes and the six performance fixes (#1017–#1023) folded in. The baseline is thev0.9.2tag, built and benchmarked from its own worktree.mainatffe51571c— the release merge alone — was measured as a third revision and is shown as a column wherever a fix moved a row, so the record says what each fix bought.Method as in the 0.9.2 against 0.9.1 record: every benchmark package on every revision under
benchstatover six counts, every row in doubt re-run interleaved across the three, the stress constellation andtests/perfworkloads at the previous record's sizes, and whole-binarysysml -validatewall time and RSS on generated models, the examples and the Apollo 11 model, start-up and binary size. A benchmark that fails on one revision prints no result line and the package's other rows still run; such rows are reported as candidate-only or baseline-only, nothing was excluded from-bench ..Verdict: nothing here blocks the release. Nothing scales worse than it did; the whole binary validates a 12 000-declaration model in 0.28 s against 0.23 s. The six fixes cleared three of the regressions the release merge alone showed and cut a fourth by a quarter:
CompiledCalc/Fib(25)/interpretedfrom 5.3× 0.9.2 to 20% faster than 0.9.2,BatchConstraintsfrom +68% to~,GRPCParseFileCachedfrom 299 allocations back to 24 (0.9.2: 22), the workspace-edit rows from 3.8–3.9× 0.9.2's bytes to 2.9–3.0×. What remains, confirmed interleaved and attributed in the record:Analyze/vehicle+85%,ConnectParseFileHTTPCached271 → 879 µs (no JSON cache on the HTTP path);Collatz(27)+59–65%,Hypot/go11 → 56 ns,Instantiate+11–31% per object;SumTo(1000000)no longer fits the default step budget, the C target refuses Integer calcs, and the stress benchmark's per-document analysis trips the new decimal-rounding lint.No regression is fixed here; this PR is the record only. It is linked from
docs/internals/performance.md,docs/project/README.mdand the mkdocs nav.How it was verified
python3 scripts/check-doc-links.py(0 broken links),make docs-check,python3 scripts/changelog.py check. The measurements themselves: Xeon Platinum 8559C, 8 CPUs, 31 GiB, Go 1.25.11,GOMAXPROCS=8, the revisions never run concurrently; the tables are thebenchstatrows with their p-values.Checklist
make testandmake lintpass locally (docs-only change;make docs-checkpasses)changes/unreleased/<slug>.<section>.md, not as an edit toCHANGELOG.mdmake docs-countsrun if a gate count moved (none moved)F4,K5) in the body, docs, or changelogLink to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/60e111c2f4c043b887f52d4aeadc556e
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/60e111c2f4c043b887f52d4aeadc556e?variant=devin
Requested by: @HuiJun