Skip to content

docs(perf): measure 0.10.0 against 0.9.2 - #1015

Open
devin-ai-integration[bot] wants to merge 3 commits into
developfrom
docs/perf-0.10.0-vs-0.9.2
Open

devin-ai-integration[bot] wants to merge 3 commits into
developfrom
docs/perf-0.10.0-vs-0.9.2

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What and why

The release-gate performance measurement of the 0.10.0 candidate against release 0.9.2. The candidate is hotfix/0.10.0-develop-sync at 82f0fac87 (#1024): main at the 0.10.0 release merge with develop's post-cut changes and the six performance fixes (#1017–#1023) folded in. The baseline is the v0.9.2 tag, built and benchmarked from its own worktree. main at ffe51571c — the release merge alone — was measured as a third revision and is shown as a column wherever a fix moved a row, so the record says what each fix bought.

Method as in the 0.9.2 against 0.9.1 record: every benchmark package on every revision under benchstat over six counts, every row in doubt re-run interleaved across the three, the stress constellation and tests/perf workloads at the previous record's sizes, and whole-binary sysml -validate wall time and RSS on generated models, the examples and the Apollo 11 model, start-up and binary size. A benchmark that fails on one revision prints no result line and the package's other rows still run; such rows are reported as candidate-only or baseline-only, nothing was excluded from -bench ..

Verdict: nothing here blocks the release. Nothing scales worse than it did; the whole binary validates a 12 000-declaration model in 0.28 s against 0.23 s. The six fixes cleared three of the regressions the release merge alone showed and cut a fourth by a quarter: CompiledCalc/Fib(25)/interpreted from 5.3× 0.9.2 to 20% faster than 0.9.2, BatchConstraints from +68% to ~, GRPCParseFileCached from 299 allocations back to 24 (0.9.2: 22), the workspace-edit rows from 3.8–3.9× 0.9.2's bytes to 2.9–3.0×. What remains, confirmed interleaved and attributed in the record:

  • loading +15–18% allocations and +10–16% time per element on connector-dense models (connector ends owned, journaled, indexed and reflected) — structural, linear;
  • executing an action or state step +20–60% with 1.3–1.5× the allocations (statement-order exploration, body-binding frames, held control values) — the release's semantics; the per-step constant is the next place to look;
  • editing beside a large document still 2.9–3.0× the bytes (the per-name snapshot of re-export claims) — the one row with an evident fix still open;
  • the import-ambiguity warning fires 134 times on the Annex A vehicle example; Analyze/vehicle +85%, ConnectParseFileHTTPCached 271 → 879 µs (no JSON cache on the HTTP path);
  • exact numerics: Collatz(27) +59–65%, Hypot/go 11 → 56 ns, Instantiate +11–31% per object;
  • for the release notes: SumTo(1000000) no longer fits the default step budget, the C target refuses Integer calcs, and the stress benchmark's per-document analysis trips the new decimal-rounding lint.

No regression is fixed here; this PR is the record only. It is linked from docs/internals/performance.md, docs/project/README.md and the mkdocs nav.

How it was verified

python3 scripts/check-doc-links.py (0 broken links), make docs-check, python3 scripts/changelog.py check. The measurements themselves: Xeon Platinum 8559C, 8 CPUs, 31 GiB, Go 1.25.11, GOMAXPROCS=8, the revisions never run concurrently; the tables are the benchstat rows with their p-values.

Checklist

  • make test and make lint pass locally (docs-only change; make docs-check passes)
  • Tests added or updated for the change (not applicable)
  • Documentation extended where it already covers the surface (see CONTRIBUTING.md)
  • Changelog entry added as changes/unreleased/<slug>.<section>.md, not as an edit to CHANGELOG.md
  • baselines regenerated and make docs-counts run if a gate count moved (none moved)
  • No internal work-item labels (waves, slices, F4, K5) in the body, docs, or changelog

Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/60e111c2f4c043b887f52d4aeadc556e
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/60e111c2f4c043b887f52d4aeadc556e?variant=devin
Requested by: @HuiJun

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review October 9, 2026 03:40

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 3 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

devin-ai-integration Bot and others added 2 commits October 9, 2026 17:58
…gainst 0.9.2

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…failing-row handling

Co-Authored-By: jason.han <hanhuijun@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant