Skip to content

feat(benchmark): harden harness, add small-host profile, refresh dependencies - #18

Merged
Coder-Blue merged 2 commits into
mainfrom
feat/hardening-graphify-small-host
Oct 20, 2026
Merged

Coder-Blue merged 2 commits into
mainfrom
feat/hardening-graphify-small-host

Conversation

@Coder-Blue

@Coder-Blue Coder-Blue commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR hardens the measurement harness, adds a supported small-host workflow, and refreshes the workspace and framework dependency matrix.

Graphify integration

  • Installed the Graphify skill for OpenCode and Codex.
  • Registered the project MCP server in .opencode/opencode.json and .codex/config.toml.
  • Added .opencode/plugins/graphify.js, .graphifyignore, and docs/graphify.md.
  • Built the code-only graph and used god-nodes, query, explain, and affected during the audit.

Correctness and resource lifecycle

  • BackgroundSampler now aborts its task in Drop and has an idempotent stop path, so an early error cannot leak a sampling task.
  • Managed containers are protected by an RAII guard; cancellation or panic now removes the target.
  • A failed compose up --wait cleans up the partially started database.
  • Deadline expiry is no longer counted as a server error in HTTP or WebSocket runs.
  • The sampler only attaches when a target PID or cgroup is actually available; it no longer reports the orchestrator's own RSS as target RSS.
  • Fixed the 100 Hz /proc/<pid>/stat fallback for PID-only targets.
  • RAPL now sums top-level package domains instead of double-counting nested zones.
  • Peak RSS is the maximum sampled window value, not the cgroup lifetime peak.
  • Readiness accepts any 2xx response and HTTP/1.0.

Security

  • Manifest validation rejects path traversal, uppercase/image-unsafe framework names, unsafe endpoint paths, invalid environment keys, and escaping Dockerfile paths.
  • Dockerfiles must be regular files under the framework directory; symlinked Dockerfiles are rejected.
  • Result categories are validated and all generated chart/table names go through a safe component helper.
  • Historical index loading rejects absolute, parent-directory, root, prefix, and non-normal paths.
  • Containers run with --init, dropped capabilities, no-new-privileges, PID/memory limits, and the no-log driver by default.
  • Benchmark actions are pinned to full commit SHAs; the self-hosted workflow has a timeout and an always-run cleanup step.
  • Small-host wrapper validates CPU, memory, cgroups v2, and non-overlapping CPU placement.
  • Autobahn target URLs are validated before JSON generation.

Performance and measurement

  • HDR recorders and worker buffers are allocated before measurement, removing per-worker startup churn from the measured window.
  • Removed unlocked Default histogram creation from production callers.
  • Removed unused bytes and http dependencies and narrowed Tokio features.
  • The target worker count now follows the target CPU set (BENCHMARK_TARGET_WORKERS overrides it), so workers no longer outnumber reserved CPUs.
  • Container logs default to none; logging is opt-in with BENCHMARK_CONTAINER_LOGS=1.
  • Small host uses 5-second warmup, 10-second measurement, 3-second cooldown, one target CPU, one load CPU, and one framework worker.
  • Closed-loop latency claims are now separated from paced, coordinated-omission-corrected runs in the architecture and acceptance docs.

Small-host support

  • Added scripts/run_small_host.sh with validation, bounded defaults, and a --check-only mode.
  • Added docs/small_host.md, docs/small_host_frameworks.md, and a weekly/downloadable benchmark_small_smoke.yml workflow.
  • Extended validate_environment.sh with profile checks.
  • Added BENCHMARK_CONCURRENCY, BENCHMARK_DURATION_SECS, BENCHMARK_TARGET_WORKERS, BENCHMARK_TARGET_HOST, BENCHMARK_TARGET_PID, and BENCHMARK_TARGET_CGROUP support.

Dependency refresh

  • Workspace: Rust 1.90, Tokio 1.54, toml 1.1, hdrhistogram 7.6, sha2 0.11; removed unused dependencies.
  • Frameworks updated across Rust, Go, C#, Java, JavaScript, Python, Ruby, PHP, C++, and Zig, including Fiber 3, ntex 4, Vert.x 5, .NET 10, Drogon 1.9.14, uWebSockets 20.80, and Dusty 0.4.
  • Docker base images, digests, CI toolchains, and lint images were refreshed together with the framework manifests.

Verification

Run locally on the branch:

  • nix develop --command cargo fmt --check
  • nix develop --command cargo clippy --workspace --all-targets -- -D warnings
  • nix develop --command cargo test --workspace --all-targets
  • sh scripts/check_conventions.sh
  • scripts/run_small_host.sh --check-only
  • Rust framework checks for actix-web and cpeak; ntex verified by the dependency sub-agent on the 1.99 toolchain
  • Go 1.26: gofmt, go mod tidy, go build ./..., go vet ./..., and endpoint smoke tests
  • Java 21: mvn package and java -jar endpoint smoke tests for all three adapters
  • .NET 10: dotnet build and dotnet format --verify-no-changes for all three adapters
  • JavaScript: Bun install/build for Elysia and Bun; npm install and endpoint smoke for Fulmine
  • Python: pinned venv installs and endpoint smoke for blacksheep, granian, and Robyn
  • Ruby 3.4: bundle install and endpoint smoke for agoo, falcon, and iodine
  • PHP 8.5: composer install and Workerman endpoint smoke; php -l for all adapters
  • Zig 0.16: zig fmt --check; Dusty musl build and endpoint smoke
  • nix build .#server-glibc and nix flake check --no-build pass
  • Graphify code-only graph built successfully (graphify update .)

Known limitations

  • Docker is not available in this environment, so not every glibc/musl image was rebuilt locally. Dockerfile tags and digests were resolved from the registries and the framework runtime checks above were run outside Docker where possible.
  • Robyn 0.88 publishes neither a musllinux wheel nor an sdist; the glibc path is verified and the limitation is documented in docs/small_host_frameworks.md.
  • C++ adapters were not compiled locally; their upstream tags/commits and base image digests were validated.
  • Small-host numbers are for adapter validation only and must not be published as comparative results.

Reviewer notes

The changes are split into small modules to preserve the 200-line ceiling. The main new boundaries are validation, validation_build, container_args, container_guard, container_paths, host_profile, suite_timing, session_results, reporter::artifacts, reporter::safety, and reporter::markdown::tables.

@Coder-Blue Coder-Blue added the enhancement New feature or request label Oct 20, 2026
@Coder-Blue Coder-Blue self-assigned this Oct 20, 2026
@Coder-Blue
Coder-Blue deployed to lint-python October 20, 2026 14:37 — with GitHub Actions Active
@Coder-Blue
Coder-Blue deployed to lint-csharp October 20, 2026 14:37 — with GitHub Actions Active
@Coder-Blue
Coder-Blue deployed to lint-javascript October 20, 2026 14:37 — with GitHub Actions Active
@Coder-Blue
Coder-Blue force-pushed the feat/hardening-graphify-small-host branch from 99a4181 to e9e4299 Compare October 20, 2026 14:39
@Coder-Blue
Coder-Blue deployed to lint-javascript October 20, 2026 14:40 — with GitHub Actions Active
@Coder-Blue
Coder-Blue deployed to lint-csharp October 20, 2026 14:40 — with GitHub Actions Active
@Coder-Blue
Coder-Blue deployed to lint-python October 20, 2026 14:40 — with GitHub Actions Active
- add Graphify project integration for OpenCode and Codex
- stop sampler tasks and managed containers on every exit path
- reject unsafe manifest paths and validate result categories
- fix deadline error accounting, PID CPU fallback, RAPL and RSS peaks
- preallocate histograms and remove unused hot-path dependencies
- add bounded 2 vCPU / 8 GB OpenStack wrapper, docs, matrix, CI smoke
- refresh workspace and framework dependencies, toolchains, and actions
@Coder-Blue
Coder-Blue force-pushed the feat/hardening-graphify-small-host branch from e9e4299 to 75e2a9b Compare October 20, 2026 14:45
@Coder-Blue
Coder-Blue deployed to lint-javascript October 20, 2026 14:45 — with GitHub Actions Active
@Coder-Blue
Coder-Blue deployed to lint-csharp October 20, 2026 14:45 — with GitHub Actions Active
@Coder-Blue
Coder-Blue deployed to lint-javascript October 20, 2026 15:44 — with GitHub Actions Active
@Coder-Blue
Coder-Blue deployed to lint-python October 20, 2026 15:44 — with GitHub Actions Active
@Coder-Blue
Coder-Blue deployed to lint-csharp October 20, 2026 15:44 — with GitHub Actions Active
@Coder-Blue
Coder-Blue merged commit c1a1d40 into main Oct 20, 2026
12 checks passed
@Coder-Blue
Coder-Blue deleted the feat/hardening-graphify-small-host branch October 20, 2026 16:09

This branch was successfully deployed

10 active deployments
lint-ruby — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via Ruby syntax #10
lint-go — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via Go format and vet #10
lint-cpp — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via C++ format #10
lint-java — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via Java compile #10
lint-javascript — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via JavaScript format #10
lint-python — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via Python ruff #10
lint-csharp — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via C# format #10
lint-rust — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via Rust format and clippy #10
lint-php — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via PHP syntax #10
lint-zig — 5025bd45 Deployed Oct 20, 2026 by Coder-Blue via Zig format #17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant