Skip to content

Add optional hotpath profiling and reduce wheel audit/upload costs - #54

Merged
egorsmkv merged 1 commit into
masterfrom
perf/optional-hotpath
Oct 7, 2026
Merged

egorsmkv merged 1 commit into
masterfrom
perf/optional-hotpath

Conversation

@egorsmkv

@egorsmkv egorsmkv commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Uploads previously cloned the complete artifact payload and recomputed its SHA-256 after distribution inspection. Wheel audit heuristics also created payload-sized ASCII/lowercase strings, while ZIP extraction repeatedly grew entry buffers. This change transfers upload ownership into storage, reuses the inspected digest, reserves bounded ZIP entry sizes, and scans overlapping 64 KiB lowercase chunks with reusable memmem finders.

Adds optional profiling features using hotpath 0.28.5, covering wheel extraction/audit, validation/hash, upload/mirror paths and optional security scanners. Profiling is absent from the disabled dependency tree; normal builds keep their allocator. Explicit Argon2 std support makes its OS RNG available independently of unrelated transitive features. Includes a profiling example, deterministic fixture generator, reproduction instructions in docs/profiling.md, and PR CI for both feature modes with a JSON-report smoke run.

On the same synthetic 16 MiB binary wheel, five audits per process, optimized profiling builds and three process runs: median user+system CPU fell from 0.264 s to 0.092 s, elapsed time from 0.264 s to 0.093 s, and total allocated bytes from 560.9 MB to 81.5 MB. Both revisions had identical hotpath instrumentation. Allocated bytes are not peak RSS, and these results do not establish production throughput; YARA/FoxGuard/AST and upload are excluded from this isolated benchmark.

Validation: 61 application tests and 119 infrastructure tests passed in each mode; executable checks passed with profiling enabled and disabled. Differential tests preserve evidence priority, overlapping/boundary matches, short printable runs and invalid UTF-8 behaviour. An upload regression checks buffer ownership, digest and size at storage. ZIP limits remain enforced. Remaining full-payload buffering and synchronous CPU work inside async use cases are documented for a separate concurrency investigation.

@egorsmkv
egorsmkv merged commit 757227e into master Oct 7, 2026
7 checks passed
@egorsmkv
egorsmkv deleted the perf/optional-hotpath branch October 7, 2026 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant