Repository navigation
Add optional hotpath profiling and reduce wheel audit/upload costs - #54
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uploads previously cloned the complete artifact payload and recomputed its SHA-256 after distribution inspection. Wheel audit heuristics also created payload-sized ASCII/lowercase strings, while ZIP extraction repeatedly grew entry buffers. This change transfers upload ownership into storage, reuses the inspected digest, reserves bounded ZIP entry sizes, and scans overlapping 64 KiB lowercase chunks with reusable memmem finders.
Adds optional
profilingfeatures using hotpath 0.28.5, covering wheel extraction/audit, validation/hash, upload/mirror paths and optional security scanners. Profiling is absent from the disabled dependency tree; normal builds keep their allocator. Explicit Argon2stdsupport makes its OS RNG available independently of unrelated transitive features. Includes a profiling example, deterministic fixture generator, reproduction instructions indocs/profiling.md, and PR CI for both feature modes with a JSON-report smoke run.On the same synthetic 16 MiB binary wheel, five audits per process, optimized profiling builds and three process runs: median user+system CPU fell from 0.264 s to 0.092 s, elapsed time from 0.264 s to 0.093 s, and total allocated bytes from 560.9 MB to 81.5 MB. Both revisions had identical hotpath instrumentation. Allocated bytes are not peak RSS, and these results do not establish production throughput; YARA/FoxGuard/AST and upload are excluded from this isolated benchmark.
Validation: 61 application tests and 119 infrastructure tests passed in each mode; executable checks passed with profiling enabled and disabled. Differential tests preserve evidence priority, overlapping/boundary matches, short printable runs and invalid UTF-8 behaviour. An upload regression checks buffer ownership, digest and size at storage. ZIP limits remain enforced. Remaining full-payload buffering and synchronous CPU work inside async use cases are documented for a separate concurrency investigation.