Repository navigation
Offload artifact inspection and mirrored wheel audit with bounded CPU admission - #55
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upload inspection and mirrored-wheel extraction/audit previously ran synchronously inside async use cases, stalling the event loop under concurrent load. Move these stages to one shared, process-wide Rayon worker with asynchronous admission. Transfer existing owned payloads without adding a full-buffer clone; retain admission until actual worker completion even if its request is cancelled. Preserve inspection errors and the existing mirror audit log-and-continue policy.
Adds a focused concurrency benchmark, cancellation/panic/error regression test, raw measurements, and reproduction instructions in docs/artifact-concurrency.md. Storage/streaming APIs remain unchanged.
Synthetic wheel with ~16 MiB contents, concurrency 32, one async worker; median of three fresh processes, optional scanners excluded:
Tradeoff: on two async workers, deflated audit throughput falls from 106.39 to 65.79 jobs/s; worst heartbeat improves from 288.79 to 1.85 ms and RSS from 36.24 to 20.02 MiB. The conservative one-worker limit bounds active snapshots, not waiting input buffers, and does not establish safe production HTTP concurrency. Measurements isolate CPU stages rather than end-to-end HTTP throughput; YARA/FoxGuard/AST/Magika, network, storage writes, and mirror digest validation are excluded.
Validation: 62 application tests pass; all three final benchmark matrices pass their opt-in regression check; infrastructure library/examples check and optimized benchmark build pass with no default features; formatting, whitespace and Python syntax checks pass. Application Clippy passes with two existing Rust 1.99 lints allowed (double_must_use and unnecessary_sort_by). Existing no-default-feature test builds contain unrelated ungated AST/storage tests; full scanner/backend/platform test coverage is not claimed.