Skip to content

perf(aws): replace channel-based last-used fetch with atomic worker pool#5

Open
watany-dev wants to merge 1 commit into
mainfrom
claude/optimize-processing-performance-ikkv89
Open

perf(aws): replace channel-based last-used fetch with atomic worker pool#5
watany-dev wants to merge 1 commit into
mainfrom
claude/optimize-processing-performance-ikkv89

Conversation

@watany-dev

Copy link
Copy Markdown
Owner

ListRoles hydrated each role's last-used timestamp through a worker pool
built on two len(roles)-sized channels plus a dedicated collector
goroutine, and always spawned 55 goroutines regardless of role count.

Extract the concurrency into hydrateLastUsed, which hands out work via an
atomic index counter and lets each worker write straight into its own
slice index. This removes both channels and the collector goroutine,
caps worker count at min(concurrency, len(roles)), and short-circuits
empty input before building the progress bar.

Benchmarks (pkg/aws, in-memory fetch):

  • orchestration overhead: 3.3-4.5x faster, ~2.4x less memory
  • under simulated API latency: equal wall-clock, ~2.2x less memory

Adds correctness tests and head-to-head benchmarks against the previous
channel-based implementation so the production code path is measurable.

ListRoles hydrated each role's last-used timestamp through a worker pool
built on two len(roles)-sized channels plus a dedicated collector
goroutine, and always spawned 55 goroutines regardless of role count.

Extract the concurrency into hydrateLastUsed, which hands out work via an
atomic index counter and lets each worker write straight into its own
slice index. This removes both channels and the collector goroutine,
caps worker count at min(concurrency, len(roles)), and short-circuits
empty input before building the progress bar.

Benchmarks (pkg/aws, in-memory fetch):
  - orchestration overhead: 3.3-4.5x faster, ~2.4x less memory
  - under simulated API latency: equal wall-clock, ~2.2x less memory

Adds correctness tests and head-to-head benchmarks against the previous
channel-based implementation so the production code path is measurable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants