Skip to content

perf(sync): batch file uploads into single HTTP requests per batch#2985

Open
tusharmath wants to merge 2 commits intomainfrom
sync-batching
Open

perf(sync): batch file uploads into single HTTP requests per batch#2985
tusharmath wants to merge 2 commits intomainfrom
sync-batching

Conversation

@tusharmath
Copy link
Copy Markdown
Collaborator

@tusharmath tusharmath commented Apr 13, 2026

Summary

Replace per-file streaming uploads with batched HTTP requests to reduce server round-trips and improve sync throughput.

Context

The previous implementation used futures::stream::iter(...).buffer_unordered(batch_size) to upload each file in its own HTTP request, with up to batch_size requests in-flight concurrently. While this bounded memory, it meant N files produced N HTTP requests, creating significant overhead for large workspaces. Streaming parallelism also made progress tracking and error handling more complex.

Changes

  • Replaced buffer_unordered parallel streaming with chunks(batch_size).then(...) sequential batching
  • Each batch of batch_size files is now collected into a single FileUpload payload and sent in one HTTP request
  • Batches are processed sequentially — only one HTTP request is in-flight at a time
  • The stream now yields the count of files uploaded per batch (rather than 1 per file)
  • Updated posthog-rs dependency to a newer revision

Key Implementation Details

The upload stream is now pinned (Box::pin) since the then combinator returns an impl Stream rather than the previously inferred concrete stream type. Peak memory usage remains bounded to batch_size × avg_file_size — identical to before — but the number of HTTP requests drops from N to ceil(N / batch_size).

Use Cases

  • Syncing a workspace with hundreds of files now completes with far fewer round-trips to the server
  • Sequential batching eliminates the race conditions and ordering complexity of concurrent uploads
  • Easier to reason about progress: each stream item represents a whole batch

Testing

# Run sync service tests
cargo insta test --accept -p forge_services

# Run a full workspace sync
cargo run -- workspace info

@github-actions github-actions bot added ci: benchmark Runs benchmarks type: performance Improved performance. labels Apr 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: benchmark Runs benchmarks type: performance Improved performance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant