Skip to content

perf: let idle workers take queued tasks off a busy peer, and move woken tasks to their waker by default - #1233

Merged
kacy merged 3 commits into
mainfrom
perf/pinned-steal
Oct 28, 2026
Merged

kacy merged 3 commits into
mainfrom
perf/pinned-steal

Conversation

@kacy

@kacy kacy commented Oct 28, 2026

Copy link
Copy Markdown
Owner

Stacked on #1232.

A task runs on the first worker that resumes it, and every wake sends it back there. So whether two tasks that talk to each other share a worker is luck, and on 16 workers the luck was expensive:

  • task_pingpong took 2.2-2.9 s in about one run in three instead of 400 ms, with all four million wakes crossing workers.
  • chan_fanout landed anywhere from 70 to 130 ms.

PITH_GREEN_MIGRATE=1 already fixed both by moving a woken task to the worker that wakes it. It stayed off because it pulls parallel compute onto one worker. On 16 workers it took cpu_parallel_sync from 911 ms to 2.6-4.7 s. docs/performance.md named the missing piece: letting an idle worker take a ready task off a peer's pinned queue.

This PR adds that piece and turns migration on by default:

  • An idle worker that finds no fresh work looks at its peers' pinned FIFOs. If a peer has queued tasks and its resume counter has not moved for 20 µs (it is inside one long task), the idle worker takes the oldest queued task and rewrites the task's owner to itself. A task in a pinned queue is Ready and held only by that queue, so popping it under the queue's lock gives the thief the same exclusive hand-over a migrating wake gets.
  • The thief never takes the affinity slot, which holds the other half of a handoff.
  • A peer's pinned queue does not count as work in has_any_work, so idle workers still park. When it did count, idle workers spun beside a busy pipeline: chan_fanout used 0.75 s of CPU for a 49 ms run instead of 0.037 s.
  • PITH_GREEN_MIGRATE=0 turns migration off.

Interleaved medians of 7 at 16 workers, b0cccee against this branch (which includes #1232):

row b0cccee this branch
chan_fanout 97 (70-130) 38
cpu_parallel_sync 921 658
lock_sweep/spawn 280 63
green_fanout 1088 869
task_pingpong 397, or 2.2-2.9 s 399

The other concurrency rows and http_head_read are within 3%. chan_fanout and task_pingpong now match their one-worker times.

HTTP server, migration off against the default, medians of three 5 s runs from a Go keep-alive client on the other CCD: 862k vs 863k req/s at 32 connections, 1173k vs 1157k at 256, and 1486k vs 1479k at 1024. The 256 row was 1-2% lower with migration on in all three rounds.

Checks:

  • Local corpus, leak, memcheck, green, result-box, cargo test, seed, TLS-barrier and panic gates pass.
  • The six examples/grpc_* programs give the same output as with migration off in 20 of 20 runs each.
  • New unit test: idle_peer_steals_from_a_busy_owners_pinned_queue.
  • The docs (concurrency, limitations, performance, bench README) now describe the new default.

@kacy
kacy changed the base branch from perf/sched-cache-lines to main October 28, 2026 14:22
# Conflicts:
#	cranelift/runtime/src/concurrency/green.rs
@kacy
kacy merged commit 15eb660 into main Oct 28, 2026
4 checks passed
@kacy
kacy deleted the perf/pinned-steal branch October 28, 2026 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant