Ephemeral runner hangs forever after "Acknowledging runner request": acquirejob never returns, no Runner.Worker, no error, runner never exits

[perbu]

Describe the bug

An ephemeral runner (v2.337.0, ARC gha-runner-scale-set 0.14.2) receives a job, logs Acknowledging runner request '<id>', reloads credentials, logs one AAD Correlation ID for this token request: Unknown line, and then does nothing, ever:

• No Runner.Worker is spawned and no Worker_*.log is written.
• No error or retry is logged. RunServer never logs Catch exception during request or Back off ..., so no exception was raised: the await on acquirejob simply never completes.
• Runner.Listener stays alive (Sl, 0 % CPU) holding one ESTABLISHED, idle HTTPS connection. It does not resume long-polling the broker.
• Exactly 10 minutes later GitHub fails the job with "The self-hosted runner lost communication with the server...". No step ever started.
• The pod stays Running indefinitely (5.5 h for the oldest when we found it). ARC still counts it as a live replica, so each hung runner permanently removes one slot of capacity (see ARC impact below).

We hit this four times on 2026-09-02, in two pairs a few minutes apart within each pair. Dozens of other jobs on the same scale set ran normally before, between and after.

This is distinct from the existing "ghost ephemeral runner" reports, all of which get an HTTP error back from acquirejob and log it:

• Ephemeral runner survives a lost job assignment (409 Conflict) and becomes a ghost #4617 / PR Exit ephemeral runner when its job assignment is lost #4618: acquirejob returns 409, runner logs Skipping message Job and continues.
• Ephemeral Runners seem to get stuck when job is canceled or interrupted actions-runner-controller#4307: acquirejob returns 404 after a cancellation (job was canceled), same continue.
• Runner hangs in job cancellation redelivery loop on v2.336.0 and never exits (#4312 not fully fixed) #4569 / Self-hosted runner gets stuck in active state, blocking queued jobs across multiple repositories #4312: hang in the cancellation redelivery loop, with a running worker.
• Runner.Worker hangs before reading the job message (30s IPC timeout), ignores SIGINT/SIGTERM; ephemeral runner exits 0 and the assigned job is orphaned in queued #4598: worker is spawned but never reads the IPC message; listener exits 0.

Here nothing comes back at all. Reading the client code (below), the request has no deadline on the response body and no TCP keepalive, so a connection that dies after headers but before the body completes can block forever. The same day, another ARC 0.14.2 user reported 25 runners wedged with the same last log line during ~40 min of packet loss to the Actions broker (actions/actions-runner-controller#4396, comment of 2026-09-02), which is consistent with a network event on the path being the trigger.

To Reproduce

Not deterministic. Observed setup:

1. Org-level ARC scale set (minRunners: 3, maxRunners: 14), ephemeral runners, dind sidecar, GitHub App auth.
2. Normal job load. Two runners that acknowledge a request within a few minutes of each other hang at the same point; ~80 minutes later, two more.
3. GitHub fails the four jobs after 10 minutes; the four pods stay Running for hours.

Expected behavior

• acquirejob should complete or fail within a bounded time (Settings.SendTimeout, default 100 s), covering the response body as well as the headers.
• An ephemeral runner that has acknowledged a request and then cannot acquire it has no future. It should exit so the orchestrator can replace it, the way Exit ephemeral runners on broker acknowledge job-not-found #4540 already made it exit when the acknowledge itself returns job-not-found.
• A Runner.Listener sitting for hours with no worker, no error and no broker polling is a hang, whatever the cause.

Runner Version and Platform

• Runner: 2.337.0 (ghcr.io/actions/actions-runner:latest at the time), Linux x64 container. The relevant code is unchanged on main, so this is not specific to 2.337.0.
• ARC: gha-runner-scale-set 0.14.2, controller 0.14.2
• Kubernetes: k3s v1.33.13+k3s2, containerd 2.2.5, single bare-metal node (96 threads / 251 GiB), kernel 7.0.0
• Pod: dind via native sidecar (docker:dind as initContainer with restartPolicy: Always), runner container privileged: true, ACTIONS_RUNNER_HOOK_JOB_STARTED set, RUNNER_WAIT_FOR_DOCKER_IN_SECONDS=120. The dind sidecar was healthy in every affected pod.
• Auth: GitHub App, org-level (githubConfigUrl: https://github.com/<org>), runner group Default.

What's not working?

Occurrences (all 2026-09-02, UTC):

runner pod	acknowledged	GitHub failed job	pod still Running at
runner-c8bqp	14:11:37	14:21:49	19:40+
runner-x5swc	~14:15:37	~14:25	19:40+
runner-sjdqk	15:33:07	15:43:12	19:40+
runner-j69ht	15:39:58	15:50:03	19:40+

GitHub's annotation on all four jobs:

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

ARC recorded JobStarted for each runner (the EphemeralRunner status has jobId, workflowRunId and jobRepositoryName filled in), so the broker did consider the job dispatched.

State inside runner-c8bqp about 5.5 h after the acknowledge:

$ ps -eo pid,etime,stat,args
    PID     ELAPSED STAT COMMAND
      1    05:44:12 Ss   /bin/bash /home/runner/run.sh
     10    05:44:12 S    /bin/bash /home/runner/run-helper.sh
     45    05:44:12 Sl   /home/runner/bin/Runner.Listener run

$ ls /home/runner/_diag
Runner_20260902-135814-utc.log          # last write 14:11; no Worker_*.log exists

$ cat /proc/net/tcp6   # one ESTABLISHED connection to an Azure IP :443, nothing else

Impact on ARC

ARC's listener computes target = min(minRunners + TotalAssignedJobs, maxRunners). GitHub correctly stops counting the failed job, but the EphemeralRunnerSet still counts the hung pod as a live replica. With N hung pods, any job that arrives while assigned <= N never gets a runner because the target is "already met". With four hung pods and minRunners: 3 we had one job queued 46 min and another 60 min on a host that was 90 % idle. Both were only served when an unrelated burst pushed the target above the zombie count. Listener log during the second incident, repeating every ~50 s for an hour:

"Calculated target runner count" "assigned job"=1 decision=4 min=3 max=14 currentRunnerCount=4

Job Log Output

Empty. No step of any of the four jobs started; the only content is the annotation quoted above.

Runner and Worker's Diagnostic Logs

Complete tail of the runner container stdout for runner-c8bqp. Nothing follows for 5.5 h. The _diag/Runner_*.log ends at the same line. There is no Worker_*.log.

[RUNNER 2026-09-02 13:58:18Z INFO Terminal] WRITE LINE: Current runner version: '2.337.0'
[RUNNER 2026-09-02 13:58:18Z INFO Terminal] WRITE LINE: 2026-09-02 13:58:18Z: Listening for Jobs
[RUNNER 2026-09-02 13:58:18Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2026-09-02 14:11:37Z INFO BrokerMessageListener] Acknowledging runner request '1163817b-8f76-50d5-a595-c971db5244ea'.
[RUNNER 2026-09-02 14:11:41Z INFO ConfigurationStore] HasCredentials()
[RUNNER 2026-09-02 14:11:41Z INFO ConfigurationStore] stored True
[RUNNER 2026-09-02 14:11:41Z INFO CredentialManager] GetCredentialProvider
[RUNNER 2026-09-02 14:11:41Z INFO CredentialManager] Creating type OAuth
[RUNNER 2026-09-02 14:11:41Z INFO CredentialManager] Creating credential type: OAuth
[RUNNER 2026-09-02 14:11:41Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2026-09-02 14:11:41Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2026-09-02 14:11:41Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2026-09-02 14:11:41Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown

The other three pods show the same lines with their own request IDs.

We still have live hung pods and can provide a dotnet-dump / createdump of Runner.Listener, the full _diag logs, or run a debug build, on request.

Analysis (actions/runner at tag v2.337.0, commit 397b032; same on main)

The broker message loop handles a run-service job request inline, so a stuck acquire freezes the whole listener: no further broker polling, no exit.

src/Runner.Listener/Runner.cs:

1. L697 AcknowledgeMessageAsync (5 s timeout) logs Acknowledging runner request. Seen.
2. L699-704: an ephemeral runner exits if the acknowledge returns job-not-found (Exit ephemeral runners on broker acknowledge job-not-found #4540). Not our path.
3. L731-733 LoadCredentials(allowAuthUrlV2: true) + runServer.ConnectAsync(...). No HTTP.
4. L735 runServer.GetJobMessageAsync(...) issues POST .../acquirejob. This never returned.
5. L738-746 catch 404/409/422 and continue; L747-759 catch anything else, log Caught exception from acquiring job message and continue. Neither ran: nothing was logged.
6. L762 jobDispatcher.Run(...) never reached, hence no Runner.Worker.

GetJobMessageAsync (src/Runner.Common/RunServer.cs:63-72) wraps the call in RetryRequest (src/Runner.Common/RunnerService.cs:85-108), which logs Catch exception during request and Back off ... on any failure. Nothing was logged, so no exception was raised: the await never completed.

Why the await can never complete: the request timeout covers the headers but not the body.

• src/Sdk/WebApi/WebApi/RawHttpClientBase.cs:66: the constructor sets m_client.Timeout = TimeSpan.FromMilliseconds(-1.0) with the comment "Disable their timeout since we handle it ourselves".
• RawHttpClientBase.cs:214-217, 245: requests are sent with HttpCompletionOption.ResponseContentRead, so HttpClient.SendAsync buffers the whole body before returning, bounded only by that infinite timeout and the caller's cancellation token.
• The timeout "handled ourselves" lives in src/Sdk/Common/Common/RawHttpMessageHandler.cs: L141-143 tokenSource.CancelAfter(Settings.SendTimeout) (100 s by default, src/Runner.Sdk/Util/VssUtil.cs:79-86) applies to m_tokenProvider.GetTokenAsync (L159, which emitted the final AAD Correlation ID log line) and to m_messageInvoker.SendAsync(request, tokenSource.Token) (L171). A message handler returns as soon as the response headers arrive; the finally at L231 then disposes the token source and its timer. The body is read afterwards by HttpClient, outside that timer.
• The caller's token, messageQueueLoopTokenSource.Token, is only cancelled on shutdown or session restart. So the body read has no deadline at all.
• No TCP keepalive is configured on this client. The only keepalive code in the tree (src/Sdk/WebApi/WebApi/VssHttpClientBase.cs:1153-1163) goes through ServicePointManager.SetTcpKeepAlive, which is a no-op for SocketsHttpHandler on .NET 8 (the runner targets net8.0), and RawHttpClientBase, the client used for acquirejob, does not call it anyway.

Put together: if the connection carrying the acquirejob response dies after the status line and headers but before the body (the full job message, often tens to hundreds of KB) is complete, and the peer's RST/FIN never reaches the runner (half-open connection: middlebox or LB state loss, NAT table churn, path outage), the read blocks forever. No timer exists to cancel it and no keepalive probe exists to detect the dead peer, so the ephemeral runner sits with one ESTABLISHED idle socket and never exits. That is exactly the process state we observed. Two runners hanging within minutes of each other, twice in one day, fits a transient network event rather than a per-job trigger. We have not yet confirmed the blocked frame with a dump; happy to do so.

VssHttpClientBase has the same infinite m_client.Timeout at L76, so the broker long-poll client has the same exposure.

Workaround we deployed

A CronJob deletes any EphemeralRunner in phase Running with status.jobId set, pod age > 12 min, and no Runner.Worker process in the runner container on two consecutive checks. Deleting the EphemeralRunner makes ARC deregister the runner and backfill the slot. Others in actions/actions-runner-controller#4307 use activeDeadlineSeconds on the pod or a log-grepping liveness probe; neither catches this variant, since nothing is logged.

[amasen02]

Spot-on root cause analysis. This is a classic leak caused by delegating request timeouts exclusively to a delegating HttpMessageHandler while invoking HttpClient.SendAsync with HttpCompletionOption.ResponseContentRead.

Since RawHttpMessageHandler disposes its tokenSource in the finally block once headers are dispatched, the subsequent stream read performed by HttpClient to buffer the response content runs without any cancellation token other than the infinite/session-lifetime messageQueueLoopTokenSource.Token. On .NET 8 (SocketsHttpHandler), half-open TCP connections or dropped NAT table entries during the body transfer will stall the socket read indefinitely.

1. The Direct Fix in Runner.cs / RunServer.cs

The cleanest immediate fix is ensuring that the top-level acquire operation is bounded by an explicit linked cancellation token rather than relying on handler-level header timers.

In src/Runner.Listener/Runner.cs around line 735:

// Current:
var jobMessage = await runServer.GetJobMessageAsync(message.Body, messageQueueLoopTokenSource.Token);

// Fix: Bound the acquire operation so stream hangs cannot freeze the listener
using var acquireCts = CancellationTokenSource.CreateLinkedTokenSource(messageQueueLoopTokenSource.Token);
acquireCts.CancelAfter(TimeSpan.FromSeconds(120)); // or VssUtil.SendTimeout

try
{
    var jobMessage = await runServer.GetJobMessageAsync(message.Body, acquireCts.Token);
    ...
}
catch (OperationCanceledException) when (!messageQueueLoopTokenSource.IsCancellationRequested)
{
    Trace.Error("Timed out while waiting for acquirejob response body from server.");
    // This allows ephemeral runners to exit or retry cleanly rather than deadlocking ARC
    throw new TimeoutException("Timed out acquiring job message payload.");
}

2. SocketsHttpHandler Configuration in RawHttpClientBase

For .NET 8, SocketsHttpHandler supports native TCP keep-alive configuration that replaces the deprecated ServicePointManager calls:

var handler = new SocketsHttpHandler
{
    // Enable TCP KeepAlive at socket level
    KeepAlivePingDelay = TimeSpan.FromSeconds(30),
    KeepAlivePingTimeout = TimeSpan.FromSeconds(15),
    KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
    // Ensure PooledConnectionIdleTimeout doesn't keep stale dead sockets
    PooledConnectionIdleTimeout = TimeSpan.FromMinutes(2),
    ConnectTimeout = TimeSpan.FromSeconds(30)
};

Alternatively, switching RawHttpClientBase to use HttpCompletionOption.ResponseHeadersRead and streaming the deserialization directly via System.Text.Json (or reading the stream with the caller's cancellation token) prevents HttpClient from doing unbounded buffering before returning control to the caller.

3. Immediate ARC Mitigation Note

For anyone hitting this on ARC before this is patched upstream: configuring the runner pod's spec with livenessProbe checking TCP socket liveness or a dead-man timer on _diag/Runner_*.log timestamp delta (e.g. restart pod if no log update after JobStarted within 10 minutes) safely evicts the zombie pod without waiting for the 5-hour timeout.

[Nikolaichukr]

Just a comment for visibility, we hit the same issue today.

[profplum700]

We have matching evidence from persistent, non-ephemeral Linux x64 runners
running as systemd services, on v2.337.0. This is not confined to ARC or
one-job ephemeral registrations. The processes stayed active while GitHub
reported the registrations offline. Shutdown exposed the same acquirejob
chunked-response-body wait on three separate listener instances.

Three shutdown stacks

We independently reread these original host log ranges on 2026-09-08. Paths
and line ranges identify our retained evidence; these logs are not public URLs.

1. /opt/runners/fhah-tools-1/_diag/Runner_20260904-160059-utc.log,
   lines 36072–36084: ChunkedEncodingReadStream.CopyToAsyncCore →
   HttpContent.LoadIntoBufferAsyncCore → RawHttpClientBase.SendAsync →
   RunServiceHttpClient.GetJobMessageAsync.
2. /opt/runners/fhah-tools-2/_diag/Runner_20260904-191912-utc.log,
   lines 27184–27196: the same call chain. Operator shutdown is logged at
   line 27168, 2026-09-08T14:46:09Z.
3. /opt/runners/fhah-tools-2/_diag/Runner_20260908-144612-utc.log,
   lines 1785–1801: the same body-buffering chain, followed by the RunServer
   retry wrapper. Installed version 2.337.0 is recorded at line 10. The last
   AAD correlation entry is line 1762, 15:01:12Z; the next entry is operator
   shutdown at line 1763, 15:26:46Z. Retry/backoff and cancellation appear at
   lines 1832–1835 only after shutdown.

The shared chain, including the outer HttpClient body buffering, is:

System.Net.Http.HttpConnection.FillAsync
System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore
System.Net.Http.HttpContent.LoadIntoBufferAsyncCore
System.Net.Http.HttpClient.<SendAsync>g__Core
Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync
GitHub.Actions.RunService.WebApi.RunServiceHttpClient.GetJobMessageAsync
GitHub.Runner.Common.RunServer / RunnerService.RetryRequest

The timeline references outside the three stack ranges are retained from the
original incident capture/handoff. Both original listeners had an ESTABLISHED
socket despite stalled diagnostic progress. That socket state does not prove
missing keepalive, a particular NAT timeout, or the exact packet-level cause.

Recurrence after connectivity returned; four-second post-409 precursor

The original incident followed a GitHub reachability interruption. After
connectivity returned and the listeners were restarted, runner 2 completed
several jobs and then wedged again, without another observed partition.
In its new log:

• 15:01:08Z: acquirejob HTTP 409, followed by handled MissingKey / already
  acquired (lines 1751–1752).
• 15:01:11Z: another runner request acknowledgement (line 1753).
• 15:01:12Z: final AAD correlation message (line 1762), four seconds after
  the 409, followed by silence until shutdown at 15:26:46Z.

This is a useful trigger candidate, not proof that 409 causes the body stall.
The original runner-1 evidence does not establish the same 409 precursor.
Likewise, the observations exclude a continuously ongoing original partition
as the sole explanation; they do not prove every relevant service endpoint
was continuously healthy between samples.

Exact v2.337.0 timeout coverage

• RawHttpClientBase constructor
  explicitly disables HttpClient.Timeout.
• RawHttpClientBase send path
  uses ResponseContentRead: the outer client waits for body buffering, with
  the caller's cancellation token.
• RawHttpMessageHandler timeout scope
  applies Settings.SendTimeout to a linked CTS for the inner transport. Its
  finally block
  disposes that CTS when the handler returns, before outer buffering completes.
• RunServer acquisition
  retries exceptions; an indefinitely pending read never reaches that retry.
  The Listener call site
  supplies the message-loop token, not a per-acquisition deadline.
• OAuth correlation logging
  happens after token HTTP SendAsync returns. The final AAD line is not
  evidence that token HTTP itself is hung; the shutdown stacks identify the
  later acquirejob body read.

Supported preventive configuration check

No supported operator configuration or released upgrade identified as of
2026-09-08 supplies a deadline for this body read. Specifically:

• Latest stable is
  v2.337.0, published 2026-08-26T14:33:29Z, already our incident version.
• GITHUB_ACTIONS_RUNNER_HTTP_TIMEOUT
  changes the inner SendTimeout (100–1200 seconds), not its lifetime.
• RawClientHttpRequestSettings
  exposes no separate body timer. VSS_Client_Request_Timeout is DEBUG-only
  and changes that same SendTimeout, not the outer body-buffering deadline.
• The checked runner command settings
  and documented .NET networking runtime options
  supply no operator override of this client's explicit infinite timeout.
  Programmatic HttpClient/handler settings would require a source change;
  connection establishment, pool eviction and HTTP/2 liveness settings do not
  impose a deadline on this in-flight HTTP/1.1 chunked response body.
• The complete RawHttpClientBase, RawHttpMessageHandler, RunServer and VssUtil
  files are byte-identical between v2.337.0 and
  upstream main 602c0085328df8cb595fc2641d69f640a11377a4,
  checked during this investigation. No fix is present in those four files.

An end-to-end acquirejob deadline would be preferable to an external watchdog:
keep its cancellation source alive through body consumption, preserve normal
shutdown cancellation, and surface timeout into appropriate recovery. A useful
regression would return valid headers and then leave a chunked body incomplete;
assert bounded failure without cancelling the whole Listener.

We have not run a deterministic stalled-body reproducer or changed runner
binaries. Can maintainers confirm the supported mitigation, if one exists
outside the inspected configuration surface?

Posted by Claude on behalf of profplum700.

[tm1000]

Adding to this we hit this as well last week and then today same time as @Nikolaichukr! Leads me to believe there might have been an undocumented github blip at that time but this bug presented itself then

[perbu]

fwiw; I measured 80% packet loss inside the github network today resulting in all sort of weirdness and stalled connections.

[Karina5005]

Same for persistent, non-ephemeral Linux x64 runners running as systemd services, on v2.337.0