HA fails when using `runnerGroup`

[alaa-ledger]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

note: this is in a github enterprise 3.17.15 environment, I already raised a ticket to github support in case it's anissue on the GHE side, and not the runners.



1. Have a working ARC install (chart 0.14.1) with no `runnerGroup:` set —
   scale set lands in the Default enterprise runner group. Works fine,
   picks up jobs, has been running in production for us for ~2 months.

2. On a second independent cluster, install ARC per the [official HA docs](https://docs.github.com/en/enterprise-server@3.17/actions/how-tos/manage-runners/use-actions-runner-controller/deploy-runner-scale-sets#high-availability-and-automatic-failover):
   - same `runnerScaleSetName` as install #1
   - `runnerGroup:` set to a second enterprise runner group (created with
     All-repositories access to mirror Default)
   - identical values file otherwise

3. Trigger workflows targeting the pool.

Expected (per HA docs): jobs distribute between the two installs (assignment
race) and either install can independently serve the workload.

Observed:
- Install #2 registers cleanly with GHE and shows min-runners online in the
  UI for its group.
- ~100% of jobs continue to be picked up by install #1.
- Jobs explicitly pinned to install #2's group
  (`runs-on: {group: enterprise/<group>, labels: <name>}`) hang at
  "Waiting for a runner to pick up this job" indefinitely.
- Install #2's listener log shows long-polls to
  `/_services/pipelines/.../runnerscalesets/{ID}/messages` returning
  `unexpected EOF` at ~50s. After 5 consecutive EOFs the application
  exits with `giving up after 5 attempt(s)`. Kubernetes restarts the pod;
  the cycle repeats every ~5 minutes.

Install #1 (the working baseline) shows the same 50s `unexpected EOF`
pattern in its logs during quiet windows — but does not crash, because
real jobs keep arriving and reset the 5-attempt retry counter before it
exhausts.

We also tried install #2 with a unique `runnerScaleSetName` (still in the
second runner group) — same failure. So it is not caused by the two
installs sharing a name.

Describe the bug

Following the official active-active HA docs [1] does not produce a
working HA pair. The second install (the one with runnerGroup: set)
registers with GHE, shows online runners in the UI, but never receives
any jobs and its listener permanently crash-loops:

• 100% of jobs continue to go to the original install (in Default, no
  runnerGroup) — expected assignment-race skew, but the "standby" side
  never picks up any share of the load.
• Jobs pinned explicitly to the second install's runner group hang at
  "Waiting for a runner" indefinitely, despite the scale set showing
  online runners in that group.
• Second install's listener repeatedly exits with
  giving up after 5 attempt(s): unexpected EOF when its long-polls to
  /_services/pipelines/.../messages return EOF at ~50s.

The same 50s unexpected EOF appears in the working install's logs on
the same chart version and same server — it just doesn't kill that
listener because real jobs arrive often enough to reset the retry counter
before it exhausts. So the retry-exhaustion path only surfaces when the
scale set genuinely has no traffic, which is exactly the state the HA
"standby" is expected to sit in most of the time.

The GHE server acknowledges the second install (registration and
.../sessions succeed quickly). Only the message long-poll consistently
EOFs and never delivers.

[1] https://docs.github.com/en/enterprise-server@3.17/actions/how-tos/manage-runners/use-actions-runner-controller/deploy-runner-scale-sets#high-availability-and-automatic-failover

Describe the expected behavior

runners from either pools should pick up jobs, and using a runnerGroup should not cause the runners to never be picked up, no explicit reason/errors in the logs as to why

Additional Context

Environment:
- GitHub Enterprise Server 3.17.15 (self-hosted)
- Chart gha-runner-scale-set 0.14.1, gha-runner-scale-set-controller 0.14.1
- Internal scaleset library v0.3.0 (visible in log source paths)
- microk8s 1.35.6, kubernetes container hooks mode, running on a linux VM
- Enterprise-level runner scope

Working control on the same version and same GHE server: a scale set that
has been receiving continuous production traffic for ~2 months. Its
listener log shows the same `unexpected EOF` at ~50s during idle windows,
but the retry counter is reset by real job deliveries before it exhausts.

Controller Logs

https://gist.github.com/alaa-ledger/07e0eaf15719cd0d1bdbfbf1d44b9c34

Runner Pod Logs

https://gist.github.com/alaa-ledger/abc0d0a47db7745ad8bb18d310ccc4ff

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.