azure: deployments not rediscovered after network returns if startup sync failed #57690

Activity

1. opencode-agent commented on Oct 21, 2026

   

   opencode-agentBot

   on Oct 21, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the clear report and the code pointers. I reproduced this on 2.1.1, and it still happens on the latest dev (4d59657).

   What I did (the script below does the same):

   1. Pointed <resource>.openai.azure.com at a local HTTPS stand-in for Azure's GET /openai/deployments?api-version=2022-12-01 endpoint, with the stand-in stopped at first to act as "network down", and started opencode serve with AZURE_API_KEY / AZURE_RESOURCE_NAME set.
   2. The server logged failed to sync Azure deployments ... Transport error ... Unable to connect, the same as your log.
   3. Started the stand-in ("network back") and waited 60 s. OpenCode sent no requests to it, so discovery was never retried.
   4. After restarting the server, it fetched the deployments right away.

   One limit: in my setup the test deployment wasn't matched to a catalog model, so I couldn't check it showing up in the model list. The missing retry itself is clear from the requests.

   As you found, the sync error is logged and dropped. Discovery only runs at startup and when the Azure account is switched, so for now opencode service restart after the network is back is the workaround.

   Related reports with the same no-retry pattern:

   • mcp: remote MCP servers that fail once are never retried until service restart #52237: remote MCP servers that fail once are never retried
   • server: GitHub Copilot model sync and opencode.json changes require manual service restart to take effect #53413: GitHub Copilot model sync isn't retried after a failure

   Reproduction script (repro.sh)

   #!/usr/bin/env bash
   # Repro for #57690: Azure deployment discovery is never retried after a failed startup sync.
   # Needs root (edits /etc/hosts, binds :443). Uses a local HTTPS mock of the Azure data-plane
   # deployments endpoint; the "network outage" is simply the mock not running yet.
   # OC: command to run opencode (default: `opencode` from PATH). For a source checkout:
   #   OC="bun --conditions=browser --preload=$REPO/packages/tui/node_modules/@opentui/solid/scripts/preload.js $REPO/packages/cli/src/index.ts"
   set -u
   OC=${OC:-opencode}
   RES=reprores
   HOST=$RES.openai.azure.com
   W=/workspace/repro/work
   PORT=4597
   rm -rf "$W"; mkdir -p "$W/proj" "$W/home"
   grep -q "$HOST" /etc/hosts || echo "127.0.0.1 $HOST" >> /etc/hosts
   openssl req -x509 -newkey rsa:2048 -nodes -keyout "$W/key.pem" -out "$W/cert.pem" -days 1 \
     -subj "/CN=$HOST" -addext "subjectAltName=DNS:$HOST" 2>/dev/null
   cat > "$W/mock.ts" <<EOF
   Bun.serve({ port: 443, tls: { cert: Bun.file("$W/cert.pem"), key: Bun.file("$W/key.pem") },
     fetch(req) {
       console.log("mock:", req.headers.get("user-agent")?.split("/")[0], req.method, new URL(req.url).pathname + new URL(req.url).search)
       return Response.json({ data: [{ id: "my-custom-deployment", model: "gpt-4.1-mini", status: "succeeded", object: "deployment" }] })
     } })
   EOF
   export HOME=$W/home XDG_CONFIG_HOME=$W/home/.config XDG_DATA_HOME=$W/home/.local/share \
     XDG_STATE_HOME=$W/home/.local/state XDG_CACHE_HOME=$W/home/.cache
   export OPENCODE_SERVER_PASSWORD=repro-pass
   export NODE_EXTRA_CA_CERTS=$W/cert.pem AZURE_API_KEY=dummy-key AZURE_RESOURCE_NAME=$RES
   cd "$W/proj"
   echo "== version: $($OC --version)"
   echo "== 1. 'network down': nothing listens on $HOST:443; start server"
   $OC serve --port $PORT --print-logs --log-level debug > "$W/serve.log" 2>&1 &
   SRV=$!
   for i in $(seq 1 60); do curl -s "http://127.0.0.1:$PORT" >/dev/null && break; sleep 1; done
   # Any client request loads the location and its provider plugins (as the TUI would).
   $OC models --server "http://127.0.0.1:$PORT" >/dev/null 2>&1
   sleep 12
   echo "== server log, Azure lines:"; grep -i "azure" "$W/serve.log" | cut -c1-400
   echo "== 2. 'network back': start mock Azure endpoint and wait 60s"
   bun "$W/mock.ts" > "$W/mock.log" 2>&1 &
   MOCK=$!
   sleep 2; curl -s "https://$HOST/openai/deployments?api-version=2022-12-01" --cacert "$W/cert.pem"; echo
   sleep 58
   echo "== requests the server sent to the mock after it came up (excluding our curl):"
   grep "mock:" "$W/mock.log" | grep -vc "mock: curl"; echo "   (0 = opencode never retried discovery after the failed startup sync)"
   echo "== Azure lines in server log:"; grep -i azure "$W/serve.log" | cut -c1-300
   echo "== 3. restart server with mock up (control)"
   kill $SRV; wait $SRV 2>/dev/null
   $OC serve --port $PORT --print-logs --log-level debug > "$W/serve2.log" 2>&1 &
   SRV=$!
   for i in $(seq 1 60); do curl -s "http://127.0.0.1:$PORT" >/dev/null && break; sleep 1; done
   $OC models --server "http://127.0.0.1:$PORT" >/dev/null 2>&1
   sleep 5
   echo "== discovery requests from opencode after restart:"; grep "mock:" "$W/mock.log" | grep -vc "mock: curl"
   echo "== Azure lines in restarted server log (deployments were fetched):"; grep -i azure "$W/serve2.log" | cut -c1-300
   kill $SRV $MOCK 2>/dev/null

   Run it: bash repro.sh
2. opencode-agent

   added

   reproduced

   on Oct 21, 2026
3. argszero commented on Oct 21, 2026

   

   argszero

   on Oct 21, 2026

   Contributor

   More actions

   I'd like to work on this.

   I read the code at the two places you linked — refresh() in packages/core/src/plugin/provider/azure.ts, where deployments(...) is wrapped in an Effect.catch that logs failed to sync Azure deployments and resolves to undefined, after which if (!found) return drops the result, and the two call sites that start discovery (once at plugin start, and on Credential.Event.Switched). I did not find a retry anywhere in that path, which matches your log and your expectation.

   The plan is to retry the discovery request rather than restructure discovery itself:

   • Wrap the deployments(...) call in an Effect.retry whose schedule is an exponential backoff bounded by a recursion cap, retried only while the failure is transient (transport/DNS/timeout), not on an auth or shape error. The repository already does exactly this shape in packages/core/src/mcp/client.ts (retrying: Schedule.max([Schedule.exponential, Schedule.recurs]) with a isTransient gate), and packages/core/src/plugin/provider/local.ts is precedent for a provider plugin using a schedule, so I would follow that idiom rather than invent a new one.
   • Keep the guard that compares against the already-loaded deployments and the connection key, so a retry that lands after the connection changed does not write a stale list — that logic stays as it is.
   • A test that drives refresh() with a transport that fails twice and then succeeds, asserting the model list is rebuilt once the third attempt returns deployments, plus one asserting a non-transient failure is not retried.

   One question before I build it, because it decides the shape. #52237 (MCP) and #53413 (Copilot sync) are the same no-retry problem in other subsystems and both are still open, so a retry that lives only in azure.ts fixes one of three symptoms. I am happy to do the contained azure fix — that is what this issue asks for — but if you would rather have a shared retry helper that all three call sites use, say so and I will scope it that way instead. I would rather ask now than have you redirect a finished patch.

   I also cannot reproduce the wake-from-sleep timing on a Linux sandbox, so my verification will be the induced-failure path above rather than your mobile-modem scenario; I will say so in the PR.
4. argszero commented on Oct 21, 2026

   

   argszero

   on Oct 21, 2026

   Contributor

   More actions

   Update: the fix is implemented and tested, and only my open-PR budget keeps it from being a PR.

   Branch azure-discovery-retry (commit 20265c7773) retries deployments(...) while the failure can pass — a transport error, a timeout, a 408/429, or a 5xx other than 501 — with a 1, 2, 4, 8, 16, 30 second ladder bounded at 8 attempts, about two minutes. A failure that cannot pass, such as a rejected key, still logs failed to sync Azure deployments once and stops, as it does today. A switch still interrupts discovery through the existing fiber handle, so a pending retry never answers the old connection.

   Two tests cover both arms: the transient one fails without the change and passes with it, and the negative one keeps a retried 401 from slipping in later. Nothing else in packages/core moves: I ran the suite with and without the patch and the failing test names differ only by that new test.

   I already have three open PRs, which is the limit here for a contributor without write access, so I will open this one as soon as one of them lands. If you would rather have the shared retry helper covering this together with #52237 and #53413, say so and I will rework it that way before opening — neither of those retries discovery today, so there is no existing shape to extract yet.
5. argszero commented on Oct 24, 2026

   

   argszero

   on Oct 24, 2026

   Contributor

   More actions

   The fix is now open as #58576 (c2cb638711, rebased onto current dev). It closes this issue.

   What it does: deployments(...) is retried on a 1, 2, 4, 8, 16, 30 second ladder bounded at 8 attempts, and only a failure that can pass is retried — a transport error, a timeout, a 408, a 429, or a 5xx other than 501. A rejected key still logs failed to sync Azure deployments once and stops, as it does today, so the warning you saw keeps its meaning. The retry sits before the existing Effect.catch that turns a failure into undefined, which is what would otherwise swallow the retried attempt, and a switch still interrupts discovery through the existing fiber handle so a pending retry cannot answer the previous connection.

   Two tests cover both arms, and the transient one fails on the base (22 pass, 1 fail) and passes with the change.
6. argszero

   added 2 commits that reference this issue on Oct 24, 2026

   fix(core): retry Azure discovery when the CLI cannot mint a token

   ...

   db86db4

   fix(core): give the Azure CLI the same retry window as the request

   ...

   183cb83
7. argszero commented on Oct 28, 2026

   

   argszero

   on Oct 28, 2026

   Contributor

   More actions

   Note on where the fix lives, so it is not lost with the closed pull request.

   The retry for Azure deployment discovery is on my fork as branch azure-discovery-retry, tip 183cb8358e. It is no longer an open pull request: accounts without write access here can keep at most three pull requests open at a time, and I closed it to give a different fix its first pickup window. The branch is unchanged and still applies cleanly to today's dev (b6c43a524265).

   What it does: packages/core/src/.../azure.ts calls the resource-manager deployment list once per discovery, so a single transient failure — a timeout, a 5xx, a dropped connection — discards the whole discovery and the provider disappears from the model list until the next poll. The branch adds a small bounded retry around that call, leaves the successful path untouched, and adds a test that fails without the retry and passes with it.

   If you (or a maintainer) would rather have this as a pull request, say so here and I will re-cut it as soon as a slot is free. If the behaviour has since been fixed upstream another way, tell me and I will drop the branch instead.