Skip to content

ci(azure): smoke test fails during revision rollout before the auth store is initialized #513

Description

@cristim

The Azure deploy's smoke test fails intermittently while the new Container Apps revision is still starting, then passes on the next deploy with no change.

Seen twice so far on dev:

  • 2026-10-04 23:22, run 37243528487: Test Deployment / Run smoke tests failed with auth_store: unhealthy, "Auth store ping failed: auth store not initialized". The same app was healthy minutes later (revision cudly-dev-72afbe00--0000755, health 200, admin_exists: true).
  • 2026-10-05 17:02, run 37344361657 (merge of test(ci): tighten destroy-script guards and fixtures #503): same step failed. The log shows Health check 1: passed, then Health check 2: failed ... auth store not initialized. The first check was answered by the old revision, the second hit the new one before it had finished initializing. The next Azure run on main (d636276, run 37344708065) passed.

Impact: a red deploy for what is a rollout race, so a red Azure run can't be told from a real failure without reading the log.

Fix: have the smoke test wait for the new revision to be ready before counting a failure, for example poll until the revision's health state is Healthy and traffic points at it, or require N consecutive passes, rather than failing on the first unhealthy response. Keep a hard timeout so a revision that never initializes still fails the step.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions