🚨 ArgoCD Deployment Failed: demo-payload #44

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   argocd-deployment-failure

   automated

   on Feb 11, 2026
2. dcasati

   added

   cluster-doctorTag and trigger the Cluster Doctor Custom AgentTag and trigger the Cluster Doctor Custom Agent

   on Feb 11, 2026
3. github-actions

   added a commit that references this issue on Feb 11, 2026

   Fix typo in order-service image name

   ...

   This issue will close once commit 29abadd is merged into the 'main' branch.

   29abadd
4. github-actions commented on Feb 11, 2026

   

   github-actionsbot

   on Feb 11, 2026 – with GitHub Actions

   Author

   More actions

   🔍 Diagnostic Summary

   Status: Root cause identified and fix PR created
   Severity: High - Application deployment failure
   Cluster Identity: ✅ Confirmed (aks-eastus2 in rg-anyscale-demo)

   ---

   🎯 Root Cause

   Typo in container image name preventing image pull from GitHub Container Registry

   The order-service deployment manifest contains a typo in the container image reference:

   • Line 196 in Act-3/argocd/apps/broken-aks-store-all-in-one.yaml
   • Current (incorrect): ghcr.io/azure-samples/aks-store-demo/order-servi**e**:2.1.0
   • Should be: ghcr.io/azure-samples/aks-store-demo/order-service:2.1.0

   The misspelled repository name "order-servie" does not exist on ghcr.io, causing the kubelet to fail with a 403 Forbidden error when attempting to pull the image.

   ---

   📊 Cluster Evidence

   Pod Status

   NAME                                READY   STATUS             RESTARTS   AGE
   order-service-5c6b567944-v5cfv      0/1     ImagePullBackOff   0          3m
   order-service-7585988949-hv6kj      0/1     ImagePullBackOff   0          2m45s
   

   Error from Events

   Failed to pull image "ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0": 
   failed to resolve reference "ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0": 
   failed to authorize: failed to fetch anonymous token: 
   unexpected status from GET request to https://ghcr.io/token: 403 Forbidden
   

   Deployment Health

   • ✅ RabbitMQ: Running and healthy (init container wait completed successfully)
   • ✅ MongoDB: Running and healthy
   • ✅ makeline-service: Running (1/1)
   • ✅ product-service: Running (1/1)
   • ✅ store-front: Running (1/1)
   • ✅ store-admin: Running (1/1)
   • ✅ virtual-customer: Running (1/1)
   • ✅ virtual-worker: Running (1/1)
   • ❌ order-service: 0/1 (ImagePullBackOff)

   ---

   🔧 Remediation

   Pull Request Created: #45
   Branch: fix/cluster-doctor/issue-44-typo-order-service-20260211

   Change:

   - image: ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0
   + image: ghcr.io/azure-samples/aks-store-demo/order-service:2.1.0

   ---

   ✅ Verification Steps

   After the PR is merged and ArgoCD syncs:

   1. Check pod status:

      kubectl get pods -n default -l app=order-service

      Expected: 1/1 Running

   2. Verify image pull:

      kubectl describe pod -n default -l app=order-service

      Expected: No ImagePullBackOff errors

   3. Check application health:

      kubectl get pods -n default

      Expected: All pods in Running state

   4. Test order service endpoint:

      kubectl port-forward -n default svc/order-service 3000:3000
      curl http://localhost:3000/health

      Expected: HTTP 200 response

   ---

   📝 Additional Notes

   • No cluster-wide issues detected - other microservices are healthy
   • No networking issues - init container successfully reached RabbitMQ
   • No RBAC issues - deployment has proper permissions
   • No resource constraints - sufficient CPU/memory available
   • Issue is isolated to the single typo in the manifest

   This is a simple configuration error that can be safely remediated via GitOps PR merge and ArgoCD resync. No manual cluster intervention required.

   ---

   Automated diagnosis completed by Cluster Doctor Agent
   Timestamp: 2026-02-11T07:03:58Z
5. github-actions commented on Feb 11, 2026

   

   github-actionsbot

   on Feb 11, 2026 – with GitHub Actions

   Author

   More actions

   🔄 Deployment Failed Again

   Timestamp: 2026-02-11T07:11:07.500Z
   Revision: def78ff78f604229ea472eb2ecb1a3f001fcec60

   Error:

   successfully synced (all tasks run)
   
6. dcasati

   closed this as not plannedon Feb 11, 2026