🚨 ArgoCD Deployment Failed: tech-connect-live #49

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   argocd-deployment-failure

   automated

   on Feb 11, 2026
2. dcasati

   added

   cluster-doctorTag and trigger the Cluster Doctor Custom AgentTag and trigger the Cluster Doctor Custom Agent

   on Feb 11, 2026
3. github-actions

   added a commit that references this issue on Feb 11, 2026

   Fix image name typo for order-service deployment

   ...

   8437d4d
4. github-actions commented on Feb 11, 2026

   

   github-actionsbot

   on Feb 11, 2026 – with GitHub Actions

   Author

   More actions

   🔍 Cluster Doctor Diagnostic Report

   Issue Status: ✅ ROOT CAUSE IDENTIFIED & FIX CREATED

   ---

   Executive Summary

   The ArgoCD deployment failure for tech-connect-live is caused by a typo in the container image name for the order-service deployment. The application synced successfully, but pods are failing with ImagePullBackOff because the specified image does not exist.

   ---

   🎯 Root Cause Analysis

   Verified Symptoms

   • Application Status: Synced but Degraded
   • Pod Status: 0/1 ImagePullBackOff for order-service pods
   • Error Duration: 70+ minutes of repeated pull failures
   • Affected Resources: 2 order-service replica pods

   Evidence Collected

   1. Cluster Verification (Identity Confirmed)

   • Cluster: aks-eastus2 in resource group rg-anyscale-demo ✅
   • Server: https://kubernetes.default.svc ✅
   • Namespace: default ✅

   2. Pod Status Analysis

   NAME                              READY   STATUS             RESTARTS
   order-service-75db49898-vrlq8     0/1     ImagePullBackOff   0
   order-service-b89d58f55-bnrj5     0/1     ImagePullBackOff   0
   

   3. Image Pull Error

   Error: ImagePullBackOff
   Back-off pulling image "ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0"
   

   4. Root Cause - Typo in Image Name

   • File: Act-3/argocd/apps/broken-aks-store-all-in-one.yaml:196
   • Incorrect: ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0 ❌
   • Correct: ghcr.io/azure-samples/aks-store-demo/order-service:2.1.0 ✅
   • Issue: Missing "c" in "service" → "servie"

   5. ArgoCD Sync Status

   • Sync operation: ✅ Succeeded
   • Resources deployed: ✅ 20/20 resources synced
   • Health status: ❌ Degraded (due to failing pods)
   • The manifest was successfully applied, but the image name is invalid

   ---

   📋 Additional Observations

   Secondary Issues (Non-Critical):

   1. Load Balancer Sync Warnings - Intermittent warnings for store-admin and store-front services attempting to sync load balancers. These appear to be transient Azure API issues and will self-resolve.

   Healthy Resources:

   • ✅ mongodb StatefulSet (1/1 Running)
   • ✅ rabbitmq StatefulSet (1/1 Running)
   • ✅ makeline-service (1/1 Running)
   • ✅ product-service (1/1 Running)
   • ✅ store-admin (1/1 Running)
   • ✅ store-front (1/1 Running)
   • ✅ virtual-customer (1/1 Running)
   • ✅ virtual-worker (1/1 Running)

   ---

   🔧 Remediation

   Pull Request Created: #50
   Branch: fix/cluster-doctor-issue-49-20260211

   Changes Applied:

   • Corrected image name from order-servie to order-service
   • Single-line change to manifest file
   • Safe, non-destructive fix

   Auto-Remediation: Once PR #50 is merged, ArgoCD will automatically:

   1. Detect the manifest change (via Git polling or webhook)
   2. Sync the updated deployment
   3. Pull the correct image
   4. Start order-service pods successfully
   5. Transition application health to Healthy

   ---

   ✅ Validation Plan

   Post-Merge Verification Commands:

   # 1. Monitor ArgoCD application sync
   kubectl get application tech-connect-live -n argocd -w
   
   # 2. Check order-service pods (should show Running within 1-2 min)
   kubectl get pods -n default -l app=order-service
   
   # 3. Verify pod readiness
   kubectl describe pod -n default -l app=order-service
   
   # 4. Check application health
   kubectl get pods -n default
   
   # 5. Verify events (should show successful image pull)
   kubectl get events -n default --sort-by='.lastTimestamp' | grep order-service

   Expected Results:

   • order-service pods: 1/1 Running
   • ArgoCD health: Healthy
   • No ImagePullBackOff errors
   • Readiness/liveness probes passing

   ---

   🛡️ Safety & Rollback

   Risk Assessment: ⬇️ LOW RISK

   • One-character typo fix
   • No architectural changes
   • No configuration changes
   • Existing resources remain untouched

   Rollback Options:

   1. Git Revert: Revert PR Fix: Correct order-service image name typo (Issue #49) #50 (ArgoCD will auto-sync back)
   2. Manual: kubectl rollout undo deployment/order-service -n default
   3. ArgoCD: Roll back to previous revision in ArgoCD UI

   ---

   📊 Diagnostics Summary

   Category	Status	Details
   Cluster Identity	✅ Verified	aks-eastus2 confirmed via multiple signals
   Network/CNI	✅ Healthy	No network policy or CNI issues detected
   RBAC/Permissions	✅ Healthy	ArgoCD successfully synced all resources
   Resource Constraints	✅ Healthy	No CPU/memory pressure issues
   Image Registry	❌ Failed	Image name typo prevents pull
   Config/Secrets	✅ Healthy	All ConfigMaps and Secrets synced
   Dependencies	✅ Healthy	RabbitMQ and MongoDB running normally

   ---

   🎓 Recommendations

   1. Immediate: Merge PR Fix: Correct order-service image name typo (Issue #49) #50 to resolve the issue
   2. Short-term: Consider adding pre-commit hooks to validate image names against registry
   3. Long-term: Implement automated manifest validation (kubeval, kustomize, helm lint) in CI/CD pipeline

   ---

   Diagnostic Agent: cluster-doctor
   Timestamp: 2026-02-11T17:22:00Z
   Confidence: 🟢 HIGH (Root cause confirmed via multiple evidence sources)
5. github-actions commented on Feb 11, 2026

   

   github-actionsbot

   on Feb 11, 2026 – with GitHub Actions

   Author

   More actions

   🔄 Deployment Failed Again

   Timestamp: 2026-02-11T22:43:21.734Z
   Revision: def78ff78f604229ea472eb2ecb1a3f001fcec60

   Error:

   successfully synced (all tasks run)