Skip to content

🚨 ArgoCD Deployment Failed: aks-labs-on-tech-connect #52

Open
#53
Open
🚨 ArgoCD Deployment Failed: aks-labs-on-tech-connect#52
#53
Labels
argocd-deployment-failureautomatedbugSomething isn't workingcluster-doctorTag and trigger the Cluster Doctor Custom Agent

Description

@github-actions
github-actions
bot
opened on Feb 11, 2026

ArgoCD Deployment Failure

Application: aks-labs-on-tech-connect
Timestamp: 2026-02-11T22:52:24.309Z

Cluster Information

Field Value
Cluster Name aks-eastus2
Resource Group rg-anyscale-demo
Namespace default

Application Status

Field Value
Health Status Degraded
Sync Status OutOfSync
Revision def78ff78f604229ea472eb2ecb1a3f001fcec60
Repository https://github.com/DevExpGbb/agentic-platform-engineering.git

Error Message

<no value>

Troubleshooting Commands

# Check application status in ArgoCD
argocd app get aks-labs-on-tech-connect

# Check pods in namespace
kubectl get pods -n default

# Describe failed pods
kubectl describe pods -n default

# Get pod logs
kubectl logs -n default <pod-name>

# Check events
kubectl get events -n default --sort-by='.lastTimestamp'

Quick Links

This issue was automatically created by ArgoCD Notifications

Activity

  1. github-actions
    added
    bugSomething isn't working
    argocd-deployment-failure
    automated
    on Feb 11, 2026

  2. github-actions commented on Feb 12, 2026

    @github-actions
    github-actionsbot
    on Feb 12, 2026 – with GitHub Actions
    Author

    🔄 Deployment Failed Again

    Timestamp: 2026-02-12T00:45:05.476Z
    Revision: def78ff78f604229ea472eb2ecb1a3f001fcec60

    Error:

    <no value>
  3. dcasati
    added
    cluster-doctorTag and trigger the Cluster Doctor Custom Agent
    on Feb 12, 2026
  4. github-actions
    added a commit that references this issue on Feb 12, 2026

    fix: correct typo in order-service image name

    2eeb1aa

  5. github-actions commented on Feb 12, 2026

    @github-actions
    github-actionsbot
    on Feb 12, 2026 – with GitHub Actions
    Author

    🔍 Cluster Doctor Diagnostic Report

    Issue Status: ✅ ROOT CAUSE IDENTIFIED & FIX CREATED
    Severity: High (Service unavailable for 8+ hours)
    Cluster: aks-eastus2 (rg-anyscale-demo)
    Namespace: default
    Timestamp: 2026-02-12T00:51:27Z

    📊 Executive Summary

    The order-service deployment failure is caused by a typo in the container image name. The manifest specifies order-servie instead of order-service, preventing the pod from pulling the container image from the registry.

    🔬 Diagnostic Evidence

    Cluster State Analysis

    Pods in namespace default:

    NAME                                READY   STATUS             RESTARTS      AGE
order-service-b89d58f55-bnrj5       0/1     ImagePullBackOff   0             8h
order-service-bf95bbdc4-hv654       0/1     ImagePullBackOff   0             128m

    Deployment Status:

    NAME            READY   UP-TO-DATE   AVAILABLE
order-service   0/1     1            0

    Critical Events (Last 3 hours):

    3m38s   Warning   Failed   pod/order-service-b89d58f55-bnrj5   Error: ImagePullBackOff
3m38s   Normal    BackOff  pod/order-service-b89d58f55-bnrj5   Back-off pulling image "ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0"

    Backoff Statistics:

    • Total failed pull attempts: 2,269+ over 8 hours
    • Error message: Back-off pulling image "ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0"

    Root Cause Verification

    Image name in manifest:

    # File: Act-3/argocd/apps/broken-aks-store-all-in-one.yaml:196
image: ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0
                                                      ^^^^^^^^
                                                      TYPO HERE

    Correct image name should be:

    image: ghcr.io/azure-samples/aks-store-demo/order-service:2.1.0
                                                      ^^^^^^^^
                                                      FIXED

    🎯 Root Cause

    Type: Configuration error (typo)
    Location: Act-3/argocd/apps/broken-aks-store-all-in-one.yaml line 196
    Issue: Image name contains order-servie instead of order-service
    Impact: Container runtime unable to resolve image from registry

    ✅ Remediation Applied

    Pull Request Created: #53
    Branch: fix/cluster-doctor/issue52-order-service-typo-20260212

    Change Summary:

    • ✏️ Fixed typo in image name
    • 📝 Single line change in manifest
    • 🔄 ArgoCD will auto-sync after merge

    Files Modified:

    • Act-3/argocd/apps/broken-aks-store-all-in-one.yaml (1 line)

    🧪 Verification Steps

    After PR #53 is merged:

    1. Monitor ArgoCD sync:

      argocd app sync aks-labs-on-tech-connect
argocd app get aks-labs-on-tech-connect

    2. Verify pod recovery:

      kubectl get pods -n default -l app=order-service -w

      Expected: Transition from ImagePullBackOff → Running in ~1-2 minutes

    3. Check deployment health:

      kubectl get deployment order-service -n default

      Expected: READY: 1/1

    4. Validate service endpoints:

      kubectl get svc order-service -n default
kubectl port-forward svc/order-service 3000:3000
curl http://localhost:3000/health

    📋 Additional Findings

    ✅ All other services operational:

    • ✅ makeline-service: 1/1 Running
    • ✅ mongodb: 1/1 Running
    • ✅ product-service: 1/1 Running
    • ✅ rabbitmq: 1/1 Running
    • ✅ store-admin: 1/1 Running
    • ✅ store-front: 1/1 Running
    • ✅ virtual-customer: 1/1 Running
    • ✅ virtual-worker: 1/1 Running

    ⚠️ Minor load balancer warnings detected:

    • Some transient SyncLoadBalancerFailed events for store-front and store-admin
    • These appear to be temporary cloud provider sync delays
    • Services are functioning despite warnings
    • Recommend monitoring but no immediate action required

    🔐 Cluster identity verified:

    • Cluster: aks-eastus2
    • Resource Group: rg-anyscale-demo
    • Namespace: default
    • ✅ Matches issue metadata

    🚀 Next Steps

    1. ✅ Review and merge PR fix: correct typo in order-service image name #53 to fix the image name typo
    2. ⏳ Wait for ArgoCD auto-sync (or trigger manually)
    3. 📊 Monitor pod transition to Running state
    4. ✅ Verify application health returns to Healthy
    5. 🔒 Close this issue once verified

    🤖 Agent Metadata

    • Agent: Cluster Doctor (Autonomous K8s Diagnostics)
    • Mode: Read-only diagnostics + GitOps PR creation
    • Cluster access: Verified via kubectl
    • Safety checks: Passed (no destructive actions taken)
    • Audit trail: All commands logged

    Diagnostic confidence: 🟢 100% - Root cause confirmed

    This diagnostic was performed autonomously by the Cluster Doctor agent with read-only cluster access. All remediation changes are proposed via GitOps PR for review and approval.

  6. github-actions commented on Feb 15, 2026

    @github-actions
    github-actionsbot
    on Feb 15, 2026 – with GitHub Actions
    Author

    🔄 Deployment Failed Again

    Timestamp: 2026-02-15T06:22:25.200Z
    Revision: def78ff78f604229ea472eb2ecb1a3f001fcec60

    Error:

    <no value>

  7. github-actions commented on Feb 15, 2026

    @github-actions
    github-actionsbot
    on Feb 15, 2026 – with GitHub Actions
    Author

    🔄 Deployment Failed Again

    Timestamp: 2026-02-15T06:40:38.507Z
    Revision: def78ff78f604229ea472eb2ecb1a3f001fcec60

    Error:

    <no value>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    argocd-deployment-failureautomatedbugSomething isn't workingcluster-doctorTag and trigger the Cluster Doctor Custom Agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions