[AIP-5397] Recover interrupted LAS Helm actions and finalize safely

[kc-splunk]

Summary

• Observe pending Helm revisions for the five-minute action timeout plus one-minute grace; recover stale, UID-owned actions using Helm history and rollback or uninstall.
• Keep AIService Ready false until the requested chart, values, and credential revision are applied; preserve actionable failure status after recovery.
• Make finalization idempotent for absent releases and missing credential Secrets, check UID ownership before uninstall, and remove only the AIService-owned compatibility Service.
• Add focused lifecycle tests and a recovery runbook.

Local validation

• go test ./pkg/ai/features/agentruntime -cover -count=1 (63.3% package coverage)
• go test ./pkg/ai/... -count=1
• go test ./internal/controller -count=1
• go vet ./pkg/ai/features/agentruntime ./internal/controller
• git diff --check

The AI package suite required local loopback socket access for an existing HTTP test. No cluster fault-injection or live runtime qualification is claimed here.

Jira: https://splunk.atlassian.net/browse/AIP-5397

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T05:41:58.164949Z	3911472	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 391147272d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kc-splunk]

Follow-up installer validation (2026-10-14): the shared agentruntime-dev cluster test interrupted a UID-owned pending-upgrade on a disposable AIService. The operator observed it as Ready=Unknown / HelmActionPending, then after the six-minute bound rolled back to deployed revision 5 (new revision 7), emitted LASHelmActionRecovered, and reported Ready=False / DesiredRevisionNotApplied before the desired service returned Ready. Finalizer deletion removed the owned release/workloads/compatibility Service while test Secret objects and PostgreSQL/Redis markers survived; all test data and resources were then cleaned up.

The original operator digest was restored, the candidate ECR tag and temporary SSH ingress were removed, and the primary AIService retained its UID and Ready condition. Its release advanced 17→19 and API/queue rolled during candidate install/restore; restored revision 19 values match revision 17. This test used the sample LAS image and does not qualify the Agent Launchpad product path or real credential consumption. Detailed evidence is in AIP-5397 comment 22403270.

[kc-splunk]

Combined AIP-5399/AIP-5406 implementation added in commit b2726b6.

This update adds LAS current-generation API/queue readiness, Service endpoint and /ok health checks, AIPlatform parent aggregation, separate ScalingReady diagnostics, and queue CPU HPA defaults/configuration/replica ownership. The packaged chart is langgraph-cloud-0.3.5-splunk.2 (SHA256 4fa2580f6e38254f399cc23e230f314dfdf42566f71df41ff1e4b23d36426058).

Validation passed: go test ./... -count=1, go vet ./..., and git diff --check. On the shared agentruntime-dev cluster, the candidate operator reconciled the AIService and parent AIPlatform Ready; HPA CPU metrics recovered after rollout; a harmless Secret metadata revision rolled both LAS workloads, and a forced no-op kept Helm revision and workload Pod UIDs unchanged. The original operator digest was restored and temporary artifacts/access were cleaned up.

Limits: no load was applied to force scale-out, and live preservation of an HPA-selected count above the minimum during upgrade/rotation was not demonstrated. Load-driven scaling qualification remains AIP-5411. The existing AIP-5397 recovery/finalizer cluster evidence remains documented in Jira.

Conflict resolution: feature branch AIP-5394 commit 206dab6 was merged into this PR in commit b816e4f. pkg/ai/features/agentruntime/las_config_test.go now retains the AIP-5394 scale-down test alongside the AIP-5406 default/autoscaling contract tests; gofmt validation passed. PR #246 remains open and unmerged. GitHub reports mergeable=true. The lint-test check completed successfully.