Skip to content

spike: IngestorCluster queue config via ConfigMap + init container (zero-restart first-boot) #1733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
vivekr-splunk wants to merge 14 commits into
base: develop
Choose a base branch
from
Draft

spike: IngestorCluster queue config via ConfigMap + init container (zero-restart first-boot) #1733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a699b95
feat(api): remove stale CredentialSecretVersion and ServiceAccount fr…
vivekr-splunk Feb 25, 2026
d96286b
feat(names): add ingestor queue config naming constants and GetIngest…
vivekr-splunk Feb 25, 2026
64fd43a
feat(enterprise): add INI builder functions and computeIngestorConfCh…
vivekr-splunk Feb 25, 2026
b121736
feat(enterprise): add buildAndApplyIngestorQueueConfigMap
vivekr-splunk Feb 25, 2026
a045c11
feat(enterprise): add setupIngestorInitContainer and update getIngest…
vivekr-splunk Feb 25, 2026
bd3bf59
refactor(enterprise): rewrite ApplyIngestorCluster — event-driven Con…
vivekr-splunk Feb 25, 2026
a2e0481
test(enterprise): rewrite ingestor unit tests for ConfigMap delivery,…
vivekr-splunk Feb 25, 2026
c5dd710
feat(enterprise): deliver IndexerCluster queue config via CM bundle push
vivekr-splunk Feb 25, 2026
c46b413
feat(enterprise): add CM queue config init container for IndexerClust…
vivekr-splunk Feb 25, 2026
2c9d42d
refactor(enterprise): extract shared queue config helpers from ingest…
vivekr-splunk Feb 25, 2026
7df2061
fix(enterprise): set defaultMode=420 on CM queue config volume to pre…
vivekr-splunk Feb 25, 2026
996fbf0
docs: rewrite IndexIngestionSeparation.md as critical user journey wi…
vivekr-splunk Feb 26, 2026
d8d6874
ci: ensure build workflows trigger for draft-to-ready PRs
vivekr-splunk Mar 5, 2026
b9c52ae
fix(ingestor): handle deletion early and align ingest test credentials
vivekr-splunk Mar 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .github/workflows/build-test-push-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ permissions:
pull-requests: write
on:
pull_request:
types:
- opened
- synchronize
- reopened
- ready_for_review
paths-ignore:
- 'docs/**'
- '*.md'
Expand Down Expand Up @@ -210,8 +215,6 @@ jobs:
EKS_SSH_PUBLIC_KEY: ${{ secrets.EKS_SSH_PUBLIC_KEY }}
CLUSTER_WIDE: "true"
DEPLOYMENT_TYPE: ""
AWS_INDEX_INGEST_SEP_ACCESS_KEY_ID: ${{ secrets.AWS_INDEX_INGEST_SEP_ACCESS_KEY_ID }}
AWS_INDEX_INGEST_SEP_SECRET_ACCESS_KEY: ${{ secrets.AWS_INDEX_INGEST_SEP_SECRET_ACCESS_KEY }}
steps:
- name: Chekcout code
uses: actions/checkout@v2
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/distroless-build-test-push-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ permissions:
pull-requests: write
on:
pull_request:
types:
- opened
- synchronize
- reopened
- ready_for_review
paths-ignore:
- 'docs/**'
- '*.md'
Expand Down Expand Up @@ -204,8 +209,6 @@ jobs:
EKS_SSH_PUBLIC_KEY: ${{ secrets.EKS_SSH_PUBLIC_KEY }}
CLUSTER_WIDE: "true"
DEPLOYMENT_TYPE: ""
AWS_INDEX_INGEST_SEP_ACCESS_KEY_ID: ${{ secrets.AWS_INDEX_INGEST_SEP_ACCESS_KEY_ID }}
AWS_INDEX_INGEST_SEP_SECRET_ACCESS_KEY: ${{ secrets.AWS_INDEX_INGEST_SEP_SECRET_ACCESS_KEY }}
steps:
- name: Chekcout code
uses: actions/checkout@v2
Expand Down
5 changes: 0 additions & 5 deletions api/v4/ingestorcluster_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,6 @@ type IngestorClusterStatus struct {
// Auxillary message describing CR status
Message string `json:"message"`

// Credential secret version to track changes to the secret and trigger rolling restart of indexer cluster peers when the secret is updated
CredentialSecretVersion string `json:"credentialSecretVersion,omitempty"`

// Service account to track changes to the service account and trigger rolling restart of indexer cluster peers when the service account is updated
ServiceAccount string `json:"serviceAccount,omitempty"`
}

// +kubebuilder:object:root=true
Expand Down
10 changes: 0 additions & 10 deletions config/crd/bases/enterprise.splunk.com_ingestorclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4612,11 +4612,6 @@ spec:
description: App Framework version info for future use
type: integer
type: object
credentialSecretVersion:
description: Credential secret version to track changes to the secret
and trigger rolling restart of indexer cluster peers when the secret
is updated
type: string
message:
description: Auxillary message describing CR status
type: string
Expand Down Expand Up @@ -4647,11 +4642,6 @@ spec:
selector:
description: Selector for pods used by HorizontalPodAutoscaler
type: string
serviceAccount:
description: Service account to track changes to the service account
and trigger rolling restart of indexer cluster peers when the service
account is updated
type: string
telAppInstalled:
description: Telemetry App installation flag
type: boolean
Expand Down
1,254 changes: 322 additions & 932 deletions docs/IndexIngestionSeparation.md
View file
Open in desktop

Large diffs are not rendered by default.

Binary file added docs/images/index_ingestion_separation.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions pkg/splunk/common/paths.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,22 @@ const (

//OperatorMountLocalServerConf
OperatorMountLocalServerConf = "/mnt/splunk-operator/local/server.conf"

//OperatorClusterManagerAppsLocalOutputsConf
OperatorClusterManagerAppsLocalOutputsConf = "/opt/splk/etc/manager-apps/splunk-operator/local/outputs.conf"

//OperatorClusterManagerAppsLocalInputsConf
OperatorClusterManagerAppsLocalInputsConf = "/opt/splk/etc/manager-apps/splunk-operator/local/inputs.conf"

//OperatorClusterManagerAppsLocalDefaultModeConf
OperatorClusterManagerAppsLocalDefaultModeConf = "/opt/splk/etc/manager-apps/splunk-operator/local/default-mode.conf"

//OperatorMountLocalOutputsConf
OperatorMountLocalOutputsConf = "/mnt/splunk-operator/local/outputs.conf"

//OperatorMountLocalInputsConf
OperatorMountLocalInputsConf = "/mnt/splunk-operator/local/inputs.conf"

//OperatorMountLocalDefaultModeConf
OperatorMountLocalDefaultModeConf = "/mnt/splunk-operator/local/default-mode.conf"
)
81 changes: 81 additions & 0 deletions pkg/splunk/enterprise/clustermanager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import (
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
k8serrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/resource"

"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/log"
Expand Down Expand Up @@ -313,12 +314,92 @@ func getClusterManagerStatefulSet(ctx context.Context, client splcommon.Controll
if smartStoreConfigMap != nil {
setupInitContainer(&ss.Spec.Template, cr.Spec.Image, cr.Spec.ImagePullPolicy, commandForCMSmartstore, cr.Spec.CommonSplunkSpec.EtcVolumeStorageConfig.EphemeralStorage)
}

// If a queue config ConfigMap exists for this CM, add a separate init container and volume.
setupCMQueueConfigInitContainer(ctx, client, cr, ss)
// Setup App framework staging volume for apps
setupAppsStagingVolume(ctx, client, cr, &ss.Spec.Template, &cr.Spec.AppFrameworkConfig)

return ss, err
}

// setupCMQueueConfigInitContainer adds a dedicated init container and ConfigMap volume for queue config
// to the ClusterManager StatefulSet if the queue config ConfigMap exists. This is a separate init container
// from the smartstore "init" container — it runs independently and symlinks outputs.conf, inputs.conf,
// and default-mode.conf from the queue config ConfigMap mount into manager-apps/splunk-operator/local/.
func setupCMQueueConfigInitContainer(ctx context.Context, client splcommon.ControllerClient, cr *enterpriseApi.ClusterManager, ss *appsv1.StatefulSet) {
configMapName := GetCMQueueConfigMapName(cr.GetName())
// Only add the init container if the queue config ConfigMap exists.
_, err := splctrl.GetConfigMap(ctx, client, types.NamespacedName{Name: configMapName, Namespace: cr.GetNamespace()})
if err != nil {
// ConfigMap doesn't exist yet — no queue config configured for this CM.
Copy link

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ClusterManager init container setup function silently returns when the ConfigMap doesn't exist. While this is intentional (the ConfigMap is only created when IndexerCluster has queueRef set), the silent return makes debugging difficult. Consider logging at Info level when the ConfigMap is not found to make it clear that queue config init is being skipped.

This would help operators understand why init containers may or may not be present on ClusterManager pods.

Suggested change
// ConfigMap doesn't exist yet — no queue config configured for this CM.
logger := log.FromContext(ctx)
if k8serrors.IsNotFound(err) {
// ConfigMap doesn't exist yet — no queue config configured for this CM.
logger.Info("Queue config ConfigMap not found; skipping queue config init container",
"configMap", configMapName,
"namespace", cr.GetNamespace(),
"clusterManager", cr.GetName())
} else {
// Unexpected error retrieving ConfigMap — skip init container but log the error for debugging.
logger.Error(err, "Failed to get queue config ConfigMap; skipping queue config init container",
"configMap", configMapName,
"namespace", cr.GetNamespace(),
"clusterManager", cr.GetName())
}

Copilot uses AI. Check for mistakes.
return
}

// Add queue config ConfigMap volume to pod spec.
// defaultMode 420 (0644) must be set explicitly to match what Kubernetes stores —
// omitting it causes MergePodUpdates to see a diff on every reconcile (infinite update loop).
defaultMode := int32(420)
ss.Spec.Template.Spec.Volumes = append(ss.Spec.Template.Spec.Volumes, corev1.Volume{
Name: cmQueueConfigVolName,
VolumeSource: corev1.VolumeSource{
ConfigMap: &corev1.ConfigMapVolumeSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: configMapName,
},
DefaultMode: &defaultMode,
},
},
})

// Determine etc volume mount name (ephemeral vs PVC)
var etcVolMntName string
if cr.Spec.CommonSplunkSpec.EtcVolumeStorageConfig.EphemeralStorage {
etcVolMntName = fmt.Sprintf(splcommon.SplunkMountNamePrefix, splcommon.EtcVolumeStorage)
} else {
etcVolMntName = fmt.Sprintf(splcommon.PvcNamePrefix, splcommon.EtcVolumeStorage)
}

runAsUser := int64(41812)
runAsNonRoot := true
privileged := false

initContainer := corev1.Container{
Name: "init-cm-queue-config",
Image: ss.Spec.Template.Spec.Containers[0].Image,
ImagePullPolicy: ss.Spec.Template.Spec.Containers[0].ImagePullPolicy,
Command: []string{"bash", "-c", commandForCMQueueConfig},
VolumeMounts: []corev1.VolumeMount{
{Name: etcVolMntName, MountPath: "/opt/splk/etc"},
{Name: cmQueueConfigVolName, MountPath: cmQueueConfigMountPath},
},
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("0.25"),
corev1.ResourceMemory: resource.MustParse("128Mi"),
},
Limits: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("1"),
corev1.ResourceMemory: resource.MustParse("512Mi"),
},
},
SecurityContext: &corev1.SecurityContext{
RunAsUser: &runAsUser,
RunAsNonRoot: &runAsNonRoot,
AllowPrivilegeEscalation: &[]bool{false}[0],
Capabilities: &corev1.Capabilities{
Drop: []corev1.Capability{"ALL"},
Add: []corev1.Capability{"NET_BIND_SERVICE"},
},
Privileged: &privileged,
SeccompProfile: &corev1.SeccompProfile{
Type: corev1.SeccompProfileTypeRuntimeDefault,
},
},
}
ss.Spec.Template.Spec.InitContainers = append(ss.Spec.Template.Spec.InitContainers, initContainer)
}

// CheckIfsmartstoreConfigMapUpdatedToPod checks if the smartstore configMap is updated on Pod or not
func CheckIfsmartstoreConfigMapUpdatedToPod(ctx context.Context, c splcommon.ControllerClient, cr *enterpriseApi.ClusterManager, podExecClient splutil.PodExecClientImpl) error {
reqLogger := log.FromContext(ctx)
Expand Down
4 changes: 4 additions & 0 deletions pkg/splunk/enterprise/clustermanager_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ func TestApplyClusterManager(t *testing.T) {
{MetaName: "*v1.Secret-test-splunk-stack1-cluster-manager-secret-v1"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-smartstore"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-smartstore"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-queue-config"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
{MetaName: "*v4.ClusterManager-test-stack1"},
Expand All @@ -97,6 +98,7 @@ func TestApplyClusterManager(t *testing.T) {
{MetaName: "*v1.Secret-test-splunk-stack1-cluster-manager-secret-v1"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-smartstore"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-smartstore"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-queue-config"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
Expand Down Expand Up @@ -578,6 +580,7 @@ func TestApplyClusterManagerWithSmartstore(t *testing.T) {
{MetaName: "*v1.Secret-test-splunk-stack1-cluster-manager-secret-v1"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-smartstore"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-smartstore"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-queue-config"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
Expand All @@ -601,6 +604,7 @@ func TestApplyClusterManagerWithSmartstore(t *testing.T) {
{MetaName: "*v1.Secret-test-splunk-stack1-cluster-manager-secret-v1"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-smartstore"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-smartstore"},
{MetaName: "*v1.ConfigMap-test-splunk-stack1-clustermanager-queue-config"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
{MetaName: "*v1.StatefulSet-test-splunk-stack1-cluster-manager"},
Expand Down
20 changes: 15 additions & 5 deletions pkg/splunk/enterprise/configuration.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -880,17 +880,27 @@ func updateSplunkPodTemplateWithConfig(ctx context.Context, client splcommon.Con

smartstoreConfigMap := getSmartstoreConfigMap(ctx, client, cr, instanceType)
if smartstoreConfigMap != nil {
items := []corev1.KeyToPath{
{Key: "indexes.conf", Path: "indexes.conf", Mode: &configMapVolDefaultMode},
{Key: "server.conf", Path: "server.conf", Mode: &configMapVolDefaultMode},
{Key: configToken, Path: configToken, Mode: &configMapVolDefaultMode},
}
// When queue config keys are present (written by applyIdxcQueueConfigToCM),
// include them so the init container symlinks resolve correctly on the CM pod.
if _, ok := smartstoreConfigMap.Data["outputs.conf"]; ok {
items = append(items,
corev1.KeyToPath{Key: "outputs.conf", Path: "outputs.conf", Mode: &configMapVolDefaultMode},
corev1.KeyToPath{Key: "inputs.conf", Path: "inputs.conf", Mode: &configMapVolDefaultMode},
corev1.KeyToPath{Key: "default-mode.conf", Path: "default-mode.conf", Mode: &configMapVolDefaultMode},
)
}
addSplunkVolumeToTemplate(podTemplateSpec, "mnt-splunk-operator", "/mnt/splunk-operator/local/", corev1.VolumeSource{
ConfigMap: &corev1.ConfigMapVolumeSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: smartstoreConfigMap.GetName(),
},
DefaultMode: &configMapVolDefaultMode,
Items: []corev1.KeyToPath{
{Key: "indexes.conf", Path: "indexes.conf", Mode: &configMapVolDefaultMode},
{Key: "server.conf", Path: "server.conf", Mode: &configMapVolDefaultMode},
{Key: configToken, Path: configToken, Mode: &configMapVolDefaultMode},
},
Items: items,
},
})

Expand Down
Loading
Loading