[MM-less extension] Cannot pass JVM flags to peons on MM-less/k8s mode

### Affected Version

34.0.0

### Description

The JVM flags defined on the pod template for the peons when running tasks natively on k8s (`druid-kubernetes-overlord-extensions`) are not passed to the task peon. As a result, tasks fail upon startup because they complain about not having enough DirectMemory space.

####  Peon logs:

```
2026-04-28T11:48:28+00:00 startup service overlord
2026-04-28T11:48:29,608 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 6.2.5.Final
2026-04-28T11:48:29,972 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-basic-security], jars: druid-basic-security-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:29,974 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:29,976 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-distinctcount], jars: druid-distinctcount-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:29,979 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: druid-kafka-indexing-service-34.0.0.jar, kafka-clients-3.9.1.jar, lz4-java-1.8.0.jar, snappy-java-1.1.10.7.jar, zstd-jni-1.5.2-3.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,003 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kubernetes-overlord-extensions], jars: druid-kubernetes-overlord-extensions-34.0.0.jar, jackson-dataformat-yaml-2.18.4.jar, jackson-datatype-jsr310-2.18.4.jar, kubernetes-client-7.2.0.jar, kubernetes-client-api-7.2.0.jar, kubernetes-httpclient-vertx-7.2.0.jar, kubernetes-model-admissionregistration-7.2.0.jar, kubernetes-model-apiextensions-7.2.0.jar, kubernetes-model-apps-7.2.0.jar, kubernetes-model-autoscaling-7.2.0.jar, kubernetes-model-batch-7.2.0.jar, kubernetes-model-certificates-7.2.0.jar, kubernetes-model-common-7.2.0.jar, kubernetes-model-coordination-7.2.0.jar, kubernetes-model-core-7.2.0.jar, kubernetes-model-discovery-7.2.0.jar, kubernetes-model-events-7.2.0.jar, kubernetes-model-extensions-7.2.0.jar, kubernetes-model-flowcontrol-7.2.0.jar, kubernetes-model-gatewayapi-7.2.0.jar, kubernetes-model-metrics-7.2.0.jar, kubernetes-model-networking-7.2.0.jar, kubernetes-model-node-7.2.0.jar, kubernetes-model-policy-7.2.0.jar, kubernetes-model-rbac-7.2.0.jar, kubernetes-model-resource-7.2.0.jar, kubernetes-model-scheduling-7.2.0.jar, kubernetes-model-storageclass-7.2.0.jar, netty-buffer-4.1.122.Final.jar, netty-codec-4.1.122.Final.jar, netty-codec-dns-4.1.122.Final.jar, netty-codec-http-4.1.122.Final.jar, netty-codec-http2-4.1.122.Final.jar, netty-codec-socks-4.1.122.Final.jar, netty-common-4.1.122.Final.jar, netty-handler-4.1.122.Final.jar, netty-handler-proxy-4.1.122.Final.jar, netty-resolver-4.1.122.Final.jar, netty-resolver-dns-4.1.122.Final.jar, netty-transport-4.1.122.Final.jar, netty-transport-native-unix-common-4.1.122.Final.jar, slf4j-api-2.0.16.jar, snakeyaml-2.3.jar, snakeyaml-engine-2.9.jar, vertx-auth-common-4.5.14.jar, vertx-core-4.5.14.jar, vertx-web-client-4.5.14.jar, vertx-web-common-4.5.14.jar, zjsonpatch-7.2.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,055 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-lookups-cached-global], jars: druid-lookups-cached-global-34.0.0.jar, mapdb-1.0.8.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,056 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-multi-stage-query], jars: druid-multi-stage-query-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,077 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-parquet-extensions], jars: aircompressor-2.0.2.jar, avro-1.11.4.jar, commons-logging-1.3.5.jar, druid-parquet-extensions-34.0.0.jar, hadoop-client-api-3.3.6.jar, hadoop-client-runtime-3.3.6.jar, javax.annotation-api-1.3.2.jar, jsr305-3.0.2.jar, parquet-avro-1.15.2.jar, parquet-column-1.15.2.jar, parquet-common-1.15.2.jar, parquet-encoding-1.15.2.jar, parquet-format-structures-1.15.2.jar, parquet-hadoop-1.15.2.jar, parquet-jackson-1.15.2.jar, slf4j-api-2.0.16.jar, snappy-java-1.1.10.7.jar, zstd-jni-1.5.2-3.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,088 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-protobuf-extensions], jars: druid-protobuf-extensions-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,092 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-s3-extensions], jars: aws-java-sdk-core-1.12.784.jar, aws-java-sdk-sts-1.12.784.jar, commons-codec-1.16.1.jar, commons-logging-1.3.5.jar, druid-s3-extensions-34.0.0.jar, httpclient-4.5.13.jar, httpcore-4.4.16.jar, jackson-dataformat-cbor-2.18.4.jar, jmespath-java-1.12.784.jar, joda-time-2.14.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,095 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [postgresql-metadata-storage], jars: checker-qual-3.42.0.jar, postgresql-42.7.2.jar, postgresql-metadata-storage-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,098 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [statsd-emitter], jars: asm-9.8.jar, asm-analysis-7.1.jar, asm-commons-9.8.jar, asm-tree-7.1.jar, asm-util-7.1.jar, java-dogstatsd-client-4.0.0.jar, jffi-1.2.23-native.jar, jffi-1.2.23.jar, jnr-a64asm-1.0.0.jar, jnr-constants-0.9.17.jar, jnr-enxio-0.30.jar, jnr-ffi-2.1.16.jar, jnr-posix-3.0.61.jar, jnr-unixsocket-0.36.jar, jnr-x86asm-1.0.2.jar, statsd-emitter-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,233 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-basic-security], jars: druid-basic-security-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,235 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,238 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-distinctcount], jars: druid-distinctcount-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,239 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: druid-kafka-indexing-service-34.0.0.jar, kafka-clients-3.9.1.jar, lz4-java-1.8.0.jar, snappy-java-1.1.10.7.jar, zstd-jni-1.5.2-3.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,240 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kubernetes-overlord-extensions], jars: druid-kubernetes-overlord-extensions-34.0.0.jar, jackson-dataformat-yaml-2.18.4.jar, jackson-datatype-jsr310-2.18.4.jar, kubernetes-client-7.2.0.jar, kubernetes-client-api-7.2.0.jar, kubernetes-httpclient-vertx-7.2.0.jar, kubernetes-model-admissionregistration-7.2.0.jar, kubernetes-model-apiextensions-7.2.0.jar, kubernetes-model-apps-7.2.0.jar, kubernetes-model-autoscaling-7.2.0.jar, kubernetes-model-batch-7.2.0.jar, kubernetes-model-certificates-7.2.0.jar, kubernetes-model-common-7.2.0.jar, kubernetes-model-coordination-7.2.0.jar, kubernetes-model-core-7.2.0.jar, kubernetes-model-discovery-7.2.0.jar, kubernetes-model-events-7.2.0.jar, kubernetes-model-extensions-7.2.0.jar, kubernetes-model-flowcontrol-7.2.0.jar, kubernetes-model-gatewayapi-7.2.0.jar, kubernetes-model-metrics-7.2.0.jar, kubernetes-model-networking-7.2.0.jar, kubernetes-model-node-7.2.0.jar, kubernetes-model-policy-7.2.0.jar, kubernetes-model-rbac-7.2.0.jar, kubernetes-model-resource-7.2.0.jar, kubernetes-model-scheduling-7.2.0.jar, kubernetes-model-storageclass-7.2.0.jar, netty-buffer-4.1.122.Final.jar, netty-codec-4.1.122.Final.jar, netty-codec-dns-4.1.122.Final.jar, netty-codec-http-4.1.122.Final.jar, netty-codec-http2-4.1.122.Final.jar, netty-codec-socks-4.1.122.Final.jar, netty-common-4.1.122.Final.jar, netty-handler-4.1.122.Final.jar, netty-handler-proxy-4.1.122.Final.jar, netty-resolver-4.1.122.Final.jar, netty-resolver-dns-4.1.122.Final.jar, netty-transport-4.1.122.Final.jar, netty-transport-native-unix-common-4.1.122.Final.jar, slf4j-api-2.0.16.jar, snakeyaml-2.3.jar, snakeyaml-engine-2.9.jar, vertx-auth-common-4.5.14.jar, vertx-core-4.5.14.jar, vertx-web-client-4.5.14.jar, vertx-web-common-4.5.14.jar, zjsonpatch-7.2.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,242 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-lookups-cached-global], jars: druid-lookups-cached-global-34.0.0.jar, mapdb-1.0.8.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,243 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-multi-stage-query], jars: druid-multi-stage-query-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,250 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-parquet-extensions], jars: aircompressor-2.0.2.jar, avro-1.11.4.jar, commons-logging-1.3.5.jar, druid-parquet-extensions-34.0.0.jar, hadoop-client-api-3.3.6.jar, hadoop-client-runtime-3.3.6.jar, javax.annotation-api-1.3.2.jar, jsr305-3.0.2.jar, parquet-avro-1.15.2.jar, parquet-column-1.15.2.jar, parquet-common-1.15.2.jar, parquet-encoding-1.15.2.jar, parquet-format-structures-1.15.2.jar, parquet-hadoop-1.15.2.jar, parquet-jackson-1.15.2.jar, slf4j-api-2.0.16.jar, snappy-java-1.1.10.7.jar, zstd-jni-1.5.2-3.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,251 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-protobuf-extensions], jars: druid-protobuf-extensions-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,252 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-s3-extensions], jars: aws-java-sdk-core-1.12.784.jar, aws-java-sdk-sts-1.12.784.jar, commons-codec-1.16.1.jar, commons-logging-1.3.5.jar, druid-s3-extensions-34.0.0.jar, httpclient-4.5.13.jar, httpcore-4.4.16.jar, jackson-dataformat-cbor-2.18.4.jar, jmespath-java-1.12.784.jar, joda-time-2.14.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,253 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [postgresql-metadata-storage], jars: checker-qual-3.42.0.jar, postgresql-42.7.2.jar, postgresql-metadata-storage-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,254 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [statsd-emitter], jars: asm-9.8.jar, asm-analysis-7.1.jar, asm-commons-9.8.jar, asm-tree-7.1.jar, asm-util-7.1.jar, java-dogstatsd-client-4.0.0.jar, jffi-1.2.23-native.jar, jffi-1.2.23.jar, jnr-a64asm-1.0.0.jar, jnr-constants-0.9.17.jar, jnr-enxio-0.30.jar, jnr-ffi-2.1.16.jar, jnr-posix-3.0.61.jar, jnr-unixsocket-0.36.jar, jnr-x86asm-1.0.2.jar, statsd-emitter-34.0.0.jar. Druid extension dependencies [[]]
2026-04-28T11:48:30,519 INFO [main] org.apache.druid.cli.CliPeon - Running peon in k8s mode
2026-04-28T11:48:30,926 INFO [main] org.apache.druid.emitter.statsd.DimensionConverter - Using metric dimensions at types at [/opt/druid/conf/druid/cluster/_common/metricsMappingStatsD.json]
2026-04-28T11:48:30,947 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping property [druid.emitter.prometheus.addServiceAsLabel]: one of it's prefixes [druid] is also used as a property key.
2026-04-28T11:48:30,947 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping property [druid.emitter.prometheus.namespace]: one of it's prefixes [druid] is also used as a property key.
2026-04-28T11:48:30,948 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping property [druid.emitter.prometheus.strategy]: one of it's prefixes [druid] is also used as a property key.
2026-04-28T11:48:30,948 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping property [druid.emitter.statsd.dimensionMapPath]: one of it's prefixes [druid] is also used as a property key.
2026-04-28T11:48:30,948 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping property [druid.emitter.prometheus.port]: one of it's prefixes [druid] is also used as a property key.
2026-04-28T11:48:30,968 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [org.apache.druid.emitter.statsd.StatsDEmitter@2fac80a8] for metrics and alerts, with dimensions [{version=34.0.0}].
2026-04-28T11:48:31,474 INFO [main] org.apache.druid.java.util.metrics.SigarUtil - Loading sigar native lib at tmpPath[/tmp/sigar10852341354949080755/libsigar-amd64-linux.so]
2026-04-28T11:48:31,502 INFO [main] org.apache.druid.server.metrics.MetricsModule - Loaded 7 monitors: org.apache.druid.java.util.metrics.JvmMonitor, org.apache.druid.java.util.metrics.JvmCpuMonitor, org.apache.druid.java.util.metrics.JvmThreadsMonitor, org.apache.druid.server.metrics.ServiceStatusMonitor, org.apache.druid.query.ExecutorServiceMonitor, org.apache.druid.curator.DruidConnectionStateListener, org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor
2026-04-28T11:48:31,507 INFO [main] org.apache.druid.cli.CliPeon - Starting up with processors [2], memory [116,391,936], maxMemory [2,147,483,648], directMemory [2,147,483,648]. Properties follow.
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authenticator.MyBasicMetadataAuthenticator.authorizerName: MyBasicMetadataAuthorizer
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authenticator.MyBasicMetadataAuthenticator.credentialIterations: 10000
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authenticator.MyBasicMetadataAuthenticator.credentialsValidator.type: metadata
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authenticator.MyBasicMetadataAuthenticator.initialAdminPassword: <masked>
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authenticator.MyBasicMetadataAuthenticator.initialInternalClientPassword: <masked>
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authenticator.MyBasicMetadataAuthenticator.skipOnFailure: false
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authenticator.MyBasicMetadataAuthenticator.type: basic
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authenticatorChain: ["MyBasicMetadataAuthenticator"]
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authorizer.MyBasicMetadataAuthorizer.initialAdminRole: admin
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authorizer.MyBasicMetadataAuthorizer.initialAdminUser: admin
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authorizer.MyBasicMetadataAuthorizer.type: basic
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.auth.authorizers: ["MyBasicMetadataAuthorizer"]
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.cache.sizeInBytes: 400Mi
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.cache.type: caffeine
2026-04-28T11:48:31,508 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter: statsd
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.prometheus.addServiceAsLabel: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.prometheus.dimensionMapPath: /opt/druid/conf/druid/cluster/_common/metricsMapping.json
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.prometheus.namespace: druid
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.prometheus.port: 9001
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.prometheus.strategy: exporter
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.statsd.dimensionMapPath: /opt/druid/conf/druid/cluster/_common/metricsMappingStatsD.json
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.statsd.dogstatsd: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.statsd.dogstatsdServiceAsTag: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.statsd.hostname: otel-collector.otel-collector.svc.cluster.local
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.statsd.includeHost: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.statsd.port: 8125
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.escalator.authorizerName: MyBasicMetadataAuthorizer
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.escalator.internalClientPassword: <masked>
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.escalator.internalClientUsername: druid_system
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.escalator.type: basic
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.extensions.loadList: ["druid-basic-security", "druid-datasketches", "druid-distinctcount", "druid-kafka-indexing-service", "druid-kubernetes-overlord-extensions", "druid-lookups-cached-global", "druid-multi-stage-query", "druid-parquet-extensions", "druid-protobuf-extensions", "druid-s3-extensions", "postgresql-metadata-storage", "statsd-emitter"]
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.disableAcl: true
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.s3Bucket: REDACTED
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.s3Prefix: tasks
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.type: s3
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.runner.javaOptsArray: ["-server", "-Xms2900M", "-Xmx2900M", "-XX:MaxDirectMemorySize=4G", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.io.tmpdir=/druid/data/temp/", "-XX:+ExitOnOutOfMemoryError", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:+UseG1GC", "--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED", "--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED", "--add-opens=java.base/java.lang=ALL-UNNAMED", "--add-opens=java.base/java.io=ALL-UNNAMED", "--add-opens=java.base/java.nio=ALL-UNNAMED", "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED", "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"]
2026-04-28T11:48:31,509 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.runner.type: k8s
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.task.baseTaskDir: /druid/data/temp
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.task.encapsulatedTask: true
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.lookup.enableLookupSyncOnStartup: false
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.connectURI: jdbc:postgresql://druid-postgresql-cluster.druid.svc:5432/druid
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.createTables: true
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.password: <masked>
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.user: druid
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.type: postgresql
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.monitoring.emissionPeriod: PT10S
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.monitoring.monitors: ["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.java.util.metrics.JvmCpuMonitor", "org.apache.druid.java.util.metrics.JvmThreadsMonitor", "org.apache.druid.server.metrics.ServiceStatusMonitor"]
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.peon.mode: remote
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.plaintextPort: 8100
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.port: 8100
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.processing.buffer.sizeBytes: 400MiB
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.processing.intermediaryData.storage.type: deepstore
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.processing.numMergeBuffers: 4
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.processing.numThreads: 4
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.processing.tmpDir: /druid/data/tmp/queries
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.query.groupBy.maxOnDiskStorage: 2000000000
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.realtime.cache.populateCache: true
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.realtime.cache.useCache: true
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.request.logging.delegate.type: slf4j
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.request.logging.queryTimeThresholdMs: 60000
2026-04-28T11:48:31,510 INFO [main] org.apache.druid.cli.CliPeon - * druid.request.logging.sqlQueryTimeThresholdMs: 60000
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.request.logging.type: filtered
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.s3.accessKey: { "type": "environment", "variable": "AWS_ACCESS_KEY_ID" }
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.s3.secretKey: <masked>
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.server.http.defaultQueryTimeout: 120000
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.server.http.numThreads: 130
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.service: druid/peon
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.startup.logging.logProperties: true
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.storage.baseKey: segments
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.storage.bucket: s73-dremel-druid-deep-storage-staging
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.storage.disableAcl: true
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.storage.type: s3
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.zk.paths.base: /druid
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.zk.service.compress: false
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * druid.zk.service.host: druid-zookeeper.druid.svc
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * file.encoding: UTF-8
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * file.separator: /
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * java.class.path: /tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/master/coordinator-overlord:lib/RoaringBitmap-0.9.49.jar:lib/accessors-smart-2.5.2.jar:lib/airline-2.8.4.jar:lib/airline-io-2.8.4.jar:lib/annotations-26.0.2.jar:lib/antlr4-runtime-4.5.3.jar:lib/aopalliance-1.0.jar:lib/apiguardian-api-1.1.2.jar:lib/asm-9.8.jar:lib/asm-commons-9.8.jar:lib/asm-tree-9.8.jar:lib/async-http-client-3.0.2.jar:lib/audience-annotations-0.12.0.jar:lib/avatica-core-1.25.0.jar:lib/avatica-metrics-1.25.0.jar:lib/avatica-server-1.25.0.jar:lib/aws-java-sdk-core-1.12.784.jar:lib/aws-java-sdk-ec2-1.12.784.jar:lib/aws-java-sdk-kms-1.12.784.jar:lib/aws-java-sdk-s3-1.12.784.jar:lib/aws-java-sdk-sts-1.12.784.jar:lib/caffeine-2.8.0.jar:lib/calcite-core-1.37.0.jar:lib/calcite-linq4j-1.37.0.jar:lib/checker-qual-3.48.1.jar:lib/classmate-1.5.1.jar:lib/commons-beanutils-1.11.0.jar:lib/commons-codec-1.16.1.jar:lib/commons-collections-3.2.2.jar:lib/commons-collections4-4.2.jar:lib/commons-compiler-3.1.11.jar:lib/commons-compress-1.26.0.jar:lib/commons-dbcp2-2.0.1.jar:lib/commons-io-2.17.0.jar:lib/commons-lang3-3.17.0.jar:lib/commons-logging-1.3.5.jar:lib/commons-math3-3.6.1.jar:lib/commons-pool2-2.2.jar:lib/commons-text-1.13.1.jar:lib/compress-lzf-1.0.4.jar:lib/cron-scheduler-0.1.jar:lib/cron-utils-9.2.0.jar:lib/curator-client-5.8.0.jar:lib/curator-framework-5.8.0.jar:lib/curator-recipes-5.8.0.jar:lib/curator-x-discovery-5.8.0.jar:lib/datasketches-java-4.2.0.jar:lib/datasketches-memory-2.2.0.jar:lib/derby-10.14.2.0.jar:lib/derbyclient-10.14.2.0.jar:lib/derbynet-10.14.2.0.jar:lib/disruptor-3.3.6.jar:lib/druid-aws-common-34.0.0.jar:lib/druid-gcp-common-34.0.0.jar:lib/druid-indexing-hadoop-34.0.0.jar:lib/druid-indexing-service-34.0.0.jar:lib/druid-processing-34.0.0.jar:lib/druid-server-34.0.0.jar:lib/druid-services-34.0.0.jar:lib/druid-sql-34.0.0.jar:lib/elasticache-java-cluster-client-1.2.0.jar:lib/error_prone_annotations-2.35.1.jar:lib/failureaccess-1.0.1.jar:lib/fastutil-8.5.4.jar:lib/fastutil-core-8.5.4.jar:lib/fastutil-extra-8.5.4.jar:lib/google-api-client-2.2.0.jar:lib/google-http-client-1.42.3.jar:lib/google-http-client-apache-v2-1.42.3.jar:lib/google-http-client-gson-1.42.3.jar:lib/google-http-client-jackson2-1.42.3.jar:lib/google-oauth-client-1.34.1.jar:lib/graal-sdk-22.3.5.jar:lib/grpc-context-1.27.2.jar:lib/gson-2.10.1.jar:lib/guava-32.1.3-jre.jar:lib/guice-5.1.0.jar:lib/guice-assistedinject-5.1.0.jar:lib/guice-servlet-5.1.0.jar:lib/hibernate-validator-6.2.5.Final.jar:lib/httpclient-4.5.13.jar:lib/httpclient5-5.2.1.jar:lib/httpcore-4.4.16.jar:lib/httpcore5-5.2.3.jar:lib/httpcore5-h2-5.2.jar:lib/icu4j-73.2.jar:lib/ipaddress-5.3.4.jar:lib/istack-commons-runtime-3.0.11.jar:lib/j2objc-annotations-2.8.jar:lib/jackson-annotations-2.18.4.jar:lib/jackson-core-2.18.4.1.jar:lib/jackson-databind-2.18.4.jar:lib/jackson-dataformat-cbor-2.18.4.jar:lib/jackson-dataformat-smile-2.18.4.jar:lib/jackson-datatype-guava-2.18.4.jar:lib/jackson-datatype-joda-2.18.4.jar:lib/jackson-jaxrs-base-2.18.4.jar:lib/jackson-jaxrs-json-provider-2.18.4.jar:lib/jackson-jaxrs-smile-provider-2.18.4.jar:lib/jackson-jq-0.0.10.jar:lib/jackson-module-guice-2.18.4.jar:lib/jackson-module-jaxb-annotations-2.18.4.jar:lib/jakarta.activation-1.2.2.jar:lib/jakarta.activation-api-1.2.2.jar:lib/jakarta.el-3.0.4.jar:lib/jakarta.inject-api-1.0.3.jar:lib/jakarta.validation-api-2.0.2.jar:lib/jakarta.xml.bind-api-2.3.3.jar:lib/janino-3.1.11.jar:lib/javax.el-api-3.0.0.jar:lib/javax.inject-1.jar:lib/javax.servlet-api-3.1.0.jar:lib/jaxb-runtime-2.3.3.jar:lib/jboss-logging-3.4.1.Final.jar:lib/jcl-over-slf4j-2.0.16.jar:lib/jcodings-1.0.50.jar:lib/jdbi-2.63.1.jar:lib/jersey-core-1.19.4.jar:lib/jersey-guice-1.19.4.jar:lib/jersey-server-1.19.4.jar:lib/jersey-servlet-1.19.4.jar:lib/jetty-client-9.4.57.v20241219.jar:lib/jetty-continuation-9.4.57.v20241219.jar:lib/jetty-http-9.4.57.v20241219.jar:lib/jetty-io-9.4.57.v20241219.jar:lib/jetty-proxy-9.4.57.v20241219.jar:lib/jetty-rewrite-9.4.57.v20241219.jar:lib/jetty-security-9.4.57.v20241219.jar:lib/jetty-server-9.4.57.v20241219.jar:lib/jetty-servlet-9.4.57.v20241219.jar:lib/jetty-servlets-9.4.57.v20241219.jar:lib/jetty-util-9.4.57.v20241219.jar:lib/jetty-util-ajax-9.4.57.v20241219.jar:lib/jmespath-java-1.12.784.jar:lib/jna-5.13.0.jar:lib/jna-platform-5.13.0.jar:lib/joda-time-2.14.0.jar:lib/joni-2.1.34.jar:lib/js-22.3.5.jar:lib/js-scriptengine-22.3.5.jar:lib/json-path-2.9.0.jar:lib/json-simple-1.1.1.jar:lib/json-smart-2.5.2.jar:lib/jsr305-3.0.2.jar:lib/jsr311-api-1.1.1.jar:lib/jts-core-1.19.0.jar:lib/jts-io-common-1.19.0.jar:lib/log4j-1.2-api-2.22.1.jar:lib/log4j-api-2.22.1.jar:lib/log4j-core-2.22.1.jar:lib/log4j-jul-2.22.1.jar:lib/log4j-slf4j2-impl-2.22.1.jar:lib/lz4-java-1.8.0.jar:lib/maven-artifact-3.6.0.jar:lib/maven-builder-support-3.6.0.jar:lib/maven-model-3.6.0.jar:lib/maven-model-builder-3.6.0.jar:lib/maven-repository-metadata-3.6.0.jar:lib/maven-resolver-api-1.3.1.jar:lib/maven-resolver-connector-basic-1.3.1.jar:lib/maven-resolver-impl-1.3.1.jar:lib/maven-resolver-provider-3.6.0.jar:lib/maven-resolver-spi-1.3.1.jar:lib/maven-resolver-transport-http-1.3.1.jar:lib/maven-resolver-util-1.3.1.jar:lib/metrics-core-4.2.22.jar:lib/netty-3.10.6.Final.jar:lib/netty-buffer-4.1.122.Final.jar:lib/netty-codec-4.1.122.Final.jar:lib/netty-codec-dns-4.1.122.Final.jar:lib/netty-codec-http-4.1.122.Final.jar:lib/netty-codec-socks-4.1.122.Final.jar:lib/netty-common-4.1.122.Final.jar:lib/netty-handler-4.1.122.Final.jar:lib/netty-handler-proxy-4.1.122.Final.jar:lib/netty-resolver-4.1.122.Final.jar:lib/netty-resolver-dns-4.1.122.Final.jar:lib/netty-transport-4.1.122.Final.jar:lib/netty-transport-native-unix-common-4.1.122.Final.jar:lib/opencensus-api-0.31.1.jar:lib/opencensus-contrib-http-util-0.31.1.jar:lib/opencsv-4.6.jar:lib/oshi-core-6.4.4.jar:lib/plexus-component-annotations-1.7.1.jar:lib/plexus-interpolation-1.25.jar:lib/plexus-utils-3.1.0.jar:lib/proj4j-1.2.2.jar:lib/protobuf-java-3.25.5.jar:lib/regex-22.3.5.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/resilience4j-core-1.3.1.jar:lib/rhino-1.7.14.jar:lib/shims-0.9.49.jar:lib/sigar-1.6.5.132.jar:lib/slf4j-api-2.0.16.jar:lib/snappy-java-1.1.10.7.jar:lib/truffle-api-22.3.5.jar:lib/txw2-2.3.3.jar:lib/uzaygezen-core-0.2.jar:lib/vavr-0.10.2.jar:lib/vavr-match-0.10.2.jar:lib/web-console-34.0.0.jar:lib/xz-1.9.jar:lib/zookeeper-3.8.4.jar:lib/zookeeper-jute-3.8.4.jar:lib/zstd-jni-1.5.2-3.jar:
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * java.class.version: 61.0
2026-04-28T11:48:31,511 INFO [main] org.apache.druid.cli.CliPeon - * java.home: /usr/lib/jvm/java-17-openjdk-amd64
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.io.tmpdir: /tmp
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.runtime.name: OpenJDK Runtime Environment
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.runtime.version: 17.0.15+6-Debian-1deb12u1
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.specification.maintenance.version: 1
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.specification.name: Java Platform API Specification
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.specification.vendor: Oracle Corporation
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.specification.version: 17
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vendor: Debian
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vendor.url: https://tracker.debian.org/openjdk-17
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vendor.url.bug: https://bugs.debian.org/openjdk-17
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.version: 17.0.15
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.version.date: 2025-04-15
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.compressedOopsMode: 32-bit
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.info: mixed mode, sharing
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.name: OpenJDK 64-Bit Server VM
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.name: Java Virtual Machine Specification
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.vendor: Oracle Corporation
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.version: 17
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.vendor: Debian
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.version: 17.0.15+6-Debian-1deb12u1
2026-04-28T11:48:31,512 INFO [main] org.apache.druid.cli.CliPeon - * jdk.debug: release
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * line.separator: 

2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * log4j.shutdownCallbackRegistry: org.apache.druid.common.config.Log4jShutdown
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * log4j.shutdownHookEnabled: true
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * log4j2.is.webapp: false
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * native.encoding: UTF-8
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * os.arch: amd64
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * os.name: Linux
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * os.version: 6.12.79-101.147.amzn2023.x86_64
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * path.separator: :
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * sun.arch.data.model: 64
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * sun.boot.library.path: /usr/lib/jvm/java-17-openjdk-amd64/lib
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * sun.cpu.endian: little
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * sun.io.unicode.encoding: UnicodeLittle
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * sun.java.command: org.apache.druid.cli.Main internal peon --taskId index_kafka_cs-decisions-realtime_2ccb995f736b135_bohodcjo /druid/data/temp 1
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * sun.java.launcher: SUN_STANDARD
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * sun.jnu.encoding: UTF-8
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * sun.management.compiler: HotSpot 64-Bit Tiered Compilers
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * user.dir: /opt/druid
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * user.home: /opt/druid
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * user.language: en
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * user.name: druid
2026-04-28T11:48:31,513 INFO [main] org.apache.druid.cli.CliPeon - * user.timezone: GMT
2026-04-28T11:48:31,658 INFO [main] org.apache.druid.query.DruidProcessingConfig - Detected max direct memory size of [2,147,483,648] bytes
2026-04-28T11:48:31,669 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy[LeastBytesUsedStorageLocationSelectorStrategy].
2026-04-28T11:48:31,669 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Number of threads to load segments into page cache - on bootstrap: [0], on download: [0].
2026-04-28T11:48:31,700 INFO [main] org.apache.druid.guice.StorageNodeModule - Segment cache not configured on ServerType [indexer-executor]. It will not be assignable for segment placement
2026-04-28T11:48:31,724 INFO [main] org.eclipse.jetty.util.log - Logging initialized @2792ms to org.eclipse.jetty.util.log.Slf4jLog
2026-04-28T11:48:31,738 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http connector with port [8100]
2026-04-28T11:48:31,814 WARN [main] org.eclipse.jetty.server.handler.gzip.GzipHandler - minGzipSize of 0 is inefficient for short content, break even is size 23
2026-04-28T11:48:31,833 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up.  Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, druid.processing.numThreads, or druid.processing.numMergeBuffers: maxDirectMemory[2,147,483,648], memoryNeeded[3,774,873,600] = druid.processing.buffer.sizeBytes[419,430,400] * (druid.processing.numMergeBuffers[4] + druid.processing.numThreads[4] + 1)
  at PeonProcessingModule.getMergeBufferPool(PeonProcessingModule.java:123)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> PeonProcessingModule
  at PeonProcessingModule.getGroupByResourcesReservationPool(PeonProcessingModule.java:146)
      \_ for 1st parameter
  at PeonProcessingModule.getGroupByResourcesReservationPool(PeonProcessingModule.java:146)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> PeonProcessingModule
  at GroupingEngine.<init>(GroupingEngine.java:135)
      \_ for 3rd parameter
  at GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:59)
      \_ for 1st parameter
  at QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:90)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> QueryRunnerFactoryModule
  while locating GroupByQueryRunnerFactory
  while locating QueryRunnerFactory annotated with @Element(setName=,uniqueId=107, type=MAPBINDER, keyType=Class<? extends Query>)
  at DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:50)
      \_ for 1st parameter
  while locating DefaultQueryRunnerFactoryConglomerate
  at QueryableModule.configure(QueryableModule.java:56)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> QueryableModule
  at QueryLifecycleFactory.<init>(QueryLifecycleFactory.java:60)
      \_ for 1st parameter
  at QueryLifecycleFactory.class(QueryLifecycleFactory.java:60)
  at QueryResource.<init>(QueryResource.java:111)
      \_ for 1st parameter
  at QueryResource.class(QueryResource.java:72)
  while locating QueryResource

1 error

======================
Full classname legend:
======================
DefaultQueryRunnerFactoryConglomerate: "org.apache.druid.query.DefaultQueryRunnerFactoryConglomerate"
Element:                               "com.google.inject.internal.Element"
GroupByQueryRunnerFactory:             "org.apache.druid.query.groupby.GroupByQueryRunnerFactory"
GroupingEngine:                        "org.apache.druid.query.groupby.GroupingEngine"
Modules$OverrideModule:                "com.google.inject.util.Modules$OverrideModule"
PeonProcessingModule:                  "org.apache.druid.guice.PeonProcessingModule"
Query:                                 "org.apache.druid.query.Query"
QueryLifecycleFactory:                 "org.apache.druid.server.QueryLifecycleFactory"
QueryResource:                         "org.apache.druid.server.QueryResource"
QueryRunnerFactory:                    "org.apache.druid.query.QueryRunnerFactory"
QueryRunnerFactoryModule:              "org.apache.druid.guice.QueryRunnerFactoryModule"
QueryableModule:                       "org.apache.druid.guice.QueryableModule"
========================
End of classname legend:
========================

        at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251) ~[guice-5.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1104) ~[guice-5.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1134) ~[guice-5.1.0.jar:?]
        at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:150) ~[druid-processing-34.0.0.jar:34.0.0]
        at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:137) [druid-services-34.0.0.jar:34.0.0]
        at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) [druid-services-34.0.0.jar:34.0.0]
        at org.apache.druid.cli.CliPeon.run(CliPeon.java:390) [druid-services-34.0.0.jar:34.0.0]
        at org.apache.druid.cli.Main.main(Main.java:112) [druid-services-34.0.0.jar:34.0.0]

```
I've configured the Overlord to pick a custom Pod template with the `customTemplateAdapter`.

#### Overlord configuration
```
druid.service=druid/overlord
druid.plaintextPort=8090

# Extensions
druid.extensions.loadList=["druid-basic-security", "druid-datasketches", "druid-distinctcount", "druid-kafka-indexing-service", "druid-kubernetes-overlord-extensions", "druid-lookups-cached-global", "druid-multi-stage-query", "druid-parquet-extensions", "druid-protobuf-extensions", "druid-s3-extensions", "postgresql-metadata-storage", "prometheus-emitter"]

# https://druid.apache.org/docs/latest/configuration/index.html#overlord-operations
druid.indexer.storage.type=metadata
druid.indexer.storage.recentlyFinishedThreshold=PT24H
druid.indexer.tasklock.batchSegmentAllocation=true
druid.indexer.tasklock.batchAllocationWaitTime=0
druid.indexer.queue.startDelay=PT30S

# Monitoring
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.java.util.metrics.JvmCpuMonitor", "org.apache.druid.java.util.metrics.JvmThreadsMonitor", "org.apache.druid.server.metrics.ServiceStatusMonitor", "org.apache.druid.server.metrics.TaskCountStatsMonitor", "org.apache.druid.server.metrics.TaskSlotCountStatsMonitor"]

## Tasks Metadata/Logs Management
## https://druid.apache.org/docs/latest/operations/clean-metadata-store/#indexer-task-logs
# Cleanup of task logs and its associated metadata
druid.indexer.logs.kill.enabled=true
# 24 hours in milliseconds
druid.indexer.logs.kill.durationToRetain=86400000
# 5 min in milliseconds
druid.indexer.logs.kill.initialDelay=300000
# 6 hours in milliseconds
druid.indexer.logs.kill.delay=21600000

## Segments metadata cache
## https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental
druid.manager.segments.useCache=ifSynced

## MiddleManager-less
# https://druid.apache.org/docs/latest/development/extensions-core/k8s-jobs/
druid.indexer.runner.type=k8s
druid.indexer.task.encapsulatedTask=true
druid.indexer.runner.namespace=druid
druid.indexer.runner.capacity=600
druid.indexer.runner.k8sjobLaunchTimeout=PT10M
druid.indexer.runner.taskCleanupDelay=P2D
druid.indexer.runner.taskCleanupInterval=PT20M
druid.processing.intermediaryData.storage.type=deepstore
druid.indexer.runner.debugJobs=true

# Pod Templates per task type
druid.indexer.runner.k8s.adapter.type=customTemplateAdapter
druid.indexer.runner.k8s.podTemplate.base=/druid/task_templates/default/pod_template.yaml
druid.indexer.runner.k8s.podTemplate.index_kafka=/druid/task_templates/realtime/pod_template.yaml

## Peons
druid.indexer.runner.peonMonitors=["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.java.util.metrics.JvmCpuMonitor", "org.apache.druid.java.util.metrics.JvmThreadsMonitor", "org.apache.druid.server.metrics.ServiceStatusMonitor", "org.apache.druid.client.cache.CacheMonitor"]
```

#### Pod Template (for `index_kafka` tasks) as ConfigMap

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: druid-analytics-task-template-realtime
  namespace: druid
data:
  pod_template.yaml: |-
    apiVersion: v1
    kind: PodTemplate
    metadata:
      name: druid-task-default
      labels:
        app: druid
        component: task
        druid_cr: analytics
        task-tmpl: realtime
    template:
      metadata:
        labels:
          app: druid
          component: task
          druid_cr: analytics
          task-tmpl: realtime
      spec:
        serviceAccountName: druid-analytics-overlord
        restartPolicy: Never
        securityContext:
          fsGroup: 1000
          runAsGroup: 1000
          runAsUser: 1000
        imagePullSecrets:
          - name: docker-registry-credentials

        containers:
          - name: main
            image: <REDACTED>/druid:34.0.0-1.0.0
            imagePullPolicy: IfNotPresent
            command:
              - sh
              - -c
            args:
              - |
                /peon.sh /druid/data/temp 1
            env:
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.name
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
              - name: AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT
                value: "true"
            envFrom:
              - secretRef:
                  name: druid-credentials
              - secretRef:
                  name: druid-kafka-credentials
              - secretRef:
                  name: druid-s3-credentials

            ports:
              - containerPort: 8100
                name: druid-port
                protocol: TCP

            resources:
              limits:
                cpu: 2000m
                memory: 8Gi
              requests:
                cpu: 2000m
                memory: 7Gi

            volumeMounts:
              - name: common-config-volume
                mountPath: /opt/druid/conf/druid/cluster/_common
                readOnly: true
              - name: task-peon-config-volume
                mountPath: /opt/druid/conf/druid/cluster/master/coordinator-overlord
                readOnly: true
              - name: temp-data-volume
                mountPath: /druid/data/temp

        volumes:
          # configmaps
          - name: common-config-volume
            configMap:
              name: analytics-druid-common-config
              defaultMode: 420
          - name: task-peon-config-volume
            configMap:
              defaultMode: 420
              name: druid-analytics-task-template-realtime-config
          # storage
          - name: temp-data-volume
            emptyDir: {}

        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: node-type
                      operator: In
                      values:
                        - druid-data-mm
        tolerations:
          - key: druid
            value: data-mm
            operator: Equal
            effect: NoSchedule

        topologySpreadConstraints:
          - topologyKey: "topology.kubernetes.io/zone"
            maxSkew: 1
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                app: druid
                component: task
                druid_cr: analytics
```

#### Config files for the task/pod as ConfigMap

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: druid-analytics-task-template-realtime-config
  namespace: druid
data:

  log4j2.xml: |-
    <?xml version="1.0" encoding="UTF-8" ?>
    <Configuration status="WARN">
      <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
          <PatternLayout pattern="%d{ISO8601} %p [%t] %c -%notEmpty{ [%markerSimpleName]} %m%n"/>
        </Console>
      </Appenders>

    <Loggers>
        
        <AsyncLogger name="org.apache.druid.curator.inventory.CuratorInventoryManager" level="debug" additivity="false">
          <AppenderRef ref="Console"/>
        </AsyncLogger>
        <AsyncLogger name="org.apache.druid.client.BatchServerInventoryView" level="debug" additivity="false">
          <AppenderRef ref="Console"/>
        </AsyncLogger>
        
        <AsyncLogger name="org.apache.druid.client.ServerInventoryView" level="debug" additivity="false">
          <AppenderRef ref="Console"/>
        </AsyncLogger>
        <AsyncLogger name ="org.apache.druid.java.util.http.client.pool.ChannelResourceFactory" level="info" additivity="false">
          <AppenderRef ref="Console"/>
        </AsyncLogger>
        <AsyncLogger name="org.apache.druid.jetty.RequestLog" additivity="false" level="info">
          <AppenderRef ref="Console"/>
        </AsyncLogger>
        <Root level="info">
          <AppenderRef ref="Console"/>
        </Root>
      </Loggers>
    </Configuration>


  jvm.config: |-
    -server
    -Duser.timezone=UTC
    -Dfile.encoding=UTF-8
    -Djava.io.tmpdir=/druid/data/temp/
    -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
    -Dorg.jboss.logging.provider=slf4j
    -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger
    -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown
    -Dlog4j.shutdownHookEnabled=true
    -XX:+ExitOnOutOfMemoryError
    -XX:+UseG1GC
    --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
    --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
    --add-opens=java.base/java.lang=ALL-UNNAMED
    --add-opens=java.base/java.io=ALL-UNNAMED
    --add-opens=java.base/java.nio=ALL-UNNAMED
    --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
    --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
    -Xms2900M
    -Xmx2900M
    -XX:MaxDirectMemorySize=4G

  runtime.properties: |
    druid.service=druid/peon
    druid.port=8100
    druid.plaintextPort=8100

    # Extensions

    druid.extensions.loadList=["druid-basic-security", "druid-datasketches", "druid-distinctcount", "druid-kafka-indexing-service", "druid-kubernetes-overlord-extensions", "druid-lookups-cached-global", "druid-multi-stage-query", "druid-parquet-extensions", "druid-protobuf-extensions", "druid-s3-extensions", "postgresql-metadata-storage", "statsd-emitter"]

    druid.peon.mode=remote
    druid.indexer.runner.type=k8s
    druid.indexer.task.encapsulatedTask=true
    druid.processing.intermediaryData.storage.type=deepstore

    # https://druid.apache.org/docs/latest/configuration/index.html#middlemanager-configuration
    druid.indexer.task.baseTaskDir=/druid/data/temp

    # HTTP server
    # https://druid.apache.org/docs/latest/configuration/index.html#indexer-concurrent-requests
    # Sum of `druid.broker.http.numConnections` across all the brokers in the cluster
    druid.server.http.numThreads=130

    # Processing threads and buffers
    druid.processing.numThreads=4
    druid.processing.numMergeBuffers=4
    druid.processing.buffer.sizeBytes=400MiB

    # Monitoring
    druid.emitter=statsd
    druid.emitter.statsd.hostname=otel-collector.otel-collector.svc.cluster.local
    druid.emitter.statsd.port=8125
    druid.emitter.statsd.dimensionMapPath=/opt/druid/conf/druid/cluster/_common/metricsMappingStatsD.json
    druid.emitter.statsd.includeHost=true
    druid.emitter.statsd.dogstatsd=true
    druid.emitter.statsd.dogstatsdServiceAsTag=true

    # Query processing
    druid.processing.tmpDir=/druid/data/tmp/queries

    #https://druid.apache.org/docs/latest/configuration/#peon-caching
    # Query cache
    druid.realtime.cache.useCache=true
    druid.realtime.cache.populateCache=true
    druid.cache.sizeInBytes=400Mi

    # Query Performance
    # Default timeout (120 seconds) can be overridden at query context
    druid.server.http.defaultQueryTimeout=120000
    # GroupBy merging buffer per-query spilling to disk (2GB)
    druid.query.groupBy.maxOnDiskStorage=2000000000
```

As you can see on Peon logs, the _runtime.properties_ file is properly picked but not the JVM flags on  _jvm.config_.

I also tried setting the `druid.indexer.runner.javaOptsArray` on the `runtime.properties` but it didn't work either.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MM-less extension] Cannot pass JVM flags to peons on MM-less/k8s mode #19385

Affected Version

Description

Peon logs:

Overlord configuration

Pod Template (for `index_kafka` tasks) as ConfigMap

Config files for the task/pod as ConfigMap

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

[MM-less extension] Cannot pass JVM flags to peons on MM-less/k8s mode #19385

Description

Affected Version

Description

Peon logs:

Overlord configuration

Pod Template (for index_kafka tasks) as ConfigMap

Config files for the task/pod as ConfigMap

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Pod Template (for `index_kafka` tasks) as ConfigMap