diff --git a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md index 35731d80363..14a72edd84a 100644 --- a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md +++ b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md @@ -743,7 +743,7 @@ To upgrade the Worker, update the Worker image version in your CloudFormation st ``` --set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282 ``` -- If you enable [disk buffering][16] for destinations, you must enable Kubernetes [persistent volumes][17] in the Observability Pipelines Helm chart. +- If you enable [disk buffering][16] for destinations, you must enable Kubernetes [persistent volumes][17] in the Observability Pipelines Helm chart. See also [Persistence and pod scheduling](#persistence-and-pod-scheduling) for more information. - If you are using a firewall, see [Add domains to firewall allowlist](#add-domains-to-firewall-allowlist). See [Update Existing Pipelines][13] if you want to make changes to your pipeline's configuration. @@ -761,6 +761,16 @@ When you install the Observability Pipelines Worker on Kubernetes, the Helm char - A ClusterIP service that provides a single virtual IP and DNS name for the Worker. This enables load balancing across Worker Pods for internal cluster traffic. +### Persistence and pod scheduling + +The Worker runs as a Kubernetes StatefulSet. When you enable persistent volumes for [disk buffering][16], Datadog recommends keeping the Helm chart's default `podManagementPolicy: Parallel` setting. + +Setting `podManagementPolicy` to `OrderedReady` reduces how often a Worker pod shows a volume multi-attach error. But it also blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown, which slows your pipeline's response to a burst of events. + +With the `Parallel` setting, a rescheduled Worker pod can show a transient volume multi-attach error if Kubernetes assigns it to a new node before its persistent volume detaches from the previous node. The pod recovers on its own. + +See [Multi-attach error when using persistence on Kubernetes][27] for more information. + ### LoadBalancer service If you set `service.type: LoadBalancer` in the Helm chart, Kubernetes provisions a load balancer in supported environments and exposes the Worker Service with an external IP/DNS name. For example, Amazon EKS with the [AWS Load Balancer Controller][19] installed. Use this `LoadBalancer` service when traffic originates outside the cluster. @@ -1096,4 +1106,5 @@ Make sure your Worker logs are [indexed][9] in Log Management for optimal functi [23]: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html [24]: /observability_pipelines/scaling_and_performance/best_practices_for_scaling_observability_pipelines/ [25]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-tutorial.html -[26]: /observability_pipelines/configuration/network_traffic/ \ No newline at end of file +[26]: /observability_pipelines/configuration/network_traffic/ +[27]: /observability_pipelines/monitoring_and_troubleshooting/troubleshooting/#multi-attach-error-when-using-persistence-on-kubernetes \ No newline at end of file diff --git a/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md b/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md index e1d60206bad..963eb9b2d48 100644 --- a/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md +++ b/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md @@ -96,6 +96,12 @@ If the Worker is not starting, Worker logs are not sent to Datadog and are not v ``` An example of `` is `opw-observability-pipelines-worker-0`. +### Multi-attach error when using persistence on Kubernetes + +If you enabled [disk buffering][24] for destinations and see a Worker pod stuck in `Pending` with a volume multi-attach error after Kubernetes reschedules it to a new node, this is expected. The error occurs because the persistent volume from the previous node hasn't finished detaching. The pod recovers on its own. + +Datadog recommends keeping the Worker StatefulSet's default `podManagementPolicy: Parallel` setting even when you see this error. Switching to `OrderedReady` reduces how often the error appears but it blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown. This slows your pipeline's response to a burst of events. + ### Certificate verify failed If you see an error with `certificate verify failed` and `self-signed certificate in certificate chain`, see [TLS certificates][16]. Observability Pipelines does not accept self-signed certificates because they are not secure. @@ -213,3 +219,4 @@ If your log timestamps are in string format and your Databricks table has a time [21]: /observability_pipelines/configuration/install_the_worker/#add-domains-to-firewall-allowlist [22]: /observability_pipelines/destinations/databricks#convert-string-timestamps-to-timestamp-format [23]: /observability_pipelines/processors/generate_metrics/#convert-string-timestamp-to-timestamp-format +[24]: /observability_pipelines/scaling_and_performance/buffering_and_backpressure/#destination-buffers