From dbf2a49ea216a033292c3aaefb03e6cd30ae0c43 Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 4 Sep 2026 14:49:27 -0400 Subject: [PATCH 1/3] add info --- .../install_the_worker/_index.mdoc.md | 15 +++++++++++++-- .../troubleshooting.md | 7 +++++++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md index 35731d80363..8b67eca3264 100644 --- a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md +++ b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md @@ -743,7 +743,7 @@ To upgrade the Worker, update the Worker image version in your CloudFormation st ``` --set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282 ``` -- If you enable [disk buffering][16] for destinations, you must enable Kubernetes [persistent volumes][17] in the Observability Pipelines Helm chart. +- If you enable [disk buffering][16] for destinations, you must enable Kubernetes [persistent volumes][17] in the Observability Pipelines Helm chart. See [Persistence and pod scheduling](#persistence-and-pod-scheduling) for more information. - If you are using a firewall, see [Add domains to firewall allowlist](#add-domains-to-firewall-allowlist). See [Update Existing Pipelines][13] if you want to make changes to your pipeline's configuration. @@ -761,6 +761,16 @@ When you install the Observability Pipelines Worker on Kubernetes, the Helm char - A ClusterIP service that provides a single virtual IP and DNS name for the Worker. This enables load balancing across Worker Pods for internal cluster traffic. +### Persistence and pod scheduling + +The Worker runs as a Kubernetes StatefulSet. When you enable persistent volumes for [disk buffering][16], Datadog recommends keeping the Helm chart's default `podManagementPolicy: Parallel` setting. + +Setting `podManagementPolicy` to `OrderedReady` reduces how often a Worker pod shows a volume multi-attach error. But it also blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown, which slows your pipeline's response to a burst of events. + +With `Parallel`, a rescheduled Worker pod can show a transient volume multi-attach error if Kubernetes assigns it to a new node before its persistent volume detaches from the previous node. The pod recovers on its own, and this doesn't cause data loss. + +See [Multi-attach error when using persistence on Kubernetes][27] for more information. + ### LoadBalancer service If you set `service.type: LoadBalancer` in the Helm chart, Kubernetes provisions a load balancer in supported environments and exposes the Worker Service with an external IP/DNS name. For example, Amazon EKS with the [AWS Load Balancer Controller][19] installed. Use this `LoadBalancer` service when traffic originates outside the cluster. @@ -1096,4 +1106,5 @@ Make sure your Worker logs are [indexed][9] in Log Management for optimal functi [23]: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html [24]: /observability_pipelines/scaling_and_performance/best_practices_for_scaling_observability_pipelines/ [25]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-tutorial.html -[26]: /observability_pipelines/configuration/network_traffic/ \ No newline at end of file +[26]: /observability_pipelines/configuration/network_traffic/ +[27]: /observability_pipelines/monitoring_and_troubleshooting/troubleshooting/#multi-attach-error-when-using-persistence-on-kubernetes \ No newline at end of file diff --git a/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md b/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md index e1d60206bad..e128bee3e73 100644 --- a/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md +++ b/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md @@ -96,6 +96,12 @@ If the Worker is not starting, Worker logs are not sent to Datadog and are not v ``` An example of `` is `opw-observability-pipelines-worker-0`. +### Multi-attach error when using persistence on Kubernetes + +If you enabled [disk buffering][24] for destinations and see a Worker pod stuck in `Pending` with a volume multi-attach error after Kubernetes reschedules it to a new node, this is expected. The error occurs because the persistent volume from the previous node hasn't finished detaching. The pod recovers on its own, usually within a few minutes, and this doesn't cause data loss. + +Datadog recommends keeping the Worker StatefulSet's default `podManagementPolicy: Parallel` setting even when you see this error. Switching to `OrderedReady` reduces how often the error appears. But it blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown, which slows your pipeline's response to a burst of events. + ### Certificate verify failed If you see an error with `certificate verify failed` and `self-signed certificate in certificate chain`, see [TLS certificates][16]. Observability Pipelines does not accept self-signed certificates because they are not secure. @@ -213,3 +219,4 @@ If your log timestamps are in string format and your Databricks table has a time [21]: /observability_pipelines/configuration/install_the_worker/#add-domains-to-firewall-allowlist [22]: /observability_pipelines/destinations/databricks#convert-string-timestamps-to-timestamp-format [23]: /observability_pipelines/processors/generate_metrics/#convert-string-timestamp-to-timestamp-format +[24]: /observability_pipelines/scaling_and_performance/buffering_and_backpressure/#destination-buffers From 1edbc24e8b7c4df75e5bcb5976392c8fc0093797 Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 4 Sep 2026 16:01:22 -0400 Subject: [PATCH 2/3] Apply batched suggestions from code review Co-authored-by: May Lee --- .../configuration/install_the_worker/_index.mdoc.md | 4 ++-- .../monitoring_and_troubleshooting/troubleshooting.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md index 8b67eca3264..220a77207a1 100644 --- a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md +++ b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md @@ -743,7 +743,7 @@ To upgrade the Worker, update the Worker image version in your CloudFormation st ``` --set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282 ``` -- If you enable [disk buffering][16] for destinations, you must enable Kubernetes [persistent volumes][17] in the Observability Pipelines Helm chart. See [Persistence and pod scheduling](#persistence-and-pod-scheduling) for more information. +- If you enable [disk buffering][16] for destinations, you must enable Kubernetes [persistent volumes][17] in the Observability Pipelines Helm chart. See also [Persistence and pod scheduling](#persistence-and-pod-scheduling) for more information. - If you are using a firewall, see [Add domains to firewall allowlist](#add-domains-to-firewall-allowlist). See [Update Existing Pipelines][13] if you want to make changes to your pipeline's configuration. @@ -767,7 +767,7 @@ The Worker runs as a Kubernetes StatefulSet. When you enable persistent volumes Setting `podManagementPolicy` to `OrderedReady` reduces how often a Worker pod shows a volume multi-attach error. But it also blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown, which slows your pipeline's response to a burst of events. -With `Parallel`, a rescheduled Worker pod can show a transient volume multi-attach error if Kubernetes assigns it to a new node before its persistent volume detaches from the previous node. The pod recovers on its own, and this doesn't cause data loss. +With the `Parallel` setting, a rescheduled Worker pod can show a transient volume multi-attach error if Kubernetes assigns it to a new node before its persistent volume detaches from the previous node. The pod recovers on its own, and this doesn't cause data loss. See [Multi-attach error when using persistence on Kubernetes][27] for more information. diff --git a/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md b/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md index e128bee3e73..8f541dd5958 100644 --- a/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md +++ b/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md @@ -100,7 +100,7 @@ If the Worker is not starting, Worker logs are not sent to Datadog and are not v If you enabled [disk buffering][24] for destinations and see a Worker pod stuck in `Pending` with a volume multi-attach error after Kubernetes reschedules it to a new node, this is expected. The error occurs because the persistent volume from the previous node hasn't finished detaching. The pod recovers on its own, usually within a few minutes, and this doesn't cause data loss. -Datadog recommends keeping the Worker StatefulSet's default `podManagementPolicy: Parallel` setting even when you see this error. Switching to `OrderedReady` reduces how often the error appears. But it blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown, which slows your pipeline's response to a burst of events. +Datadog recommends keeping the Worker StatefulSet's default `podManagementPolicy: Parallel` setting even when you see this error. Switching to `OrderedReady` reduces how often the error appears but it blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown. This slows your pipeline's response to a burst of events. ### Certificate verify failed From 3b12f2eed448850b312b035bb447cf4009030fdd Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 4 Sep 2026 16:09:49 -0400 Subject: [PATCH 3/3] updates --- .../configuration/install_the_worker/_index.mdoc.md | 2 +- .../monitoring_and_troubleshooting/troubleshooting.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md index 220a77207a1..14a72edd84a 100644 --- a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md +++ b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md @@ -767,7 +767,7 @@ The Worker runs as a Kubernetes StatefulSet. When you enable persistent volumes Setting `podManagementPolicy` to `OrderedReady` reduces how often a Worker pod shows a volume multi-attach error. But it also blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown, which slows your pipeline's response to a burst of events. -With the `Parallel` setting, a rescheduled Worker pod can show a transient volume multi-attach error if Kubernetes assigns it to a new node before its persistent volume detaches from the previous node. The pod recovers on its own, and this doesn't cause data loss. +With the `Parallel` setting, a rescheduled Worker pod can show a transient volume multi-attach error if Kubernetes assigns it to a new node before its persistent volume detaches from the previous node. The pod recovers on its own. See [Multi-attach error when using persistence on Kubernetes][27] for more information. diff --git a/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md b/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md index 8f541dd5958..963eb9b2d48 100644 --- a/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md +++ b/hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md @@ -98,7 +98,7 @@ If the Worker is not starting, Worker logs are not sent to Datadog and are not v ### Multi-attach error when using persistence on Kubernetes -If you enabled [disk buffering][24] for destinations and see a Worker pod stuck in `Pending` with a volume multi-attach error after Kubernetes reschedules it to a new node, this is expected. The error occurs because the persistent volume from the previous node hasn't finished detaching. The pod recovers on its own, usually within a few minutes, and this doesn't cause data loss. +If you enabled [disk buffering][24] for destinations and see a Worker pod stuck in `Pending` with a volume multi-attach error after Kubernetes reschedules it to a new node, this is expected. The error occurs because the persistent volume from the previous node hasn't finished detaching. The pod recovers on its own. Datadog recommends keeping the Worker StatefulSet's default `podManagementPolicy: Parallel` setting even when you see this error. Switching to `OrderedReady` reduces how often the error appears but it blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown. This slows your pipeline's response to a burst of events.