Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -743,7 +743,7 @@ To upgrade the Worker, update the Worker image version in your CloudFormation st
```
--set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282
```
- If you enable [disk buffering][16] for destinations, you must enable Kubernetes [persistent volumes][17] in the Observability Pipelines Helm chart.
- If you enable [disk buffering][16] for destinations, you must enable Kubernetes [persistent volumes][17] in the Observability Pipelines Helm chart. See also [Persistence and pod scheduling](#persistence-and-pod-scheduling) for more information.
- If you are using a firewall, see [Add domains to firewall allowlist](#add-domains-to-firewall-allowlist).

See [Update Existing Pipelines][13] if you want to make changes to your pipeline's configuration.
Expand All @@ -761,6 +761,16 @@ When you install the Observability Pipelines Worker on Kubernetes, the Helm char
- A ClusterIP service that provides a single virtual IP and DNS name for the Worker.
This enables load balancing across Worker Pods for internal cluster traffic.

### Persistence and pod scheduling

The Worker runs as a Kubernetes StatefulSet. When you enable persistent volumes for [disk buffering][16], Datadog recommends keeping the Helm chart's default `podManagementPolicy: Parallel` setting.

Setting `podManagementPolicy` to `OrderedReady` reduces how often a Worker pod shows a volume multi-attach error. But it also blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown, which slows your pipeline's response to a burst of events.

With the `Parallel` setting, a rescheduled Worker pod can show a transient volume multi-attach error if Kubernetes assigns it to a new node before its persistent volume detaches from the previous node. The pod recovers on its own.

See [Multi-attach error when using persistence on Kubernetes][27] for more information.

### LoadBalancer service

If you set `service.type: LoadBalancer` in the Helm chart, Kubernetes provisions a load balancer in supported environments and exposes the Worker Service with an external IP/DNS name. For example, Amazon EKS with the [AWS Load Balancer Controller][19] installed. Use this `LoadBalancer` service when traffic originates outside the cluster.
Expand Down Expand Up @@ -1096,4 +1106,5 @@ Make sure your Worker logs are [indexed][9] in Log Management for optimal functi
[23]: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html
[24]: /observability_pipelines/scaling_and_performance/best_practices_for_scaling_observability_pipelines/
[25]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-tutorial.html
[26]: /observability_pipelines/configuration/network_traffic/
[26]: /observability_pipelines/configuration/network_traffic/
[27]: /observability_pipelines/monitoring_and_troubleshooting/troubleshooting/#multi-attach-error-when-using-persistence-on-kubernetes
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,12 @@ If the Worker is not starting, Worker logs are not sent to Datadog and are not v
```
An example of `<pod-name>` is `opw-observability-pipelines-worker-0`.

### Multi-attach error when using persistence on Kubernetes

If you enabled [disk buffering][24] for destinations and see a Worker pod stuck in `Pending` with a volume multi-attach error after Kubernetes reschedules it to a new node, this is expected. The error occurs because the persistent volume from the previous node hasn't finished detaching. The pod recovers on its own.

Datadog recommends keeping the Worker StatefulSet's default `podManagementPolicy: Parallel` setting even when you see this error. Switching to `OrderedReady` reduces how often the error appears but it blocks the StatefulSet from scaling up while terminating replicas finish their graceful shutdown. This slows your pipeline's response to a burst of events.

### Certificate verify failed

If you see an error with `certificate verify failed` and `self-signed certificate in certificate chain`, see [TLS certificates][16]. Observability Pipelines does not accept self-signed certificates because they are not secure.
Expand Down Expand Up @@ -213,3 +219,4 @@ If your log timestamps are in string format and your Databricks table has a time
[21]: /observability_pipelines/configuration/install_the_worker/#add-domains-to-firewall-allowlist
[22]: /observability_pipelines/destinations/databricks#convert-string-timestamps-to-timestamp-format
[23]: /observability_pipelines/processors/generate_metrics/#convert-string-timestamp-to-timestamp-format
[24]: /observability_pipelines/scaling_and_performance/buffering_and_backpressure/#destination-buffers
Loading