Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions content/en/logs/log_configuration/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ For each pipeline, administrators can choose the following edit scopes:
- **Editor**: Only specified users, teams, or roles can edit pipeline configuration and processors.
- **Processor Editor**: Only the processors (including nested pipelines) can be edited by specified users, teams, or roles. No one can modify the pipeline attributes, such as its filter query or its order in the global pipeline list.

<div class="alert alert-warning">Granting a user access to a pipeline's restriction list does not automatically grant the <code>logs_write_pipelines</code> or <code>logs_write_processors</code> permissions. Administrators must grant those permissions separately.</div>
<div class="alert alert-warning">Granting a user access to a pipeline's restriction list does not automatically grant the <code>logs_write_pipelines</code> or <code>logs_write_processors</code> permissions. Administrators must grant those permissions separately.</div>

You can manage these permissions programmatically through [**API**][14] and **Terraform**.

Expand Down Expand Up @@ -224,7 +224,8 @@ By default, Datadog SDKs can [automatically inject span IDs into your logs][1].

4. Name your pipeline.
5. (Optional) Add a description and tags to the pipeline to indicate its purpose and ownership. Pipeline tags do not affect logs, but can be used to filter and search within the [Pipelines page][8].
6. Press {{< ui >}}Create{{< /ui >}}.
6. (Optional) Add a [Pipeline Simulation](#simulate-pipeline-changes) to preview the impact of your changes before applying them.
7. Press {{< ui >}}Create{{< /ui >}}.

An example of a log transformed by a pipeline:

Expand Down Expand Up @@ -265,6 +266,7 @@ It's also possible to copy an integration pipeline using the clone button.
1. Navigate to [Pipelines][8] in Datadog.
2. Hover over a pipeline and click the arrow next to it to expand processors and nested pipelines.
3. Select {{< ui >}}Add Processor{{< /ui >}} or {{< ui >}}Add Nested Pipeline{{< /ui >}}.
4. (Optional) Add a [Pipeline Simulation](#simulate-pipeline-changes) to preview the impact of your changes before applying them.

### Processors

Expand All @@ -286,6 +288,25 @@ Move a pipeline into another pipeline to make it into a nested pipeline:
1. Select the pipeline you want to move the original pipeline into. **Note**: Pipelines containing nested pipelines can only be moved to another top level position. They cannot be moved into another pipeline.
1. Click {{< ui >}}Move{{< /ui >}}.

## Simulate pipeline changes

Pipeline Simulation previews how your changes affect logs before you apply them. It uses a live tail of your logs, processed with your proposed changes. Open it while creating or editing a pipeline or processor.

{{< img src="logs/processing/pipelines/pipeline_simulation.png" alt="The Pipeline Simulation view showing the pipeline's processors on the left and the diff of a selected log on the right" >}}

When you add or edit processors, compare each log's before and after state:

- **Before** shows the log as processed by the pipeline's current, published configuration.
- **After** previews the log with the changes you're making in this simulation applied.

**Note**: **Before** reflects the state prior to the specific change you're making, not the raw log before the entire pipeline. Both states reflect the rest of the pipeline chain, including processors that run after this step.

To narrow the log list, use the query filter or filter by impact:

- **All logs**: every log in the live tail.
- **Impacted logs**: only logs changed by your edits in this session.
- **Not impacted logs**: only logs your edits leave unchanged.

## Manage your pipelines

Identify when the last change to a pipeline or processor was made and which user made the change using the modification information on the pipeline. Filter your pipelines using this modification information, as well as other faceted properties such as whether the pipeline is enabled or read-only.
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading