diff --git a/content/en/tracing/live_debugger/_index.md b/content/en/tracing/live_debugger/_index.md index 0faee3a5423..d3c3db210b2 100644 --- a/content/en/tracing/live_debugger/_index.md +++ b/content/en/tracing/live_debugger/_index.md @@ -47,46 +47,179 @@ Live Debugger provides: ## Requirements and setup -Live Debugger supports Python, Java, .NET, Ruby, Node.js, PHP, and Go. It requires the [Datadog Agent][2] (version 7.49.0 or later), an [APM-instrumented application][3], and [Remote Configuration][4]. +Live Debugger supports Python, Java, .NET, Ruby, Node.js, PHP, and Go. It requires: -### Enablement modes +- [Datadog Agent][2] version 7.49.0 or later +- [Datadog SDK][3] installed +- [Unified Service Tagging][27] configured with `service`, `env`, and `version` tags on your deployment +- [Remote Configuration][4] enabled in the Agent +- (Recommended) [Source Code Integration][28] set up -Manage Live Debugger for each service and environment from the Live Debugger {{< ui >}}Settings{{< /ui >}} page. Each service can be in one of three modes: +### Minimum SDK versions -- {{< ui >}}Automatic{{< /ui >}}: Eligible services are enabled automatically. Switch a service to {{< ui >}}Enabled{{< /ui >}} for a faster debugging experience with debug symbols. -- {{< ui >}}Enabled{{< /ui >}}: Live Debugger is ready to use on this service in this environment, with faster setup and a better debugging experience. -- {{< ui >}}Disabled{{< /ui >}}: This service is explicitly disabled and is not enabled automatically. +Live Debugger requires the following minimum Datadog SDK versions: -### Minimum tracer versions - -Live Debugger requires the following minimum tracer versions: - -- [Python][5] ≥ 4.11.0 - [Java][6] ≥ 1.64.0 +- [Python][5] ≥ 4.11.0 - [.NET][7] ≥ 3.46.0 -- [Ruby][9] ≥ 2.35.0 - [Node.js][8] ≥ 5.109.0 - [PHP][10] ≥ 1.21.0 +- [Ruby][9] ≥ 2.35.0 - [Go][22] ≥ 2.9.0 -Older tracer versions might require enabling Live Debugger through an environment variable. Follow the setup instructions for your language: - -{{< card-grid >}} - {{< image-card href="/dynamic_instrumentation/enabling/java" src="integrations_logos/java.png" alt="Java" >}} - {{< image-card href="/dynamic_instrumentation/enabling/python" src="integrations_logos/python.png" alt="Python" >}} - {{< image-card href="/dynamic_instrumentation/enabling/dotnet" src="integrations_logos/dotnet-core.png" alt="Dotnet" >}} - {{< image-card href="/dynamic_instrumentation/enabling/dotnet" src="integrations_logos/dotnet-framework.png" alt="Dotnet" >}} - {{< image-card href="/dynamic_instrumentation/enabling/nodejs" src="integrations_logos/nodejs.png" alt="Node.js" >}} - {{< image-card href="/dynamic_instrumentation/enabling/ruby" src="integrations_logos/ruby.png" alt="Ruby" >}} - {{< image-card href="/dynamic_instrumentation/enabling/php" src="integrations_logos/php.png" alt="PHP" >}} - {{< image-card href="/dynamic_instrumentation/enabling/go" src="integrations_logos/go-metro.png" alt="Go" >}} -{{< /card-grid >}} - -
-Why DI instructions? -Live Debugger is built on Dynamic Instrumentation (DI), so its -setup instructions and limitations also apply here. -
+### Enabling Live Debugger + +For Java, Python, .NET, and Node.js services that meet the minimum SDK and setup requirements, enable Live Debugger from the [Live Debugger Settings page][26]. For other runtime languages or older SDK versions, use the configuration steps below. You can disable Live Debugger for a service and environment from the [Live Debugger Settings page][26], regardless of runtime language or SDK version. + +{{< programming-lang-wrapper langs="java,python,.NET,nodejs,ruby,php,go" >}} + +{{< programming-lang lang="java" >}} +**Requirement**: Datadog Java SDK version 1.64.0 or higher, running on JDK 8 or higher. + +Start your service with `DD_DYNAMIC_INSTRUMENTATION_ENABLED=true`, along with `DD_SERVICE`, `DD_ENV`, and `DD_VERSION`. The `-javaagent` argument must come before `-jar`: + +```shell +export DD_SERVICE= +export DD_ENV= +export DD_VERSION= +export DD_DYNAMIC_INSTRUMENTATION_ENABLED=true +java \ + -javaagent:dd-java-agent.jar \ + -jar .jar +``` + +**Note**: On JDK 18 and below, classes compiled with the `-parameters` flag (default in Spring 6+, Spring Boot 3+, and Scala) may fail to instrument. +{{< /programming-lang >}} + +{{< programming-lang lang="python" >}} +**Requirement**: Datadog Python SDK (ddtrace) version 4.11.0 or higher. + +Install `ddtrace`, then start your service with `DD_DYNAMIC_INSTRUMENTATION_ENABLED=true` and `ddtrace-run`: + +```shell +pip install ddtrace +export DD_SERVICE= +export DD_ENV= +export DD_VERSION= +export DD_DYNAMIC_INSTRUMENTATION_ENABLED=true +ddtrace-run python -m myapp.py +``` +{{< /programming-lang >}} + +{{< programming-lang lang=".NET" >}} +**Requirement**: Datadog .NET SDK version 3.46.0 or higher. + +Start your service with the following environment variables set: + +```shell +DD_SERVICE= +DD_ENV= +DD_VERSION= +DD_DYNAMIC_INSTRUMENTATION_ENABLED=true +``` +{{< /programming-lang >}} + +{{< programming-lang lang="nodejs" >}} +**Requirement**: Datadog Node.js SDK (dd-trace-js) version 5.109.0 or higher. If your source code is transpiled or bundled (for example, TypeScript, Babel, or Webpack), publish source maps with the deployed application so that logpoints map to the correct lines. + +Start your service with the following environment variables set: + +```shell +DD_SERVICE= +DD_ENV= +DD_VERSION= +DD_DYNAMIC_INSTRUMENTATION_ENABLED=true +``` +{{< /programming-lang >}} + +{{< programming-lang lang="ruby" >}} +Ruby services must be enabled through environment variables. Auto-enablement from the Settings page is not available for Ruby. + +**Requirements:** + +- Datadog Ruby SDK (ddtrace) version 2.35.0 or higher +- Ruby 2.6 or higher (MRI/CRuby only; JRuby is not supported) +- A Rack-based framework (Rails, Sinatra, or other Rack-compatible frameworks). Background workers (such as Sidekiq or Resque) are not supported. +- `RAILS_ENV` or `RACK_ENV` set to `production` + +Start your service with the following environment variables set: + +```shell +export DD_SERVICE= +export DD_ENV= +export DD_VERSION= +export DD_DYNAMIC_INSTRUMENTATION_ENABLED=true +``` + +**Note**: Live Debugger initializes on the first HTTP request. Your service must receive at least one request before you can create a logpoint. +{{< /programming-lang >}} + +{{< programming-lang lang="php" >}} +PHP services must be enabled through environment variables. Auto-enablement from the Settings page is not available for PHP. + +**Requirement**: Datadog PHP SDK (dd-trace-php) version 1.21.0 or higher. + +Start your service with the following environment variables set: + +```shell +DD_SERVICE= +DD_ENV= +DD_VERSION= +DD_DYNAMIC_INSTRUMENTATION_ENABLED=true +``` +{{< /programming-lang >}} + +{{< programming-lang lang="go" >}} +Go services require enabling Live Debugger in both the Datadog Agent and the application. + +**Requirements:** + +- Datadog Agent version 7.73.0 or higher, running on the same host as your application +- Datadog Go SDK version 2.9.0 or higher (or 1.74.6 or higher on the v1 line) +- Linux kernel 5.17 or higher + +**Configure the Datadog Agent** using one of the following methods, depending on how you deploy the Agent: + +- **Configuration YAML file**: Update `system-probe.yaml` (located alongside `datadog.yaml`) with the following. For more information, see Agent configuration files. + + ```yaml + dynamic_instrumentation: + enabled: true + ``` + +- **Environment variable**: Add the following to your Datadog Agent manifest: + + ``` + DD_DYNAMIC_INSTRUMENTATION_ENABLED=true + ``` + +- **Helm**: Add the following to your Helm chart: + + ```yaml + datadog: + dynamicInstrumentationGo: + enabled: true + ``` + +**Configure your service**: After the Agent is configured, services on the same host can be enabled from the Live Debugger Settings page, or by starting the service with the following environment variables set: + +```shell +DD_SERVICE= +DD_ENV= +DD_VERSION= +DD_DYNAMIC_INSTRUMENTATION_ENABLED=true +``` +{{< /programming-lang >}} + +{{< /programming-lang-wrapper >}} + +#### Enablement modes + +Each service and environment is in one of three modes on the Live Debugger Settings page: + +- **Automatic**: Live Debugger has not been set to Enabled or Disabled yet on this service and environment. This setting changes to **Enabled** automatically the first time a Debug Session is started. For a faster first-time debugging experience, switch the setting to **Enabled** in advance. +- **Enabled**: For eligible services, this setting means Live Debugger is activated on the selected service and environment, including debug symbol uploads and faster delivery of new logpoints. +- **Disabled**: This setting blocks logpoints from being created or re-activated on a given service and environment. It applies regardless of runtime language or SDK version. ### Permissions @@ -105,7 +238,7 @@ Live Debugger generates logs that are sent to Datadog and appear alongside your If you use [Exclusion filters][11], make sure Live Debugger logs are not filtered: 1. Create a logs index and [configure it][12] to the desired retention with **no sampling**. -2. Set the filter to match on the `source:dd_debugger` tag. All Dynamic Instrumentation logs have this source. +2. Set the filter to match on the `source:dd_debugger` tag. All Live Debugger logs have this source. 3. Make sure the new index takes precedence over any other with filters that match that tag, because the first match wins. ### Link your source code @@ -148,14 +281,14 @@ Live Debugger data might contain sensitive information, especially when using th Live Debugger has two redaction modes: -- {{< ui >}}Strict Mode{{< /ui >}}: Redacts all values except numbers and Booleans. -- {{< ui >}}Targeted Mode{{< /ui >}}: Redacts known sensitive patterns such as credit card numbers, API keys, IPs, and other PII. It also runs a high-entropy secrets scanner that automatically redacts likely secrets, which appear as `[REDACTED:HIGH_ENTROPY]` in captured data. +- **Strict Mode**: Redacts all values except numbers and Booleans. Bits Live Debugger is not available for service and environment combinations set to Strict Mode. +- **Targeted Mode**: Redacts known sensitive patterns such as credit card numbers, API keys, and IPs. It also runs a high-entropy secrets scanner that automatically redacts likely secrets, which appear as `[REDACTED:HIGH_ENTROPY]` in captured data. These redaction modes cannot be disabled, only switched. Targeted Mode is applied automatically in common pre-production environments such as `staging` or `preprod`. Changing the redaction mode requires the **Live Debugger Redaction Write** permission. #### Identifier-based redaction -Variable values associated with common sensitive identifiers (for example, `password`, `accessToken`, and similar terms) are scrubbed before captured data leaves the host. Each tracer includes additional language-specific redaction rules. +Variable values associated with common sensitive identifiers (for example, `password`, `accessToken`, and similar terms) are scrubbed before captured data leaves the host. Additional language-specific redaction rules are built into each SDK. You can extend redaction behavior through: @@ -167,15 +300,13 @@ See the [sensitive data scrubbing][1] instructions and [Sensitive Data Scanner][ ### Bits Live Debugger -[Bits Live Debugger][23] lets you investigate a running service by describing the issue in plain language. Bits Code handles logpoint placement, captures variable snapshots, and helps interpret the results. - -## Impact on performance and billing - -Enabling Live Debugger on a service does not trigger data capture or impact performance. Data capture only occurs when there are active Debug Sessions on that service. +{{< beta-callout url="https://www.datadoghq.com/product-preview/debug-with-bits/" >}} +Bits Live Debugger is in Preview. Request access to join the waiting list. +{{< /beta-callout >}} -**Performance impact**: Datadog's agent-driven instrumentation ensures minimal impact on application performance; sampling logic, rate limits, and built-in budgets prevent runaway data capture. +[Bits Live Debugger][23] lets you investigate a running service by describing the issue in plain language. Bits Code handles logpoint placement, captures variable snapshots, and helps interpret the results. -**Pricing impact**: Logs captured by Datadog are all billed the same way. This applies whether they are generated from Live Debugger or logger lines in your source code. With Live Debugger, the logpoints automatically expire after the set time period, limiting unnecessary data accumulation and costs. Monitor your [Datadog Plan & Usage page][18] for any unexpected increases after utilizing a new feature. +**Note**: Bits Live Debugger requires the service and environment to be in Targeted Mode. See [Mode-based redaction][24] for details. ## Limitations @@ -207,10 +338,13 @@ The following constraints apply to Live Debugger usage and configuration: [15]: /dynamic_instrumentation/expression-language/ [16]: /dynamic_instrumentation/enabling [17]: /dynamic_instrumentation/sensitive-data-scrubbing/#redact-based-on-variable-values-with-sensitive-data-scanner -[18]: https://app.datadoghq.com/account/billing [19]: /dynamic_instrumentation/ [20]: /tracing/code_origin [21]: /account_management/rbac/permissions#apm [22]: /tracing/trace_collection/automatic_instrumentation/dd_libraries/go [23]: /tracing/live_debugger/bits-live-debugger/ [24]: #mode-based-redaction +[26]: https://app.datadoghq.com/debugging/settings +[27]: /getting_started/tagging/unified_service_tagging/ +[28]: /integrations/guide/source-code-integration/ +[29]: /agent/configuration/agent-configuration-files/?tab=agentv6v7#agent-main-configuration-file diff --git a/content/en/tracing/trace_collection/dynamic_instrumentation/_index.md b/content/en/tracing/trace_collection/dynamic_instrumentation/_index.md index 9c6eff3ebb1..9438ee264fe 100644 --- a/content/en/tracing/trace_collection/dynamic_instrumentation/_index.md +++ b/content/en/tracing/trace_collection/dynamic_instrumentation/_index.md @@ -1,6 +1,6 @@ --- title: Dynamic Instrumentation -description: Add instrumentation to your running production systems without restarts to collect logs, metrics, spans, and tags from any location in your code. +description: Add instrumentation to your running production systems without restarts to collect metrics, spans, and tags from any location in your code. aliases: - /dynamic_instrumentation/how-it-works/ - /dynamic_instrumentation/ @@ -11,7 +11,7 @@ further_reading: - link: "/dynamic_instrumentation/expression-language/" tag: "Documentation" text: "Learn more about the Dynamic Instrumentation Expression Language" -- link: "dynamic_instrumentation/sensitive-data-scrubbing/" +- link: "/dynamic_instrumentation/sensitive-data-scrubbing/" tag: "Documentation" text: "Removing sensitive information from your Dynamic Instrumentation data" - link: "/tracing/trace_collection/dd_libraries" @@ -26,14 +26,11 @@ further_reading: - link: "/metrics" tag: "Documentation" text: "Learn more about Metrics" -- link: "https://www.datadoghq.com/blog/dynamic-instrumentation-application-logging/" - tag: "Blog" - text: "Use Datadog Dynamic Instrumentation to add application logs without redeploying" --- ## Overview -Dynamic Instrumentation allows you to add instrumentation into your running production systems without any restarts and at any location in your application's code, including third-party libraries. You can add or modify telemetry for logs, metrics, spans, and corresponding tagging, from the Datadog UI. Dynamic Instrumentation has low overhead and has no side effects on your system. +Dynamic Instrumentation lets you add metrics, spans, and span tags to running production systems without restarts or code changes, including in third-party libraries. If you are interested in trying out the latest user experience improvements for Dynamic Instrumentation, consider opting into the [autocomplete and search Preview][17]. @@ -58,165 +55,119 @@ The following permissions are required to use Dynamic Instrumentation: - One of the following write permissions: - **Dynamic Instrumentation Write Configuration** (`debugger_write`) - Required to create or modify instrumentations in any environment. - **Dynamic Instrumentation Write Pre-Prod** (`debugger_write_preprod`) - Required to create or modify instrumentations in known pre-production environments only (such as staging or QA). -- **Dynamic Instrumentation Capture Variables** (`debugger_capture_variables`) - Required to use the {{< ui >}}Capture method parameters and local variables{{< /ui >}} option. - For more information about roles and how to assign roles to users, see [Role Based Access Control][8]. -### Create a logs index +### Enable Dynamic Instrumentation -Dynamic Instrumentation creates "dynamic logs" that are sent to Datadog and appear alongside your regular application logs. +Dynamic Instrumentation supports Java, Python, .NET, and PHP. For Java, Python, and .NET services that meet the minimum SDK requirements, manage Dynamic Instrumentation for each service and environment from the [Dynamic Instrumentation Settings page][16]. For PHP and older SDK versions, use manual enablement. -If you use [Exclusion filters][9], verify that Dynamic Instrumentation logs are not filtered: +On supported SDK versions, Datadog automatically attempts to enable the service the first time you create an instrumentation for it, as long as all prerequisites are met. -1. Create a logs index and [configure it][10] to the desired retention with **no sampling**. -2. Set the filter to match on the `source:dd_debugger` tag. All Dynamic Instrumentation logs have this source. -3. Verify that the new index takes precedence over any other with filters that match that tag, because the first match wins. +#### In-app enablement -### Enable Dynamic Instrumentation +In-app enablement is supported on the following minimum SDK versions: + +- [Java][18] ≥ 1.64.0 +- [Python][19] ≥ 4.11.0 +- [.NET][20] ≥ 3.46.0 + +Datadog recommends keeping your SDK up to date to take advantage of in-app enablement. + +#### Manual enablement -To enable Dynamic Instrumentation on a service, go to the [in-app setup page][16]. +Manual enablement is required for PHP, and for older SDK versions of Java, Python, and .NET. You can also choose manual enablement on supported SDK versions if you prefer to manage enablement through environment variables, for example, to enable Dynamic Instrumentation in bulk across many services. -For more detailed instructions, select your runtime below: +Select your runtime for manual enablement instructions: {{< card-grid card_width="170px" >}} {{< image-card href="/dynamic_instrumentation/enabling/java" src="integrations_logos/java.png" alt="Java" >}} {{< image-card href="/dynamic_instrumentation/enabling/python" src="integrations_logos/python.png" alt="Python" >}} {{< image-card href="/dynamic_instrumentation/enabling/dotnet" src="integrations_logos/dotnet-core.png" alt="Dotnet" >}} {{< image-card href="/dynamic_instrumentation/enabling/dotnet" src="integrations_logos/dotnet-framework.png" alt="Dotnet" >}} - {{< image-card href="/dynamic_instrumentation/enabling/nodejs" src="integrations_logos/nodejs.png" alt="Node.js" >}} - {{< image-card href="/dynamic_instrumentation/enabling/ruby" src="integrations_logos/ruby.png" alt="Ruby" >}} {{< image-card href="/dynamic_instrumentation/enabling/php" src="integrations_logos/php.png" alt="PHP" >}} - {{< image-card href="/dynamic_instrumentation/enabling/go" src="integrations_logos/go-metro.png" alt="Go" >}} {{< /card-grid >}} +
Dynamic Instrumentation and Live Debugger share the same enablement state per service and environment. Enabling or disabling one also enables or disables the other for that service and environment, although the two products have separate permissions and Settings pages.
### Limitations - Dynamic Instrumentation is not compatible with Azure App Services or serverless environments. -- Not all probe types are supported in every language. See the [language-specific setup instructions](#enable-dynamic-instrumentation) for supported features and limitations. +- Not all instrumentation types are supported in every language. See the [language-specific setup instructions](#enable-dynamic-instrumentation) for supported features and limitations. - The Java SDK does not support Kotlin coroutines. ## Explore Dynamic Instrumentation -Dynamic Instrumentation can help you understand what your application is doing at runtime. By adding a Dynamic Instrumentation probe you are exporting additional data from your application, without the need to change code or redeploy it. +Dynamic Instrumentation can help you understand what your application is doing at runtime. By adding an instrumentation at a specific code location, you can capture additional telemetry from your application without the need to change code or redeploy it. -### Using probes +### Using instrumentations -A probe allows you to collect data from specific points in your code without halting the execution of the program. +An instrumentation allows you to collect additional telemetry from specific points in your code without halting the execution of the program. -Think of using probes as enhancing your observability by adding dynamic logs, metrics, and spans to a running application without needing to change code, deploy it, or restart a service. You can gather data immediately without disturbing the user experience or requiring lengthy deployments. +Dynamic spans, span tags, and metrics are a UI-based alternative to adding custom instrumentation directly to your source code. Datadog receives the instrumentation configurations you define and dynamically applies them to the running service without requiring restarts. -As a developer, you can also think of a probe as a "non-breaking breakpoint". In traditional debugging, a breakpoint is a point in the program where the execution stops, allowing the developer to inspect the state of the program at that point. However, in real-world production environments, it's not practical or even possible to stop the execution of the program. Probes fill in this gap by allowing you to inspect variable state in production environments in a non-intrusive way. +Datadog captures and processes spans, tags, and metrics generated by Dynamic Instrumentation like other telemetry from the running application. Unlike manual custom instrumentation, Dynamic Instrumentation does not require code changes, deployments, or service restarts. To stop collecting data, disable the instrumentation in Datadog. -### Creating a probe +### Creating an instrumentation -All probe types require the same initial setup: +All instrumentation types require the same initial setup: 1. Go to the [Dynamic Instrumentation page][12]. -1. Click {{< ui >}}Create Probe{{< /ui >}} in the top right, or click the three-dot menu on a service and select {{< ui >}}Add a probe for this service{{< /ui >}}. -1. If they are not prefilled, choose service, runtime, environment, and version. -1. In the source code, specify where to set the probe by selecting either a class and method or a source file and line. If you opted into the [autocomplete and search Preview][17], autocomplete shows suggestions for selecting a class or method. - -See the individual probe types below for specific creation steps for each probe type. - -Alternatively, you can create a probe from these other contexts: - -Profiling -: On a profiler flame graph, you can create a probe for a method by selecting {{< ui >}}Instrument this frame with a probe{{< /ui >}} from the frame's context menu. - -Error Tracking -: On a stack trace, mouse over a stack frame and click {{< ui >}}Instrument{{< /ui >}}. This prefills the probe creation form with the Issue context. - - -### Creating log probes +1. Click **Create Instrumentation** in the top right, or click the three-dot menu on a service and select **Add an instrumentation for this service**. +1. If they are not prefilled, choose service, runtime, environment, and version (optional). +1. Specify where to set the instrumentation in the source code by selecting either a class and method, or a specific line of code in a file. When autocomplete and search are available, use them to find files, methods, or symbols. -A *log probe* emits a log when it executes. +For the best experience, set up [Source Code Integration][7] to view code directly in Datadog and select instrumentation locations as you would with breakpoints in an IDE. -To create a log probe: +See the individual instrumentation types below for specific creation steps for each instrumentation type. -1. Select {{< ui >}}Log{{< /ui >}} as the probe type. -1. Complete the [generic probe setup](#creating-a-probe) (choose service, environment, version, and probe location). -1. Define a log message template. You can use the Dynamic Instrumentation expression language to reference values from the execution context. -1. (In Preview) Optionally enable extra data capturing from the probe. -1. Optionally define a condition using the Dynamic Instrumentation expression language. The log is emitted when the expression evaluates to true. +
Dynamic log instrumentations are supported in Live Debugger. Use Live Debugger to capture logs and variable snapshots in real time from running applications.
-Log probes are enabled by default on all service instances that match the specified environment and version. They are rate-limited to execute at most 5000 times per second, on each instance of your service. +### Creating dynamic metrics -You must set a log message template on every log probe. The template supports embedding [expressions][15] inside curly brackets. For example: `User {user.id} purchased {count(products)} products`. +A dynamic metric emits a metric when it executes. -You can also set a condition on a log probe using the [expression language][15]. The expression must evaluate to a Boolean. The probe executes if the expression is true, and does not capture or emit any data if the expression is false. +To create a dynamic metric: -{{< img src="dynamic_instrumentation/log_probe.png" alt="Creating a Dynamic Instrumentation log probe" >}} - -(In Preview) If you enable {{< ui >}}Capture method parameters and local variables{{< /ui >}} on the log probe, all execution context is added the log event: - - **Method arguments**, **local variables**, and **fields**, with the following default limits: - - Follow references three levels deep (configurable in the UI). - - The first 100 items inside collections. - - The first 255 characters for string values. - - 20 fields inside objects. Static fields are not collected. - - Call **stack trace**. - - Caught and uncaught **exceptions**. - -Probes with this setting enabled are rate-limited to one hit per second. - -

Warning: The captured data may contain sensitive information, including personal data, passwords, and secrets such as AWS keys.

To ensure this information is properly redacted:

    -
  • Datadog Dynamic Instrumentation employs several techniques to redact sensitive information. To learn more about the default mechanisms or how to extend the it to meet your needs, read Sensitive Data Scrubbing.
  • -
  • Turn off the {{< ui >}}Capture method parameters and local variables{{< /ui >}} option and explicitly select the variables you want to include in the log message template. Doing so ensures that log probes contain only data related to the variables that you specifically identify, thus reducing the risk of unintentional sensitive data leaks.
  • -
  • If you are the Administrator of your Datadog account and would like to prevent other users from being able to use the {{< ui >}}Capture method parameters and local variables{{< /ui >}} option, you can revoke their Dynamic Instrumentation Capture Variables (debugger_capture_variables) permission.

Alternatively, if you need to log this data but want to mitigate the risk associated with it being accessible in the Datadog product, you can limit which users in your organization can view the captured data by setting up a Restriction query on source:dd_debugger.

- -### Creating metric probes - -A *metric probe* emits a metric when it executes. - -To create a metric probe: - -1. Select {{< ui >}}Metric{{< /ui >}} as the probe type. -1. Complete the [generic probe setup](#creating-a-probe) (choose service, environment, version, and probe location). +1. Select **Metric** as the instrumentation type. +1. Complete the [generic instrumentation setup](#creating-an-instrumentation) (choose service, environment, and location). 1. Specify a name for the metric, which will be prefixed with `dynamic.instrumentation.metric.probe.`. 1. Select a metric type (count, gauge, or histogram). 1. Choose the value of the metric using the [Dynamic Instrumentation expression language][15]. You can use any numeric value you'd like from the execution context, such as a method parameter, local variable, a class field, or an expression that yields a numeric value. For count metrics this is optional, and if you omit it, every invocation increments the count by one. -{{< img src="dynamic_instrumentation/metric_probe.png" alt="Creating a Dynamic Instrumentation metric probe" >}} - -Metric probes are automatically enabled on all service instances that match the configured environment and version. Metric probes are not rate limited and execute every time the method or line is invoked. +Metric instrumentations are automatically enabled on all service instances that match the configured environment and version. Metric instrumentations are not rate limited and execute every time the method or line is invoked. -Dynamic Instrumentation metric probes support the following metric types: +Dynamic Instrumentation supports the following metric types: - {{< ui >}}Count{{< /ui >}}: Counts how many times a given method or line is executed. Can be combined with [metric expressions][15] to use the value of a variable to increment the count. - {{< ui >}}Gauge{{< /ui >}}: Generates a gauge based on the last value of a variable. This metric requires a [metric expression][15]. - {{< ui >}}Histogram{{< /ui >}}: Generates a statistical distribution of a variable. This metric requires a [metric expression][15]. -### Creating span probes - -A *span probe* emits a span when a method is executed. +### Creating dynamic spans -To create a span probe: +A *dynamic span* emits a span when a method is executed. -1. Select {{< ui >}}Span{{< /ui >}} as the probe type. -1. Complete the [generic probe setup](#creating-a-probe) (choose service, environment, version, and probe location). +To create a dynamic span: -{{< img src="dynamic_instrumentation/span_probe.png" alt="Creating a Dynamic Instrumentation span probe" >}} +1. Select **Span** as the instrumentation type. +1. Complete the [generic instrumentation setup](#creating-an-instrumentation) (choose service, environment, version, and location). -You can use a *span probe* as an alternative to [creating new spans with Custom Instrumentation][13]. If the method throws an exception, the details of the exception are associated with the newly created span's `error` tag. +You can use a *dynamic span* as an alternative to [creating new spans with Custom Instrumentation][13]. If the method throws an exception, the details of the exception are associated with the newly created span's `error` tag. -### Creating span tag probes +### Creating dynamic span tags -A *span tag* probe adds a tag value to an existing span. You can add a tag either to the _active_ span or to the _service entry_ span. +A *dynamic span tag* adds a tag value to an existing span. You can add a tag either to the _active_ span or to the _service entry_ span. Keep in mind that internal spans are not indexed by default and so might not be searchable in APM. -To create a span tag probe: +To create a dynamic span tag: -1. Select {{< ui >}}Span Tag{{< /ui >}} as the probe type. -1. Complete the [generic probe setup](#creating-a-probe) (choose service, environment, version, and probe location). +1. Select **Span Tag** as the instrumentation type. +1. Complete the [generic instrumentation setup](#creating-an-instrumentation) (choose service, environment, version, and location). 1. Specify a name for the tag. 1. Specify the value of the tag using the [Dynamic Instrumentation expression language][15]. 1. Optionally define a condition using the Dynamic Instrumentation expression language. The tag will only be added when the expression evaluates to true. 1. Optionally add additional tags, each with their own name, expression, and optional condition. - -{{< img src="dynamic_instrumentation/span_tag_probe.png" alt="Creating a Dynamic Instrumentation span tag probe" >}} - -You can use a *span tag probe* as an alternative to [using Custom Instrumentation to add tags in code][14]. +You can use a *dynamic span tag* as an alternative to [using Custom Instrumentation to add tags in code][14]. ## Further Reading @@ -228,11 +179,12 @@ You can use a *span tag probe* as an alternative to [using Custom Instrumentatio [6]: /getting_started/tagging/unified_service_tagging/ [7]: /integrations/guide/source-code-integration/ [8]: /account_management/rbac/permissions#apm -[9]: /logs/log_configuration/indexes/#exclusion-filters -[10]: /logs/log_configuration/indexes/#add-indexes [12]: https://app.datadoghq.com/dynamic-instrumentation [13]: /tracing/trace_collection/custom_instrumentation/java/#adding-spans [14]: /tracing/trace_collection/custom_instrumentation/java/#adding-tags [15]: /dynamic_instrumentation/expression-language -[16]: https://app.datadoghq.com/dynamic-instrumentation/setup +[16]: https://app.datadoghq.com/dynamic-instrumentation/settings [17]: /dynamic_instrumentation/symdb/ +[18]: /tracing/trace_collection/automatic_instrumentation/dd_libraries/java/ +[19]: /tracing/trace_collection/automatic_instrumentation/dd_libraries/python/ +[20]: /tracing/trace_collection/automatic_instrumentation/dd_libraries/dotnet-core diff --git a/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/_index.md b/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/_index.md index 11e1f6f995d..a0edbd210a7 100644 --- a/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/_index.md +++ b/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/_index.md @@ -1,6 +1,6 @@ --- title: Enabling Dynamic Instrumentation -description: Enable Dynamic Instrumentation for your applications to add probes and capture runtime data without code changes. +description: Enable Dynamic Instrumentation for your applications to add instrumentations without code changes. type: multi-code-lang aliases: - /dynamic_instrumentation/enabling/ @@ -14,6 +14,10 @@ further_reading: Dynamic Instrumentation is a feature of supporting Datadog SDKs. If you are already using [APM to collect traces][1] for your application, ensure your SDK is up-to-date and then enable Dynamic Instrumentation for your application. +For Java, Python, and .NET on recent SDK versions, Dynamic Instrumentation can be enabled in-app from the [Dynamic Instrumentation Settings page][2]. See [Enable Dynamic Instrumentation][3] for details and minimum SDK versions. + +The instructions below cover **manual enablement** through environment variables. Manual enablement is required for PHP, and for older SDK versions of Java, Python, and .NET. It is also available on supported SDK versions if you prefer to manage enablement through environment variables instead of in-app. + Select your runtime below to learn how to enable Dynamic Instrumentation for your application: {{< card-grid card_width="170px" >}} @@ -21,10 +25,7 @@ Select your runtime below to learn how to enable Dynamic Instrumentation for you {{< image-card href="/dynamic_instrumentation/enabling/python" src="integrations_logos/python.png" alt="Python" >}} {{< image-card href="/dynamic_instrumentation/enabling/dotnet" src="integrations_logos/dotnet-core.png" alt="Dotnet" >}} {{< image-card href="/dynamic_instrumentation/enabling/dotnet" src="integrations_logos/dotnet-framework.png" alt="Dotnet" >}} - {{< image-card href="/dynamic_instrumentation/enabling/nodejs" src="integrations_logos/nodejs.png" alt="Node.js" >}} - {{< image-card href="/dynamic_instrumentation/enabling/ruby" src="integrations_logos/ruby.png" alt="Ruby" >}} {{< image-card href="/dynamic_instrumentation/enabling/php" src="integrations_logos/php.png" alt="PHP" >}} - {{< image-card href="/dynamic_instrumentation/enabling/go" src="integrations_logos/go-metro.png" alt="Go" >}} {{< /card-grid >}} ## Further reading @@ -32,3 +33,5 @@ Select your runtime below to learn how to enable Dynamic Instrumentation for you {{< partial name="whats-next/whats-next.html" >}} [1]: /tracing/trace_collection/ +[2]: https://app.datadoghq.com/dynamic-instrumentation/settings +[3]: /dynamic_instrumentation/#enable-dynamic-instrumentation diff --git a/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/go.md b/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/go.md index 1b94459134a..e0f4e5c0ab3 100644 --- a/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/go.md +++ b/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/go.md @@ -1,10 +1,7 @@ --- title: Enable Dynamic Instrumentation for Go description: Set up Dynamic Instrumentation for Go applications to add probes and capture data without code changes. -private: false -code_lang: go -type: multi-code-lang -code_lang_weight: 70 +private: true aliases: - /dynamic_instrumentation/enabling/go further_reading: diff --git a/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/nodejs.md b/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/nodejs.md index 9321e586566..a6459e344e0 100644 --- a/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/nodejs.md +++ b/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/nodejs.md @@ -1,10 +1,7 @@ --- title: Enable Dynamic Instrumentation for Node.js description: Set up Dynamic Instrumentation for Node.js applications to add probes and capture data without code changes. -private: false -code_lang: nodejs -type: multi-code-lang -code_lang_weight: 40 +private: true aliases: - /dynamic_instrumentation/enabling/nodejs - /tracing/dynamic_instrumentation/enabling/nodejs diff --git a/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/ruby.md b/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/ruby.md index 02905048258..6b4a1af0cf6 100644 --- a/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/ruby.md +++ b/content/en/tracing/trace_collection/dynamic_instrumentation/enabling/ruby.md @@ -4,10 +4,7 @@ description: Set up Dynamic Instrumentation for Ruby applications to add probes aliases: - /dynamic_instrumentation/enabling/ruby/ - /tracing/dynamic_instrumentation/enabling/ruby -private: false -code_lang: ruby -type: multi-code-lang -code_lang_weight: 50 +private: true further_reading: - link: 'agent' tag: 'Documentation' @@ -93,7 +90,6 @@ Dynamic Instrumentation tracks code as it loads. For line probes to work correct ## Supported features -- [Dynamic Logs][8] (log probes) - Line probes (capture variables at a specific line) - Method probes (capture method entry and exit) - Probe conditions using [Expression Language][11] diff --git a/content/en/tracing/trace_collection/dynamic_instrumentation/expression-language.md b/content/en/tracing/trace_collection/dynamic_instrumentation/expression-language.md index a6be86320fd..a2aa6623ffb 100644 --- a/content/en/tracing/trace_collection/dynamic_instrumentation/expression-language.md +++ b/content/en/tracing/trace_collection/dynamic_instrumentation/expression-language.md @@ -8,7 +8,7 @@ aliases: ## Overview -The Dynamic Instrumentation Expression Language helps you formulate log probe message templates, metric probe expressions, span tag values, and probe conditions. It borrows syntax elements from common programming languages, but also has its own unique rules. The language lets you access local variables, method parameters, and nested fields within objects, and it supports the use of comparison and logical operators. +The Dynamic Instrumentation Expression Language helps you formulate metric instrumentation expressions, span tag values, and instrumentation conditions. It borrows syntax elements from common programming languages, but also has its own unique rules. The language lets you access local variables, method parameters, and nested fields within objects, and it supports the use of comparison and logical operators. Examples: - `someVar.someField` @@ -16,8 +16,6 @@ Examples: - `any(post.tags, {@it == "debugger"})` - `@duration > 10 && len(p.data) < 100` -{{< img src="tracing/dynamic_instrumentation/expressions.png" alt="Example log probe with expressions" style="width:100%;" >}} - Generally, the Expression Language supports: * Accessing local variables, method parameters, and deeply nested fields and attributes within objects. * Using comparison operators (`<`, `>`, `>=`, `<=`, `==`, `!=`, `instanceof`) to compare variables, fields, and constants in your conditions, for example: `localVar1.field1.field2 != 15`. @@ -32,13 +30,13 @@ Try [autocomplete and search (in Preview)][6] for an improved user experience us ## Applications -Expressions can be used to produce metrics or logs, and as conditions to emit filtered data. +Expressions can be used to produce metrics and as conditions to emit filtered data. For example, you can create a histogram from the length of a string using `len(data)` as the metric expression. Metric expressions must evaluate to a number. -Logs can be emitted using templates. In log templates and tag values, expressions are delimited from the static parts of the template with brackets, for example: `User name is {user.name}`. Log template expressions can evaluate to any value. +In span tag values, expressions are delimited from the static parts of the template with brackets, for example: `User name is {user.name}`. Tag value expressions can evaluate to any value. -Probe conditions must evaluate to a Boolean, for example: +Instrumentation conditions must evaluate to a Boolean, for example: - `startsWith(user.name, "abc")` - `len(str) > 20` - `a == b` @@ -88,7 +86,7 @@ The following examples assume a variable named `mySequence` with value `[1,2,3,4 ## Try your own conditions -This interactive simulator helps you experiment with the Expression Language syntax in a realistic environment. It shows how conditions affect whether a log line will be generated when instrumenting a method. +This interactive simulator helps you experiment with the Expression Language syntax in a realistic environment. It shows how conditions affect whether data is captured when instrumenting a method. Select one of the examples or enter an expression in the "when" field and click "SIMULATE" to see if the log would be generated based on your condition. diff --git a/content/en/tracing/trace_collection/dynamic_instrumentation/sensitive-data-scrubbing.md b/content/en/tracing/trace_collection/dynamic_instrumentation/sensitive-data-scrubbing.md index 88c4f9acd1d..2e068381afd 100644 --- a/content/en/tracing/trace_collection/dynamic_instrumentation/sensitive-data-scrubbing.md +++ b/content/en/tracing/trace_collection/dynamic_instrumentation/sensitive-data-scrubbing.md @@ -15,7 +15,7 @@ further_reading: ## Overview -Datadog Dynamic Instrumentation enhances the observability and debugging capabilities of your applications by capturing variable data at arbitrary code locations in production environments. It also can craft and evaluate expressions in real-time, and integrate their outputs into log messages or add them as span tags. +Datadog Dynamic Instrumentation enhances the observability and debugging capabilities of your applications by capturing variable data at arbitrary code locations in production environments. It also can craft and evaluate expressions in real-time, and integrate their outputs as span tags. While this functionality is powerful, it also presents the possibility of sensitive data leaks, both intentional and unintentional. Alongside the product's robust data capture capabilities, it also provides comprehensive measures to safeguard sensitive information. diff --git a/content/en/tracing/trace_collection/dynamic_instrumentation/symdb/_index.md b/content/en/tracing/trace_collection/dynamic_instrumentation/symdb/_index.md index 73128314f4b..616a3bd2c9e 100644 --- a/content/en/tracing/trace_collection/dynamic_instrumentation/symdb/_index.md +++ b/content/en/tracing/trace_collection/dynamic_instrumentation/symdb/_index.md @@ -47,16 +47,9 @@ Select your runtime below: ## Explore autocomplete and search -With autocomplete and search, the user experience of Dynamic Instrumentation is improved to behave more like an IDE. +Autocomplete and search make Dynamic Instrumentation behave more like an IDE by supporting class and method search, code display, and expression autocomplete. -Dynamic Instrumentation provides search for class and method names: -{{< img src="dynamic_instrumentation/symdb_method_search.png" alt="Search for methods when creating a Dynamic Instrumentation log probe" style="width:60%;" >}} - -When you select a method in the Dynamic Instrumentation configuration, the code for that method is displayed: -{{< img src="dynamic_instrumentation/symdb_method_highlight.png" alt="Autocomplete and search highlights the selected method" >}} - -Dynamic Instrumentation also provides autocomplete for log templates and other templates that use the [Dynamic Instrumentation expression language][5]: -{{< img src="dynamic_instrumentation/symdb_completion.png" alt="Autocomplete suggestions for log templates" style="width:80%;" >}} +Use class and method search to find where to add instrumentation. When you select a method in the Dynamic Instrumentation configuration, Datadog displays the code for that method. Dynamic Instrumentation also provides autocomplete for expression templates that use the [Dynamic Instrumentation expression language][5]. ## Further Reading diff --git a/static/images/dynamic_instrumentation/log_probe.png b/static/images/dynamic_instrumentation/log_probe.png deleted file mode 100644 index b6c60859056..00000000000 Binary files a/static/images/dynamic_instrumentation/log_probe.png and /dev/null differ