diff --git a/hugo/content/en/disaster_recovery/_index.md b/hugo/content/en/disaster_recovery/_index.md index f69a1a145c8..b4a1e4e2dce 100644 --- a/hugo/content/en/disaster_recovery/_index.md +++ b/hugo/content/en/disaster_recovery/_index.md @@ -4,25 +4,20 @@ aliases: - /agent/guide/datadog-disaster-recovery/ site_support_id: datadog_disaster_recovery further_reading: -- link: "agent/remote_config/?tab=configurationyamlfile" - tag: "Documentation" - text: "Remote Configuration" -- link: "/getting_started/site/" - tag: "Documentation" - text: "Getting Started with Datadog Sites" - link: "https://www.datadoghq.com/blog/ddr-mitigates-cloud-provider-outages/" tag: "Blog" text: "Datadog Disaster Recovery mitigates cloud provider outages" --- -## Overview +Datadog Disaster Recovery (DDR) keeps your observability running when a cloud provider region or the Datadog services within it are disrupted. With DDR, you configure a secondary Datadog organization in a different region ahead of time and replicate your resources to it. When you fail over, the secondary site already has the dashboards, monitors, and users your team needs. -Datadog Disaster Recovery (DDR) provides you with observability continuity during events that may impact a cloud service provider region or Datadog services running within a cloud provider region. Using DDR, you can recover live observability at an alternate, functional Datadog site, enabling you to meet your critical observability availability goals. +DDR uses an active-passive model: your secondary site stays in sync but passive until you decide to fail over to it. Failover is never automatic; you choose when to cut over. -DDR also allows you to periodically conduct disaster recovery drills to not only test your ability to recover from outage events, but to also meet your business and regulatory compliance needs. +DDR also lets you run periodic disaster recovery drills to test your ability to recover from outages and to meet your business and regulatory compliance needs. ## Prerequisites -The minimum version of the Datadog Agent you need depends on the types of telemetry you need to use: + +The minimum Datadog Agent version you need depends on which products you use: |Supported telemetry |Supported products |Agent version required | |--------------------|----------------------------|-----------------------| @@ -30,33 +25,26 @@ The minimum version of the Datadog Agent you need depends on the types of teleme |Metrics |Infrastructure Monitoring | v7.54+ | |Traces |APM | v7.68+ | - -
Datadog is continuously evaluating customer requests to support DDR for additional products. Contact the Disaster Recovery team to learn about upcoming capabilities and your specific needs if they are not covered above.
-
## Setup -To enable Datadog Disaster Recovery, follow these steps. If you have any questions about any of the steps, contact your [Customer Success Manager][14] or [Datadog Support][15]. +Follow these steps to enable Datadog Disaster Recovery. If you have questions about any of the steps, contact your [Customer Success Manager][14] or [Datadog Support][15]. ### 1. Create a DDR org and link it to your primary org {{% collapse-content title="Create and share your DDR org" level="h4" %}} -
If required, Datadog can set this up for you.
+
If you prefer, Datadog can set this up for you.
#### Create your DDR org -1. Go to [Get Started with Datadog][16]. You may need to log out of your current session, or use incognito mode to access this page. -2. Choose a different Datadog site than your primary (for example, if you're on `US1`, choose `EU` or `US5`). +1. Go to [Get Started with Datadog][16]. You may need to log out of your current session, or use incognito mode, to access this page. +2. Choose a different Datadog site than your primary (for example, if you're on `US1`, choose `EU` or `US5`). See the [Datadog site list][17] for options. All Datadog sites are geographically separated. 3. Follow the prompts to create an account. -All Datadog sites are geographically separated. Reference the [Datadog Site List][17] for options. - -If you are also sending telemetry to Datadog using cloud provider integrations, you must add your cloud provider accounts in the DDR org. Datadog does not use cloud providers to receive telemetry data while the DDR site is passive (not in failover). - #### Share the DDR org information with Datadog Email your new org name to your [Customer Success Manager][14]. Then, your Customer Success Manager sets this new org as your DDR org. @@ -65,31 +53,30 @@ Email your new org name to your [Customer Success Manager][14]. Then, your Custo {{% collapse-content title="Retrieve the public IDs and link your DDR and primary orgs" level="h4" %}} -For security reasons, Datadog is unable to link the orgs on your behalf. +
For security reasons, Datadog is unable to link the orgs on your behalf.
-After the Datadog team has set your DDR org, use the Datadog [public API endpoint][1] to retrieve the public IDs of the primary and DDR org. +After Datadog designates your DDR organization, link your DDR org to your primary org: -To link your DDR org to your primary org: +1. Use the [List your managed organizations][1] endpoint to retrieve the public IDs of your primary and DDR organizations. +1. Add the `disaster_recovery_status_write` scope to your application key in the primary org. +1. Run the following commands, replacing the placeholders with the appropriate values. -- Add the `disaster_recovery_status_write` scope to your application key in the primary org. -- Run the following commands, replacing the placeholders with the appropriate values. + ```shell + export PRIMARY_DD_API_KEY= + export PRIMARY_DD_APP_KEY= + export PRIMARY_DD_API_URL= -```shell -export PRIMARY_DD_API_KEY= -export PRIMARY_DD_APP_KEY= -export PRIMARY_DD_API_URL= - -export DDR_ORG_ID= -export PRIMARY_ORG_ID= -export USER_EMAIL= -export CONNECTION='{"data":{"id":"'${PRIMARY_ORG_ID}'","type":"hamr_org_connections","attributes":{"TargetOrgUuid":"'${DDR_ORG_ID}'","HamrStatus":1,"ModifiedBy":"'${USER_EMAIL}'", "IsPrimary":true}}}' - -curl -v -H "Content-Type: application/json" -H \ -"dd-api-key:${PRIMARY_DD_API_KEY}" -H \ -"dd-application-key:${PRIMARY_DD_APP_KEY}" --data "${CONNECTION}" --request POST ${PRIMARY_DD_API_URL}/api/v2/hamr -``` + export DDR_ORG_ID= + export PRIMARY_ORG_ID= + export USER_EMAIL= + export CONNECTION='{"data":{"id":"'${PRIMARY_ORG_ID}'","type":"hamr_org_connections","attributes":{"TargetOrgUuid":"'${DDR_ORG_ID}'","HamrStatus":1,"ModifiedBy":"'${USER_EMAIL}'", "IsPrimary":true}}}' -After linking your orgs, only the failover org displays this banner: + curl -v -H "Content-Type: application/json" -H \ + "dd-api-key:${PRIMARY_DD_API_KEY}" -H \ + "dd-application-key:${PRIMARY_DD_APP_KEY}" --data "${CONNECTION}" --request POST ${PRIMARY_DD_API_URL}/api/v2/hamr + ``` + +After you link your orgs, the failover org displays this banner: {{< img src="agent/guide/ddr/ddr-banner.png" alt="The DDR banner in the DDR org" >}} @@ -101,7 +88,7 @@ After linking your orgs, only the failover org displays this banner: **Datadog recommends using Single Sign On (SSO)** to enable all your users to log in to your Disaster Recovery org during an outage. -Go to the [Organization Settings][2] in your DDR org to configure [SAML][3] or {{< ui >}}Google Login{{< /ui >}} for your users. +Go to [Organization Settings][2] in your DDR org to configure [SAML][3] or {{< ui >}}Google Login{{< /ui >}} for your users. Managed sync replicates user accounts from your primary org to your DDR org. Datadog recommends configuring [Just-in-Time provisioning with SAML][4] so users can access the DDR org during a failover without needing to reset their password. @@ -111,7 +98,7 @@ Managed sync replicates user accounts from your primary org to your DDR org. Dat See the [AWS][5], [Azure][6], and [Google Cloud][7] integrations for setup steps. -Your cloud integrations must be configured in both primary and DDR orgs, but they run in only one org at a time: by default in the primary org, and in the DDR org during failover. +Your cloud integrations must be configured in both primary and DDR orgs, but they only run in one org at a time: by default in the primary org, and during failover in the DDR org. For more information, see the [Cloud integrations failover](#id-for-cloud) section. @@ -129,7 +116,7 @@ Managed sync replicates resources from your primary org to your DDR org on a reg {{% /collapse-content %}} -{{% collapse-content title="Enable Remote Configuration [**RECOMMENDED]" level="h4" %}} +{{% collapse-content title="Enable Remote Configuration (RECOMMENDED)" level="h4" %}} [Remote Configuration (RC)][11] allows you to remotely configure and change the behavior of Datadog Agents deployed in your infrastructure. @@ -141,7 +128,6 @@ Datadog strongly recommends using Remote Configuration for better failover contr {{% collapse-content title="Dual ship telemetry to DDR org during failover or drills" level="h4" %}} - To enable Dual Shipping, Datadog recommends using [Fleet Automation][12] for management at scale. Alternatively, you can configure it manually by editing your `datadog.yaml` file. Contact your Datadog Customer Success Manager to schedule dedicated time windows for failover testing to measure performance and Recovery Time Objective (RTO). @@ -149,13 +135,13 @@ Contact your Datadog Customer Success Manager to schedule dedicated time windows {{< tabs >}} {{% tab "Using Fleet Automation (recommended)" %}} -From the [Fleet Automation][100] page in your failover org, on the {{< ui >}}Configure Agents{{< /ui >}} tab, you can create a failover policy or reuse an existing one, and apply it to your fleet of Agents. Soon after the policy is enabled, Agents begin dual-shipping telemetry to both the primary and DDR (failover) observability sites. +Go to [Fleet Automation][100] > {{< ui >}}Configure Agents{{< /ui >}} in your DDR org to create a failover policy or reuse an existing one, and apply it to your fleet of Agents. Soon after the policy is enabled, Agents begin dual-shipping telemetry to both the primary and DDR (failover) observability sites. -To create a failover policy, click on {{< ui >}}Create Failover Policy{{< /ui >}}. +To create a failover policy, click {{< ui >}}Create Failover Policy{{< /ui >}}. {{< img src="/agent/guide/ddr/ddr-fa-policy.png" alt="Manage DDR policies" style="width:80%;" >}} -Then, follow the prompt to scope the hosts and telemetry (metrics, logs, traces) that you are required to failover. +Then, follow the prompt to scope the hosts and telemetry (metrics, logs, traces) that you are required to fail over. {{< img src="/agent/guide/ddr/ddr-fa-policy-scope.png" alt="Scope the hosts and telemetry required to failover" style="width:80%;" >}} @@ -167,7 +153,7 @@ Then, follow the prompt to scope the hosts and telemetry (metrics, logs, traces) {{% tab "Manually" %}} -During a failover or failover exercises, update your Datadog Agent's `datadog.yaml` configuration file as shown in the example below and restart the Agent. +During a failover or failover exercises, update your Datadog Agent's `datadog.yaml` configuration file as shown in the following example, and restart the Agent. - `enabled: true` allows the Agent to send {{< tooltip text="metadata" tooltip="Data about the Agent and the infrastructure host. For example, `host name`, `host tags`, `Agent version`." >}} to the DDR Datadog site so you can view Agents and your Infra hosts in the DDR org. This allows you to see your Agents and infrastructure hosts in the failover org. @@ -179,7 +165,7 @@ multi_region_failover: failover_metrics: false failover_logs: false failover_apm: false - site: # For example "site: us5.datadoghq.com" for a US5 site + site: # For example, "site: us5.datadoghq.com" for a US5 site api_key: ``` @@ -196,11 +182,11 @@ DNS-based failover is a complementary approach to Agent-based failover. Instead #### Receive your custom DNS endpoint -If you choose to use DNS-based failover, Datadog provisions a custom intake URL for your organization (for example, `.intake.datadoghq.com`). Configure all your data sources (Agents, log shippers, and custom instrumentation) to send telemetry to this endpoint instead of the default Datadog intake URL. This is a one-time configuration change. +If you choose to use DNS-based failover, Datadog provisions a custom intake URL for your organization (for example, `.intake.datadoghq.com`). Configure all your data sources (such as Agents, log shippers, and custom instrumentation) to send telemetry to this endpoint instead of to the default Datadog intake URL. This is a one-time configuration change. #### Trigger a DNS failover -To initiate a DNS failover, contact Datadog through your [Customer Success Manager][14] or [Datadog Support][15]. Datadog updates the DNS record to redirect traffic from your primary site to your DDR site. The target Recovery Time Objective (RTO) from the time failover is initiated is 2 hours. +Contact your [Customer Success Manager][14] or [Datadog Support][15] to initiate a DNS failover. Datadog updates the DNS record to redirect traffic from your primary site to your DDR site. The target Recovery Time Objective (RTO) is 2 hours from the time failover is initiated.
A customer-controlled way to trigger DNS failover directly from the DDR org is in Preview. Contact your Customer Success Manager to learn more.
@@ -210,7 +196,7 @@ To initiate a DNS failover, contact Datadog through your [Customer Success Manag {{% collapse-content title="Activate and test DDR failover in Agent-based environments" level="h4" %}} -To trigger a failover of your Agents, you can click on one of the policies in [Fleet Automation][13] in your DDR org, and then click {{< ui >}}Enable{{< /ui >}}. The status of each host updates as the failover occurs. +To trigger a failover of your Agents, click one of the policies in [Fleet Automation][13] in your DDR org, then click {{< ui >}}Enable{{< /ui >}}. The status of each host updates as the failover occurs. {{< img src="/agent/guide/ddr/ddr-fa-policy-enable3.png" alt="Enable the failover policy in the DDR org" style="width:80%;" >}} @@ -299,7 +285,7 @@ During testing, integration telemetry is spread over both organizations. If you [1]: /api/latest/organizations/#list-your-managed-organizations [2]: https://app.datadoghq.com/organization-settings/users -[3]: /account_management/saml/#overview +[3]: /account_management/saml/ [4]: /account_management/saml/#just-in-time-jit-provisioning [5]: /integrations/amazon-web-services/ [6]: /integrations/azure/