Skip to content

ECI-2165: add CUSTOM_DD_SITE for custom intake endpoints - #185

Merged
yuhuyoyo merged 7 commits into
masterfrom
yuhu/custom-dd-site-intake
Sep 8, 2026
Merged

ECI-2165: add CUSTOM_DD_SITE for custom intake endpoints#185
yuhuyoyo merged 7 commits into
masterfrom
yuhu/custom-dd-site-intake

Conversation

@yuhuyoyo

@yuhuyoyo yuhuyoyo commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a CUSTOM_DD_SITE environment variable that lets the OCI forwarders send to custom intake endpoints without provisioning a per-customer TLS certificate.

Introduces client.IntakeURL(prefix, path, site) to centralize intake URL construction across the events, logs, and metrics forwarders (previously each handler inlined its own fmt.Sprintf).

Why

The certificate covering custom intake endpoints has its SAN at a wildcard level (e.g. *.mrf.datadoghq.com). To stay under that single cert for any customer, the host must use dashes instead of dots to separate the prefix labels — e.g. http-intake-logs-customerA.mrf.datadoghq.com rather than http-intake.logs.customerA.mrf.datadoghq.com. This avoids creating a certificate per customer.

How it works

IntakeURL builds the intake URL for a given host prefix and API path:

  • CUSTOM_DD_SITE unset (default, unchanged behavior):

    https://{prefix}.{DD_SITE}{path}
    

    e.g. https://http-intake.logs.datadoghq.com/api/v2/logs

  • CUSTOM_DD_SITE set — provides the full host base following the (dot→dash) prefix:

    https://{prefix-with-dashes}-{CUSTOM_DD_SITE}{path}
    

    e.g. with CUSTOM_DD_SITE=customerA.mrf.datadoghq.com:

    • https://cloudplatform-intake-customerA.mrf.datadoghq.com/api/v2/cloudchanges
    • https://http-intake-logs-customerA.mrf.datadoghq.com/api/v2/logs
    • https://ocimetrics-intake-customerA.mrf.datadoghq.com/api/v2/ocimetrics

The base domain is not hardcodedCUSTOM_DD_SITE carries the full base, so it can be repointed (e.g. off mrf.datadoghq.com) with no code change.

Changes

  • lib/client/client.go — new IntakeURL helper.
  • events-forwarder/internal/handler/handler.go — use IntakeURL.
  • logs-forwarder/internal/handler/handler.go — use IntakeURL.
  • metrics-forwarder/internal/handler/handler.go — use IntakeURL.
  • lib/client/client_test.goTestIntakeURL covering default and custom-site forms.

Verification

  • go build ./... passes for lib, events-forwarder, logs-forwarder, metrics-forwarder.
  • All existing tests pass; new TestIntakeURL passes (5 subtests).
  • Backfill bucket routing (backfillBucketName) is unaffected — it matches on path substrings (cloudchanges/logs/ocimetrics), which are preserved.

Out of scope / follow-ups

  • Terraform wiring to pass CUSTOM_DD_SITE into the function config (datadog-terraform-onboarding / datadog-integration locals.tf) is not included in this PR. The forwarders read the env var directly, so it can be set on the function config out-of-band until the Terraform variable is added.

🤖 Generated with Claude Code

@yuhuyoyo yuhuyoyo changed the title feat(datadog-functions): add CUSTOM_DD_SITE for custom intake endpoints ECI-2165: add CUSTOM_DD_SITE for custom intake endpoints Sep 4, 2026
@yuhuyoyo
yuhuyoyo marked this pull request as ready for review September 4, 2026 19:32
@yuhuyoyo
yuhuyoyo requested a review from a team as a code owner September 4, 2026 19:32
@yuhuyoyo
yuhuyoyo requested a review from rheei September 4, 2026 19:32
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Version bumped: 1.2.41.2.5 (patch auto-bump)

Override if needed:

  • Breaking change → manually set VERSION and datadog-terraform-onboarding/VERSION to bump the major digit.
  • New feature → manually bump the minor digit.
  • Minor bug fix → no change needed; the patch bump above is correct.

Comment thread datadog-integration/schema.yaml
yuhuyoyo and others added 7 commits September 8, 2026 14:51
Introduce client.IntakeURL to centralize intake URL construction across
the events, logs, and metrics forwarders. When CUSTOM_DD_SITE is set, it
provides the full host base following the (dot→dash) prefix, keeping the
whole host under a single wildcard certificate (e.g. *.mrf.datadoghq.com)
so no per-customer certificate is needed. When unset, behavior is
unchanged (standard {prefix}.{DD_SITE}{path} form).

Co-Authored-By: Claude Code <noreply@anthropic.com>
Add a custom_datadog_site variable (default "") to both the
datadog-terraform-onboarding and datadog-integration roots, threading
it through to the regional-stacks function config as CUSTOM_DD_SITE. In
the integration root the value flows through the
create_apply_regional_stack.sh shell arg and Resource Manager variables
JSON. Also surface it as an advanced option in the ORM schema.yaml so it
only appears when "Use advanced options" is enabled. Empty by default, so
existing deployments are unaffected.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Address code-review findings on the CUSTOM_DD_SITE change:

- backfillBucketName matched strings.Contains on the full URL, so a
  CUSTOM_DD_SITE whose host contained "ocimetrics"/"cloudchanges"/"logs"
  could misroute 5xx payloads to the wrong backfill bucket. Match on the
  API path (/api/v2/...) instead, which is stable and host-independent.
- IntakeURL now sanitizes CUSTOM_DD_SITE: trims whitespace and strips any
  pasted "https://"/"http://" scheme and trailing slashes so a host base
  misconfiguration produces a well-formed URL rather than a malformed one.
- Align the custom_datadog_site variable description across all four
  variables.tf files (the two module-level copies were missing the
  wildcard-certificate rationale).

Add TestBackfillBucketName and extend TestIntakeURL with sanitization
cases.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…can be cleared

The OCI provider suppresses empty-string values in the functions application
config map from its diff, so a key present with "" cannot clear a
previously-set value on re-apply. Make CUSTOM_DD_SITE conditionally included
via merge() only when var.custom_datadog_site != "" so unsetting it
produces a real structural diff (key removed) that the provider honors.

Verified end-to-end: re-applying without custom_datadog_site now clears
CUSTOM_DD_SITE from dd-function-app across all subscribed regions.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@yuhuyoyo
yuhuyoyo force-pushed the yuhu/custom-dd-site-intake branch from 43a0623 to 335f416 Compare September 8, 2026 18:53
@yuhuyoyo
yuhuyoyo merged commit 20b3705 into master Sep 8, 2026
18 checks passed
@yuhuyoyo
yuhuyoyo deleted the yuhu/custom-dd-site-intake branch September 8, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants