Skip to content

feat: automatic log group/stream creation at startup when using otlphttp exporter for logs#2083

Draft
jj22ee wants to merge 1 commit intoaws:mainfrom
jj22ee:otlp-exporter-auto-loggroupcreation-v1
Draft

feat: automatic log group/stream creation at startup when using otlphttp exporter for logs#2083
jj22ee wants to merge 1 commit intoaws:mainfrom
jj22ee:otlp-exporter-auto-loggroupcreation-v1

Conversation

@jj22ee
Copy link
Copy Markdown
Contributor

@jj22ee jj22ee commented Apr 13, 2026

Description of the issue

The CloudWatch OTLP Logs endpoint (https://logs.<region>.amazonaws.com/v1/logs) requires the target log group and log stream to already exist before accepting log data. Unlike the native PutLogEvents API used by awscloudwatchlogsexporter — which auto-creates log groups and streams on first write — the otlphttp exporter sends standard OTLP HTTP requests with no built-in CW resource provisioning.

This forces customers to manually pre-create log groups and streams) before CWAgent can export logs via the OTLP path, adding friction to the setup experience.

Description of changes

Adds a confmap.Converter that runs during CWAgent's config resolution phase — after the JSON config is translated to OTel YAML but before the OTel pipeline components (receivers, processors, exporters) are started.

How it works:

  1. Scans the merged OTel config for all otlphttp and otlphttp/* exporters
  2. Filters to only those used in logs/* pipelines (via service.pipelines cross-reference)
  3. Checks if their logs_endpoint or endpoint matches the CW Logs OTLP pattern (https://logs.<region>.amazonaws.com)
  4. Extracts x-aws-log-group and x-aws-log-stream from the exporter's headers config
  5. Calls CreateLogGroup then CreateLogStream via the AWS SDK (both idempotent — ResourceAlreadyExistsException is silently ignored)

Files added:

  • service/configprovider/otlphttp_log_provisioner.goconfmap.Converter implementation with config scanning, CW endpoint detection, and log group/stream creation
  • service/configprovider/otlphttp_log_provisioner_test.go — Unit tests covering: single exporter, exporter not in logs pipeline, non-CW endpoint ignored, missing log group header, default log stream fallback, multiple exporters, empty config, and region extraction

Files modified:

  • service/configprovider/provider.go — Registered NewOTLPHTTPLogProvisionerFactory() in the converter chain after otlphttpValidator

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Unit tests:

  • TestExtractRegionFromLogsEndpoint — validates region extraction from CW Logs OTLP endpoints (us-east-1, eu-west-1, non-logs endpoint, empty, non-AWS endpoint)
  • TestFindLogTargets — validates config scanning logic for when to create log group/stream and when not to based on the configuration.

E2E testing:

  • Deployed on EKS with CW Observability add-on, and patched CWAgent with my changes
  • Configured otelConfig with otlphttp exporter targeting CW Logs OTLP endpoint with x-aws-log-group and x-aws-log-stream headers
  • Verified log group and stream were auto-created at CWAgent startup, through AWS CloudWatch console

Requirements

Before commiting your code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

@jj22ee jj22ee requested a review from a team as a code owner April 13, 2026 23:52
@jj22ee jj22ee marked this pull request as draft April 14, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant