Skip to content

Support log group/stream creation in new cwlogsauth extension#2084

Draft
jj22ee wants to merge 2 commits intoaws:mainfrom
jj22ee:otlp-exporter-auto-loggroupcreation-v2
Draft

Support log group/stream creation in new cwlogsauth extension#2084
jj22ee wants to merge 2 commits intoaws:mainfrom
jj22ee:otlp-exporter-auto-loggroupcreation-v2

Conversation

@jj22ee
Copy link
Copy Markdown
Contributor

@jj22ee jj22ee commented Apr 14, 2026

Description of the issue

When using the otlphttp exporter to send logs to the CloudWatch OTLP Logs endpoint (https://logs.<region>.amazonaws.com/v1/logs), the endpoint requires the target log group and log stream to already exist — unlike the native PutLogEvents API used by awscloudwatchlogsexporter, which auto-creates them on first write.

This means customers using the OTLP logs pipeline must manually pre-create log groups and streams before CWAgent can export logs, or the endpoint rejects the requests silently.

Description of changes

Adds a new cwlogsauth extension that lazily auto-creates CloudWatch log groups and log streams when CWAgent sends logs via the otlphttp exporter to the CW OTLP Logs endpoint.

The extension implements extensionauth.HTTPClient and wraps an inner auth extension (typically sigv4auth). It intercepts outgoing HTTP requests as a RoundTripper and:

  1. Reads x-aws-log-group and x-aws-log-stream from the outgoing request headers
  2. On first request per unique (group, stream) pair, calls CreateLogGroup then CreateLogStream via the AWS SDK (both idempotent — ResourceAlreadyExistsException is ignored)
  3. Caches successful creations in a sync.Map so subsequent requests have zero overhead
  4. On creation failure, logs a warning and forwards the request anyway (non-blocking). Does not cache failures, so the next request retries creation.

This mirrors the lazy creation behavior of the native cwlogs.Client.CreateStream used by awscloudwatchlogsexporter, but for the otlphttp exporter path.

Files added:

  • extension/cwlogsauth/config.go — Config struct with optional auth field for chaining with an inner auth extension
  • extension/cwlogsauth/factory.go — Extension factory registration
  • extension/cwlogsauth/extension.go — RoundTripper implementation with lazy log group/stream creation and sync.Map cache

Files modified:

  • service/defaultcomponents/components.go — Registered cwlogsauth extension factory

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Deployed application in EKS, using CW Observability EKS Add-on, patched Agent to include my changes, and used the following configuration. My log group&stream (/test/telemetry & test-otlp) were automatically created once I wrote logs to the otlp/http_0_0_0_0_4316 logs receiver.

    agent = {
      config = {
        logs = {
          metrics_collected = {
            application_signals = {}
            kubernetes = {
              enhanced_container_insights = true
            }
          }
        }
        traces = {
          traces_collected = {
            application_signals = {}
          }
        }
      }
      otelConfig = {
        processors = {
          "batch/test" = {}
        }
        extensions = {
          "sigv4auth/logs" = {
            region  = var.region
            service = "logs"
          }
          "cwlogsauth" = {              // <-----------------------------------------—
            auth = "sigv4auth/logs"
          }
        }
        exporters = {
          "otlphttp/cw-test-logs" = {
            logs_endpoint = "https://logs.${var.region}.amazonaws.com/v1/logs"
            auth = {
              authenticator = "cwlogsauth"           // <-----------------------------------------—
            }
            headers = {
              "x-aws-log-group"  = "/test/telemetry"
              "x-aws-log-stream" = "test-otlp"
            }
          }
        }
        service = {
          extensions = ["sigv4auth/logs", "cwlogsauth"]         // <-----------------------------------------—
          pipelines = {
            "logs/test" = {
              receivers  = ["otlp/http_0_0_0_0_4316"]
              processors = ["batch/test"]
              exporters  = ["otlphttp/cw-test-logs"]
            }
          }
        }
      }
    }

Requirements

Before commiting your code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

@jj22ee jj22ee requested a review from a team as a code owner April 14, 2026 00:29
@jj22ee jj22ee changed the title Support log group/stream creation in new cwlogsprovision extension Support log group/stream creation in new cwlogsauth extension Apr 14, 2026
@jj22ee jj22ee marked this pull request as draft April 14, 2026 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant