Add support for managing OpenTelemetry attributes

[stevehipwell]

Is your feature request related to a problem? Please describe.

I'd like to be able to export logs to Grafana via the OpenTelemetry format and this requires the ability to set OpenTelemetry attributes.

Describe the solution you'd like

I'd like the ability to both modify attributes as a filter and as part of the OpenTelemetry output.

Describe alternatives you've considered

I can use the Loki output but this isn't a documented log ingestion pattern in Grafana Cloud so support might be removed at any point.

Additional context

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[stevehipwell]

@patrick-stephens this is the issue for grafana/loki#4648 (comment).

[r0mdau]

If you want logRecords.attributes[] you need to use fluentbit metadata and then, in the opentelemetry output you need to use logs_attributes_metadata_key: $key example:

pipeline:
  inputs:
    # - name: tail
    #   path: /log.json
    #   parser: json
    - name: dummy
      dummy: '{"message": "Hello World","verb": "GET"}'
      metadata: '{"meta": {"attribute1":"developper","attribute2":"networking"}}'

...

  outputs:

    - name: opentelemetry
...
      logs_attributes_metadata_key: meta

But if you need resource.attributes[] I need it too and we could work on a feature to fetch them from the metadata like

logs_resource_attributes_metadata_key: $key

[r0mdau]

I finally succeeded to have all resource and log attributes, configuration example:

service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: dummy
      dummy: '{"resource":{"attributes":{"service.name":"devialet","online.project.key":"france"}},"message": "Hello World!","level": "info"}'
      metadata: '{"otlp":{"attributes":{"attr1":"value1"}}}'

  outputs:
    - name : stdout
      match: '*'

    - name: opentelemetry
      match: '*'
      host: localhost
      port: 3100
      logs_uri: /otlp/v1/logs
      logs_body_key: body
      logs_severity_text_message_key: level

So in the in_opentelemetry plugin, the logRecords.attributes[] are stored in the fluentbit metadata, under $otlp['attributes'].

The out_opentelemetry plugin can extract resource.attributes[] from the log record and so logRecords.attributes[] from the metadata.

It is not consistent.

I haven't figured out why precisely the opentelemetry_envelope processor is not needed.

[williamdenton]

i found there is an incomplete implementation for logs_resource_metadata_key which was added in #8475 as noted by @r0mdau.

oddly it was claude that suggested i use this property even though its undocumented. Are we aware of any work arounds for getting the open telemetery exporter to attach resource labels?