Skip to content

Delta exponential histogram never goes back to the maximum scale after a wide-range interval #5716

Description

@David-Wu1119

Describe your environment

OS: macOS (arm64)
Python version: 3.12
SDK version: main at 18d7e24 (opentelemetry-sdk installed from source)
API version: main at 18d7e24

What happened?

With DELTA temporality, the exponential histogram aggregation lowers its scale when an interval's values span a wide range, but never raises it again. Collection resets the buckets but not self._mapping (_ExponentialBucketHistogramAggregation.collect, the DELTA→DELTA branch in opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/aggregation.py). So every later delta point keeps the lowest scale any earlier interval needed, even when it holds a single value.

The SDK spec says: "When the histogram contains not more than one value in either of the positive or negative ranges, the implementation SHOULD use the maximum scale." A delta point covers only its own interval.

Steps to Reproduce

from opentelemetry.sdk.metrics import Histogram, MeterProvider
from opentelemetry.sdk.metrics.export import AggregationTemporality, InMemoryMetricReader
from opentelemetry.sdk.metrics.view import ExponentialBucketHistogramAggregation, View

reader = InMemoryMetricReader(preferred_temporality={Histogram: AggregationTemporality.DELTA})
provider = MeterProvider(
    metric_readers=[reader],
    views=[View(instrument_name="latency", aggregation=ExponentialBucketHistogramAggregation())],
)
hist = provider.get_meter("demo").create_histogram("latency")


def point():
    return reader.get_metrics_data().resource_metrics[0].scope_metrics[0].metrics[0].data.data_points[0]


hist.record(1e-6)
hist.record(1e6)
print(point().scale)  # interval 1: 2

for _ in range(3):
    hist.record(5.0)
    print(point().scale)  # each later interval holds one value: 2, 2, 2

Expected Result

Each interval after the first holds a single value, so it's exported at the maximum scale (20 by default), with positive.offset = 2434718.

Actual Result

Every later interval is exported at scale 2, with positive.offset = 9. That bucket is (4.757, 5.657], a relative error of about 8.6% instead of about 3.3e-7 at scale 20. It never recovers.

In a worse case, one interval that records 1e-300 and 1e300 pins later intervals at scale -4. After that, four measurements of 10, 12, 15 and 20 ms all land in the single bucket (2^-16, 1].

Additional context

Adding self._mapping = self._new_mapping(self._max_scale) in that DELTA→DELTA branch, next to the bucket reset, restores the maximum scale for every interval. With it, the reproduction above prints 20 for each later interval, and the outlier case goes back to scale 7 with the four values in separate buckets. The 335 tests in opentelemetry-sdk/tests/metrics pass with and without the change, so none of them currently covers this.

For comparison, jmacd noted during review of #2964 that when the structure goes from 0 measurements to 1, "the scale is supposed to reset to the maximum scale again".

Found while auditing the metrics SDK with Claude Code. The outputs above come from running the script against main at 18d7e24.

Would you like to implement a fix?

Not at the moment. The one-line change above is what I tested, and anyone is welcome to pick it up.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions