rw2: add stateless mode (rc.6)

[bwplotka]

NOTE: Sorry for the large proposal in a PR (majority is written without AI assist).

Specify stateless mode for io.prometheus.write.v2.Request in Remote-Write 2.0-rc.6, following the Prometheus Dev Summit consensus and RFC 9413.

2.0-rc.6 defines an optional stateless mode (Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless), a strict subset of the default specification where senders MUST guarantee (and receivers CAN expect):

1. Metadata.type is set (not METRIC_TYPE_UNSPECIFIED).
2. Exemplars are attached to a TimeSeries that contains at least one sample or histogram (no exemplar-only TimeSeries).
3. start_timestamp is set on every sample and histogram for cumulative (and delta temporality ofc) metric types (METRIC_TYPE_COUNTER and METRIC_TYPE_HISTOGRAM).
4. Classic histograms (_bucket, _sum, _count float series) are not sent; essentially all histogram type metrics needs "histogram" sample. Classic histograms should use NHCBs

By default (when no mode is specified via Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request), all valid field permutations permitted by the base specification may be sent.

• The stateless mode is OPTIONAL for sender and receivers.
• The stateless mode does NOT break receivers in any way -- it is a "sub-protocol", fully embedded in the current RW2 features.

NOTE: stateless mode does not make summaries structured. This is accepted and could be improved in future RW versions, given we generally plan to deprecate summaries in the ecosystem.

Motivation

In the ecosystem we see two core types of adopters of RW/RW2:

1. Prometheus TSDB-style append engines (Prometheus, Mimir, Thanos Receive, VictoriaMetrics):
   • Append float series and exemplars independently by label set (__name__), infer or ignore metadata, and don't require start timestamps or complex types to store a point.
   • For these receivers, none of the 4 capabilities are required (default).
2. Stateless / OTel-aligned backends (OpenTelemetry Collector PRW receiver, Google Cloud Monitoring, Chronosphere, Datadog, ClickHouse/Parquet columnar schemas):
   • They translate each TimeSeries into a self-contained, typed metric point on arrival.
   • They generally need start_timestamp and it's impossible to synthesize it on stateless receiver.
   • They don't easily support metrics without type
     • In OTel/complex-type models, exemplars live inside the data point (or histogram point), not in a detached series table.

To enable easier adoption of (2) while allowing efficiency and compatibility for (1) we propose to group our four features into one mode. This allows:

A) Easier discovery of client vs receive compatibility (e.g. will Thanos Ruler RW work with Otel? Does it support stateless mode?) in the ecosystem. This is much easier than caring about each permutation of features.
B) Avoid too many permutations for different ways of sending the same data.

IETF Protocol Robustness (RFC 9413 — Maintaining Robust Protocols): Explicitly warns against Postel's Law ("be liberal in what you accept") causing protocol ossification and interoperability traps when optional sender behaviors proliferate, recommending strict validation ("virtuous intolerance") and well-defined invariants instead of open-ended sender leniency.

C) User UX in terms of configuring senders

In Prometheus today, producing stateless-ready output requires coordinating multiple separate scrape/TSDB/remote-write settings:

• send_exemplars: true (plus co-locating exemplars with samples)
• metadata_config.send: true (or type-and-unit-labels / metadata-wal-records)
• created-timestamp-zero-ingestion / start timestamp tracking
• convert-classic-histograms-to-nhcb

If the protocol exposes 4 capabilities, users have to diagnose 415/400 errors capability-by-capability and hunt down 4 different flags. With mode: stateless in remote_write config, Prometheus (and other senders like Alloy or OTel Collector) can either enable/validate the required pipeline settings as a single unit or fail fast at startup with a clear error message.

C) Clear separation of typical protocol uses, allowing both to co-exists and evolve. In future (e.g. 3.x) we might get to the state where stateless is a default and only mode (likely ST will remain optional though).

Precedence in other protocols

1. Prometheus / OpenMetrics Content-Type parameters (RFC 9110 §8.3): Prometheus already uses media type parameters on Content-Type / Accept to select wire representation modes without minting a new media type, such as escaping=allow-utf-8 vs escaping=underscores in common/expfmt.
2. HTTP Media Type Profiles (RFC 6906) and ActivityPub / JSON-LD (W3C): Defines a profile/mode as additional structural and semantic constraints on a base media type (application/ld+json; profile="...") such that any conformant payload in the constrained mode remains 100% valid in the base media type, preserving interoperability with unprofiled receivers.
3. (kind of similar) WebSocket Sec-WebSocket-Protocol (RFC 6455 §1.9) & gRPC Content-Type sub-modes (application/grpc+proto): Negotiate a single named sub-mode on top of the framing layer so the server can reject unsupported modes upfront (415) rather than discovering missing capabilities mid-stream.

Alternatives

1. Introduce separate capabilities for each or group of those features

This was discussed a bit on DevSummit. For example typed-metrics, exemplar-per-sample, known-start-timestamp, nhcb

Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;capabilities=typed-metrics,exemplar-per-sample,known-start-timestamp,nhcb

Cons:

• Counter-productive to the rule of organizing a few ways of using RW2 instead of 2^n ways. We have an opportunity to do this, given we do see general 2 ways.
• We have clear examples of other protocols retracting from capabilities:
  • TLS 1.2 allowed negotiating independent capabilities (key exchange + signature + bulk cipher + MAC + extensions). The result was dozens of broken or untested combinations and interoperability failures when peers supported overlapping but mismatched subsets.
  • TLS 1.3 (RFC 8446) and HTTP/2 (RFC 9113) explicitly abandoned à-la-carte capability combinations in favor of a small, fixed set of profiles.
• RFC 9413 (Maintaining Robust Protocols) makes the same point: reducing the number of optional wire states is the primary way to prevent protocol ossification and subtle sender/receiver drift.
• Compliance suite explosion: In prometheus/compliance, testing senders and receivers against default and mode=stateless is 2 test profiles. With 4 capabilities, a receiver might demand any of 15 non-empty subsets, and senders have to map arbitrary combinations of internal flags to header strings.

However, probably the more reasonable choice would be to group it as follows:

• structured - implying exemplar and NHCB (and summary one day)
• typed - implying metric types)
• st-aware- implying ST set on cumulatives

This technically limits the spread. There's also a separation as well given:

• Transport (structured) vs instrumentation features (typed / st-aware)
• Instrumentation vs RW drift (instrumentation allows untyped and ST-less metrics).

Similar cons apply though.

2. Instead of mode call it profile

While it would be similar to e.g. HTTP media type profiles etc, it can be confusing given "profiling" means pprof profile in observability. Mode is clearer (thanks @metalmatze for pointing out!)

3. Instead of stateless call it structured

Generally it allows stateless receiving, so stateless is a clearer name. structured makes sense for type/NHCB/exemplars, except the ST feature.

4. Have stateless mode as a RW 3.0 version

Especially with NHCB we are sure that's a future so why not having 3.x version for stateless mode?

Not ideal because:

• We are not sure/not ready for the receive ecosystem to move to stateless mode fully. Metric are still untyped, start timestamp on cumulatives is not a common need for the native TSDB backends, the same with exemplars per sample. It's better to experiment first to see if the dual category will converge and how in practice.
• Not much change for major version?
• There's a mismatch between scrape protocols and RW in this stricter "stateless" mode. Notably we don't have a good, accepted heuristic for handling unknown metric types.