Add a Prometheus reporter for the Storm V2 metrics

[jnioche]

StormCrawler can register its metrics with the Storm V2 API (stormcrawler.metrics.version: "v2" or "both"), but Storm only ships Console, CSV, JMX and Graphite reporters for them. Storm's storm-metrics-prometheus module only pushes the Nimbus cluster summary metrics to a Pushgateway. With JMX or Graphite, the dimensions end up flattened in names like storm.worker.<topology>.<host>.<component>.<task>.<port>-fetcher_counter.status_200, which have to be mapped back to labels with regular expressions.

This adds a new module, external/prometheus, with PrometheusReporter, a V2 StormReporter which serves the metrics of each worker over HTTP for Prometheus to scrape.

How it works

Storm passes its reporters a MetricRegistryProvider, whose getTaskMetrics() gives the metrics of each task under their short name along with the task's dimensions. These become labels directly (topology, host, port, component, task, stream), so a metric reported by several tasks ends up in a single family, without any name parsing:

fetcher_counter_total{component="fetcher",host="worker1",port="6700",scope="status_200",task="3",topology="crawl"} 1234.0
status_count{component="status",host="worker1",key="FETCHED",port="6700",task="4",topology="crawl"} 56789.0
storm_ack_count_total{component="status",host="worker1",port="6700",source_component="spout",stream="default",task="4",topology="crawl"} 42.0

• Scoped metrics (fetcher_counter.status_200) become the family fetcher_counter with a scope label. Only the first dot separates the scope, so fetcher_counter.robots.fetched has scope="robots.fetched".
• Gauges whose value is a map, such as status.count from StatusMetricsBolt, become a family named after the whole metric (status_count) with a key label per entry.
• Storm's own metrics (__ack-count-spout:default) become storm_ack_count with source_component and stream labels. The .m1_rate gauges Storm registers next to its counters are skipped, since Prometheus computes rates from the counters.
• Counters and meters are exposed as counters, histograms and timers as summaries plus a _max and a _mean gauge. Timers are converted to seconds.
• A gauge which throws when read is skipped rather than failing the whole scrape.

Configuration

  stormcrawler.metrics.version: "v2"
  topology.metrics.reporters:
    - class: "org.apache.stormcrawler.prometheus.PrometheusReporter"
      port.offset: 1000

Each worker listens on its port plus port.offset (7700 for a worker on 6700), using the worker.port system property set by the supervisor, so Prometheus scrapes one target per slot. In local mode, where the worker port is unknown, the workers share a single endpoint on port (9400 by default). The JVM metrics (memory, GC, threads, process_start_time_seconds) are exposed as well unless jvm.metrics is false, and the usual Storm filter applies. See the module's README.

Testing

• Unit tests for the naming rules and the port selection.
• PrometheusReporterTest starts a real StormMetricRegistry, registers a metric of each type (including a map gauge and one of Storm's rate counters), scrapes the HTTP endpoint and checks the samples.
• Outside of the tests, I fed two scrapes of metrics named as in a typical crawl topology into VictoriaMetrics and checked that the PromQL queries of a set of Grafana dashboards (fetching, spout, status updates, parsing, Storm capacity, latency, JVMs) all returned the expected series.

Known limitations

• Dropwizard histograms do not track a sum, so the summaries have no _sum; use the _mean gauge instead.
• The source component of Storm's per-stream metrics is found by assuming their names have two words (ack-count, process-latency...), which holds for all of Storm's current ones.
• Splitting on the first dot suits StormCrawler's metric names, but puts Storm's own system metrics into families like memory{scope="heap.used"}. The JVM metrics cover the same ground.
• cache.hits and cache.misses from AbstractStatusUpdaterBolt are reset on read, which is fixed separately in AbstractStatusUpdaterBolt: report cache hits/misses with a counter, not reset-on-read gauges #2229.

🤖 Generated with Claude Code

[jnioche]

@rzo1 move upstream to Apache Storm instead?

[rzo1]

@rzo1 move upstream to Apache Storm instead?

Yes, adding it upstream make sense. Think all can benefit from it.

[jnioche]

done in apache/storm#9158