Repository navigation
Conversation
Storm ships Console, CSV, JMX and Graphite reporters for the V2 metrics, and storm-metrics-prometheus only pushes the cluster metrics to a Pushgateway. With JMX or Graphite the task dimensions end up flattened in the metric names and have to be mapped back to labels with regular expressions. PrometheusStormReporter is a StormReporter which serves the metrics of each worker over HTTP for Prometheus to scrape. It reads the metrics of each task from the MetricRegistryProvider, so the dimensions become the labels topology, host, port, component, task and stream, and a metric reported by several tasks ends up in a single family: - a scoped metric such as fetcher_counter.status_200 becomes fetcher_counter with scope="status_200" - a gauge whose value is a map, such as status.count, becomes status_count with a key label per entry - Storm's own metrics such as __ack-count-spout:default become storm_ack_count with source_component and stream labels; the .m1_rate gauges registered next to Storm's counters are skipped Counters and meters are exposed as counters, histograms and timers as summaries with a _max and a _mean gauge, timers in seconds. Each worker listens on its port plus port.offset (1000 by default), using the worker.port system property set by the supervisor. In local mode the workers share a single endpoint on port (9400 by default). The JVM metrics are exposed as well unless jvm.metrics is false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rzo1
left a comment
rzo1
left a comment
There was a problem hiding this comment.
Thanks for this, the label-based layout is a good fit for the V2 metrics. A few things inline, mostly around port handling and family naming. The suggestions compile together and the module tests pass with them applied; where two overlap I folded them into one.
| gauges.forEach((name, gauge) -> { | ||
| // gauges call arbitrary code, one failing must not break the scrape | ||
| try { | ||
| gauge(name, gauge.getValue()); | ||
| } catch (RuntimeException e) { | ||
| LOG.debug("Could not read gauge {}", name, e); | ||
| } | ||
| }); | ||
| counters.forEach((name, counter) -> counter(name, counter.getCount())); | ||
| meters.forEach((name, meter) -> counter(name, meter.getCount())); |
There was a problem hiding this comment.
Gauges are converted first, so a gauge X.<scope> takes family X as GAUGE before counter or meter X gets there. Families.add then drops the counter, and the only trace is one WARN. You can hit this through TopologyContext with registerCounter("pages") plus registerGauge("pages.foo", ...) on the same task. A non-__ RateCounter hits it too, because its .m1_rate gauge gets past the skip on line 262. Converting counters and meters first keeps the counter, and skipping any .m1_rate gauge that has a matching counter covers RateCounters outside the __ namespace.
| gauges.forEach((name, gauge) -> { | |
| // gauges call arbitrary code, one failing must not break the scrape | |
| try { | |
| gauge(name, gauge.getValue()); | |
| } catch (RuntimeException e) { | |
| LOG.debug("Could not read gauge {}", name, e); | |
| } | |
| }); | |
| counters.forEach((name, counter) -> counter(name, counter.getCount())); | |
| meters.forEach((name, meter) -> counter(name, meter.getCount())); | |
| counters.forEach((name, counter) -> counter(name, counter.getCount())); | |
| meters.forEach((name, meter) -> counter(name, meter.getCount())); | |
| gauges.forEach((name, gauge) -> { | |
| // the rate gauge of a RateCounter, the counter itself is exported | |
| if (name.endsWith(RATE_SUFFIX) && counters.containsKey( | |
| name.substring(0, name.length() - RATE_SUFFIX.length()))) { | |
| return; | |
| } | |
| // gauges call arbitrary code, one failing must not break the scrape | |
| try { | |
| gauge(name, gauge.getValue()); | |
| } catch (RuntimeException e) { | |
| LOG.debug("Could not read gauge {}", name, e); | |
| } | |
| }); |
A test with a counter pages and a gauge pages.foo on the same task, asserting that the counter values are exported, would cover this.
| try { | ||
| endpoint = new Endpoint(port, host, jvmMetrics); | ||
| } catch (IOException e) { | ||
| LOG.error("Could not serve the metrics on port {}", port, e); | ||
| return; | ||
| } |
There was a problem hiding this comment.
Only IOException is caught here. A resolved port above 65535 (e.g. a slot port above 64535 with the default port.offset of 1000, or a bad explicit port) makes InetSocketAddress throw IllegalArgumentException from buildAndStart(). Nothing in StormMetricRegistry.startReporter or Worker.start catches it, so the worker dies and the supervisor relaunches it into the same failure, while a bind failure is only logged. Catch IOException | IllegalArgumentException here (folded into the suggestion on lines 111-123 below, since they overlap). If you'd rather fail fast, range-check in resolvePort and throw with a message naming port/port.offset, instead of the bare port out of range:65600.
| static int resolvePort(Map<String, Object> reporterConf, String workerPort) { | ||
| Integer port = ObjectReader.getInt(reporterConf.get(PORT_KEY), null); | ||
| if (port != null) { | ||
| return port; | ||
| } |
There was a problem hiding this comment.
An explicit port wins even when worker.port is set, so on a real cluster every worker on a host resolves to the same port. Only the first one binds. The others hit the IOException branch in start(), log an ERROR and keep running with no endpoint, and nothing else reports it, so Prometheus silently loses those workers. The README only says port "takes precedence over port.offset". At minimum, warn when port is used under a supervisor and document that it is meant for local mode or single-slot hosts. Rethrowing from start() would be louder but would fail worker startup, so I'd keep the log there.
| static int resolvePort(Map<String, Object> reporterConf, String workerPort) { | |
| Integer port = ObjectReader.getInt(reporterConf.get(PORT_KEY), null); | |
| if (port != null) { | |
| return port; | |
| } | |
| static int resolvePort(Map<String, Object> reporterConf, String workerPort) { | |
| Integer port = ObjectReader.getInt(reporterConf.get(PORT_KEY), null); | |
| if (port != null) { | |
| if (workerPort != null) { | |
| LOG.warn("{} is set to {} for worker port {}, workers sharing this host " | |
| + "will fail to bind, use {} instead", | |
| PORT_KEY, port, workerPort, PORT_OFFSET_KEY); | |
| } | |
| return port; | |
| } |
The README needs an update as well (see extra changes).
Other changes needed
--- a/external/storm-metrics-prometheus/README.md
+++ b/external/storm-metrics-prometheus/README.md
@@ -34 +34 @@
-| `port` | | Port to listen on, takes precedence over `port.offset`. Used as is in local mode, where it defaults to 9400. |
+| `port` | | Port to listen on, takes precedence over `port.offset`. Meant for local mode, where it defaults to 9400. On a cluster all workers of a host would get the same port and only the first one binds, so use `port.offset` there. || labels = Labels.of("source_component", base.substring(dash + 1)); | ||
| base = base.substring(0, dash); | ||
| } | ||
| } |
There was a problem hiding this comment.
The Netty client gauges from Client (__send-iconnection-{reconnects,sent,pending,lostOnSend}-<host>:<port>, on by default via topology.enable.send.iconnection.metrics) are registered on __system with no stream, so they fall through to line 194 and the remote host and port get sanitized into the family name, e.g. storm_send_iconnection_sent_worker2_example_com_6701. That makes 4 families per remote connection, which you can't aggregate without regex on __name__. Split the trailing -<host>:<port> into labels instead:
| } | |
| } else if (stream == null && base.startsWith(ICONNECTION_PREFIX)) { | |
| // e.g. __send-iconnection-sent-worker2.example.com:6701 | |
| int dash = base.indexOf('-', ICONNECTION_PREFIX.length()); | |
| int colon = base.lastIndexOf(':'); | |
| if (dash > 0 && colon > dash + 1) { | |
| labels = Labels.of("remote_host", base.substring(dash + 1, colon), | |
| "remote_port", base.substring(colon + 1)); | |
| base = base.substring(0, dash); | |
| } | |
| } |
This also needs the ICONNECTION_PREFIX constant next to STORM_PREFIX, and a test (see diff). The README could list the resulting storm_send_iconnection_* families with their remote_host/remote_port labels.
Other changes needed
--- a/external/storm-metrics-prometheus/src/main/java/org/apache/storm/metrics/prometheus/StormMetricsCollector.java
+++ b/external/storm-metrics-prometheus/src/main/java/org/apache/storm/metrics/prometheus/StormMetricsCollector.java
@@ -83,6 +83,9 @@
/** Prefix of the metrics Storm registers for each task. */
private static final String STORM_PREFIX = "__";
+ /** Netty client gauges, suffixed with the remote host and port. */
+ private static final String ICONNECTION_PREFIX = "send-iconnection-";
+
/**
* Storm registers a gauge with this suffix next to each of its counters. Prometheus computes
* rates from the counter itself.
--- a/external/storm-metrics-prometheus/src/test/java/org/apache/storm/metrics/prometheus/StormMetricsCollectorTest.java
+++ b/external/storm-metrics-prometheus/src/test/java/org/apache/storm/metrics/prometheus/StormMetricsCollectorTest.java
@@ -60,6 +60,16 @@
StormMetricsCollector.parse("__receive.population", null));
}
+ @Test
+ void stormConnectionMetric() {
+ assertEquals(new Name("storm_send_iconnection_sent",
+ Labels.of("remote_host", "worker-2.example.com", "remote_port", "6701")),
+ StormMetricsCollector.parse("__send-iconnection-sent-worker-2.example.com:6701", null));
+ assertEquals(new Name("storm_send_iconnection_lostOnSend",
+ Labels.of("remote_host", "10.0.0.5", "remote_port", "6700")),
+ StormMetricsCollector.parse("__send-iconnection-lostOnSend-10.0.0.5:6700", null));
+ }
+
@Test
void sanitize() {
assertEquals("bytes_fetched_perSec", StormMetricsCollector.sanitize("bytes_fetched_perSec"));| /** Data points grouped by family. */ | ||
| private static final class Families { | ||
|
|
||
| private final Map<String, Family> byName = new TreeMap<>(); | ||
| private final Set<String> conflicts; | ||
|
|
||
| Families(Set<String> conflicts) { | ||
| this.conflicts = conflicts; | ||
| } | ||
|
|
||
| void add(String name, Type type, DataPointSnapshot point) { | ||
| Family family = byName.computeIfAbsent(name, n -> new Family(type)); | ||
| if (family.type != type) { | ||
| if (conflicts.add(name)) { | ||
| LOG.warn("Metric family {} is reported both as a {} and a {}, " | ||
| + "ignoring the latter", name, family.type, type); | ||
| } | ||
| return; | ||
| } | ||
| family.points.putIfAbsent(point.getLabels(), point); | ||
| } | ||
|
|
||
| MetricSnapshots build() { | ||
| MetricSnapshots.Builder builder = MetricSnapshots.builder(); | ||
| byName.forEach((name, family) -> builder.metricSnapshot(family.snapshot(name))); | ||
| return builder.build(); | ||
| } | ||
| } |
There was a problem hiding this comment.
The first point seen in a scrape fixes the family type, and every point of another type is dropped. Across tasks that order is ConcurrentHashMap hash order. Within a task, gauges always come before counters, meters, histograms and timers. So with counter fetcher.pages in one bolt and gauge fetcher.queue in another, whichever task is iterated second loses all of its fetcher series on every scrape. Because conflicts is never cleared, that happens silently after the first warning. Prometheus cannot mix types in one family, but the resolution should not depend on iteration order. Keep the points per type and suffix every type on a collision, so nothing is dropped and the names stay stable:
| /** Data points grouped by family. */ | |
| private static final class Families { | |
| private final Map<String, Family> byName = new TreeMap<>(); | |
| private final Set<String> conflicts; | |
| Families(Set<String> conflicts) { | |
| this.conflicts = conflicts; | |
| } | |
| void add(String name, Type type, DataPointSnapshot point) { | |
| Family family = byName.computeIfAbsent(name, n -> new Family(type)); | |
| if (family.type != type) { | |
| if (conflicts.add(name)) { | |
| LOG.warn("Metric family {} is reported both as a {} and a {}, " | |
| + "ignoring the latter", name, family.type, type); | |
| } | |
| return; | |
| } | |
| family.points.putIfAbsent(point.getLabels(), point); | |
| } | |
| MetricSnapshots build() { | |
| MetricSnapshots.Builder builder = MetricSnapshots.builder(); | |
| byName.forEach((name, family) -> builder.metricSnapshot(family.snapshot(name))); | |
| return builder.build(); | |
| } | |
| } | |
| /** Data points grouped by family. */ | |
| private static final class Families { | |
| private final Map<String, Map<Type, Family>> byName = new TreeMap<>(); | |
| private final Set<String> conflicts; | |
| Families(Set<String> conflicts) { | |
| this.conflicts = conflicts; | |
| } | |
| void add(String name, Type type, DataPointSnapshot point) { | |
| byName.computeIfAbsent(name, n -> new EnumMap<>(Type.class)) | |
| .computeIfAbsent(type, Family::new) | |
| .points.putIfAbsent(point.getLabels(), point); | |
| } | |
| MetricSnapshots build() { | |
| MetricSnapshots.Builder builder = MetricSnapshots.builder(); | |
| byName.forEach((name, types) -> { | |
| if (types.size() == 1) { | |
| types.forEach((type, family) -> builder.metricSnapshot(family.snapshot(name))); | |
| return; | |
| } | |
| // a family has a single type in Prometheus, suffix all of them so that the result | |
| // does not depend on the order in which the tasks are reported | |
| if (conflicts.add(name)) { | |
| LOG.warn("Metric family {} is reported as {}, exposing it as {}_<type>", name, | |
| types.keySet(), name); | |
| } | |
| types.forEach((type, family) -> { | |
| String suffixed = name + "_" + type.name().toLowerCase(Locale.ROOT); | |
| if (byName.containsKey(suffixed)) { | |
| LOG.debug("Skipping {} as {} already exists", type, suffixed); | |
| return; | |
| } | |
| builder.metricSnapshot(family.snapshot(suffixed)); | |
| }); | |
| }); | |
| return builder.build(); | |
| } | |
| } |
This also needs java.util.EnumMap and java.util.Locale imports, plus a line in the README Naming section (diff below). A collect() test with a counter and a gauge under the same prefix in two tasks would cover it.
Other changes needed
--- a/external/storm-metrics-prometheus/src/main/java/org/apache/storm/metrics/prometheus/StormMetricsCollector.java
+++ b/external/storm-metrics-prometheus/src/main/java/org/apache/storm/metrics/prometheus/StormMetricsCollector.java
@@ -38,9 +38,11 @@
import io.prometheus.metrics.model.snapshots.SummarySnapshot.SummaryDataPointSnapshot;
import java.util.ArrayList;
import java.util.Collection;
+import java.util.EnumMap;
import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
--- a/external/storm-metrics-prometheus/README.md
+++ b/external/storm-metrics-prometheus/README.md
@@ -50,5 +50,7 @@
* Counters and meters are exposed as counters, so `rate()` gives their rate per second.
* Histograms and timers are exposed as summaries plus a `_max` and a `_mean` gauge. Timers are in seconds. Dropwizard
histograms do not track a sum, so the summaries have no `_sum`; use the `_mean` gauge instead.
+* A family can only have one type. If metrics of different types map to the same family (e.g. a counter `fetcher.pages`
+ and a gauge `fetcher.queue`), each type gets a suffix instead: `fetcher_counter`, `fetcher_gauge`, `fetcher_summary`.
### Prometheus| Endpoint endpoint = ENDPOINTS.get(port); | ||
| if (endpoint == null) { | ||
| try { | ||
| endpoint = new Endpoint(port, host, jvmMetrics); | ||
| } catch (IOException e) { | ||
| LOG.error("Could not serve the metrics on port {}", port, e); | ||
| return; | ||
| } | ||
| ENDPOINTS.put(port, endpoint); | ||
| LOG.info("Serving the metrics for Prometheus on port {}", | ||
| endpoint.server.getPort()); | ||
| } | ||
| endpoint.collector.add(source); |
There was a problem hiding this comment.
host and jvm.metrics only take effect for the reporter that creates the endpoint. A later reporter on the same port (two topologies in one LocalCluster, which all default to 9400) has its values dropped, and nothing is logged. Either document that the first reporter's settings apply, or warn when they differ.
| Endpoint endpoint = ENDPOINTS.get(port); | |
| if (endpoint == null) { | |
| try { | |
| endpoint = new Endpoint(port, host, jvmMetrics); | |
| } catch (IOException e) { | |
| LOG.error("Could not serve the metrics on port {}", port, e); | |
| return; | |
| } | |
| ENDPOINTS.put(port, endpoint); | |
| LOG.info("Serving the metrics for Prometheus on port {}", | |
| endpoint.server.getPort()); | |
| } | |
| endpoint.collector.add(source); | |
| Endpoint endpoint = ENDPOINTS.get(port); | |
| if (endpoint == null) { | |
| try { | |
| endpoint = new Endpoint(port, host, jvmMetrics); | |
| } catch (IOException | IllegalArgumentException e) { | |
| LOG.error("Could not serve the metrics on port {}", port, e); | |
| return; | |
| } | |
| ENDPOINTS.put(port, endpoint); | |
| LOG.info("Serving the metrics for Prometheus on port {}", | |
| endpoint.server.getPort()); | |
| } else if (!Objects.equals(host, endpoint.host) || jvmMetrics != endpoint.jvmMetrics) { | |
| LOG.warn("Endpoint on port {} already runs with host={} and jvm.metrics={}, " | |
| + "ignoring host={} and jvm.metrics={}", | |
| port, endpoint.host, endpoint.jvmMetrics, host, jvmMetrics); | |
| } | |
| endpoint.collector.add(source); |
Endpoint also has to keep host and jvmMetrics, and java.util.Objects needs an import (see the diff). The catch also includes IllegalArgumentException, see the comment on lines 113-118. If the Endpoint suggestion further down is applied as well, the two fields go into that version.
Other changes needed
--- a/external/storm-metrics-prometheus/src/main/java/org/apache/storm/metrics/prometheus/PrometheusStormReporter.java
+++ b/external/storm-metrics-prometheus/src/main/java/org/apache/storm/metrics/prometheus/PrometheusStormReporter.java
@@ -19,6 +19,7 @@
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
+import java.util.Objects;
import org.apache.storm.Config;
import org.apache.storm.metrics2.MetricRegistryProvider;
import org.apache.storm.metrics2.reporters.ScheduledStormReporter;
@@ -156,8 +157,12 @@
private final StormMetricsCollector collector = new StormMetricsCollector();
private final HTTPServer server;
+ private final String host;
+ private final boolean jvmMetrics;
Endpoint(int port, String host, boolean jvmMetrics) throws IOException {
+ this.host = host;
+ this.jvmMetrics = jvmMetrics;
PrometheusRegistry registry = new PrometheusRegistry();
registry.register(collector);
if (jvmMetrics) {| private static final class Endpoint { | ||
|
|
||
| private final StormMetricsCollector collector = new StormMetricsCollector(); | ||
| private final HTTPServer server; | ||
|
|
||
| Endpoint(int port, String host, boolean jvmMetrics) throws IOException { | ||
| PrometheusRegistry registry = new PrometheusRegistry(); | ||
| registry.register(collector); | ||
| if (jvmMetrics) { | ||
| JvmMetrics.builder().register(registry); | ||
| } | ||
| HTTPServer.Builder builder = HTTPServer.builder().port(port).registry(registry); | ||
| if (host != null) { | ||
| builder.hostname(host); | ||
| } | ||
| server = builder.buildAndStart(); | ||
| } | ||
| } |
There was a problem hiding this comment.
JvmMetrics.builder().register(registry) adds each new PrometheusRegistry to the static JvmMetrics.REGISTERED set, and nothing ever removes it. JvmMemoryPoolAllocationMetrics also adds a GC NotificationListener on every register and never removes it. So every endpoint create/close cycle in one JVM, e.g. local-mode submit/kill loops on 9400, leaves behind a registry, the JVM collectors and a GC listener. Register the JVM metrics once on a lazily created static registry and have each endpoint registry delegate to it. Closing an endpoint then leaks nothing. Needs import io.prometheus.metrics.model.registry.MultiCollector;. The cast avoids ambiguity between the register(Collector) and register(MultiCollector) overloads.
| private static final class Endpoint { | |
| private final StormMetricsCollector collector = new StormMetricsCollector(); | |
| private final HTTPServer server; | |
| Endpoint(int port, String host, boolean jvmMetrics) throws IOException { | |
| PrometheusRegistry registry = new PrometheusRegistry(); | |
| registry.register(collector); | |
| if (jvmMetrics) { | |
| JvmMetrics.builder().register(registry); | |
| } | |
| HTTPServer.Builder builder = HTTPServer.builder().port(port).registry(registry); | |
| if (host != null) { | |
| builder.hostname(host); | |
| } | |
| server = builder.buildAndStart(); | |
| } | |
| } | |
| private static final class Endpoint { | |
| private final StormMetricsCollector collector = new StormMetricsCollector(); | |
| private final HTTPServer server; | |
| Endpoint(int port, String host, boolean jvmMetrics) throws IOException { | |
| PrometheusRegistry registry = new PrometheusRegistry(); | |
| registry.register(collector); | |
| if (jvmMetrics) { | |
| registry.register((MultiCollector) JvmRegistry.INSTANCE::scrape); | |
| } | |
| HTTPServer.Builder builder = HTTPServer.builder().port(port).registry(registry); | |
| if (host != null) { | |
| builder.hostname(host); | |
| } | |
| server = builder.buildAndStart(); | |
| } | |
| } | |
| /** JVM metrics registered once per JVM, as {@link JvmMetrics} never releases a registry. */ | |
| private static final class JvmRegistry { | |
| static final PrometheusRegistry INSTANCE = new PrometheusRegistry(); | |
| static { | |
| JvmMetrics.builder().register(INSTANCE); | |
| } | |
| } |
Other changes needed
--- a/external/storm-metrics-prometheus/src/main/java/org/apache/storm/metrics/prometheus/PrometheusStormReporter.java
+++ b/external/storm-metrics-prometheus/src/main/java/org/apache/storm/metrics/prometheus/PrometheusStormReporter.java
@@ -15,6 +15,7 @@
import com.codahale.metrics.MetricRegistry;
import io.prometheus.metrics.exporter.httpserver.HTTPServer;
import io.prometheus.metrics.instrumentation.jvm.JvmMetrics;
+import io.prometheus.metrics.model.registry.MultiCollector;
import io.prometheus.metrics.model.registry.PrometheusRegistry;
import java.io.IOException;
import java.util.HashMap;
2 participants
Storm ships Console, CSV, JMX and Graphite reporters for the V2 metrics, and
storm-metrics-prometheusonly pushes the cluster metrics to a Pushgateway. With JMX or Graphite, the task dimensions end up flattened in names likestorm.worker.<topology>.<host>.<component>.<task>.<port>-<metric>, which have to be mapped back to labels with regular expressions.This adds
PrometheusStormReportertostorm-metrics-prometheus, a V2StormReporterwhich serves the metrics of each worker over HTTP for Prometheus to scrape. It is a port of apache/stormcrawler#2230.How it works
The reporter reads the metrics of each task from the
MetricRegistryProvider, along with the task's dimensions. These become labels directly (topology,host,port,component,task,stream), so a metric reported by several tasks ends up in a single family, without any name parsing:fetcher_counter.status_200) become the familyfetcher_counterwith ascopelabel. Only the first dot separates the scope, sofetcher_counter.robots.fetchedhasscope="robots.fetched".status.count->status_count) with akeylabel per entry.__ack-count-spout:default) becomestorm_ack_countwithsource_componentandstreamlabels. The.m1_rategauges registered next to Storm's counters are skipped, since Prometheus computes rates from the counters._maxand a_meangauge. Timers are converted to seconds.Configuration
Each worker listens on its port plus
port.offset(7700 for a worker on 6700), using theworker.portsystem property set by the supervisor, so Prometheus scrapes one target per slot. In local mode, where the worker port is unknown, the workers share a single endpoint onport(9400 by default). The JVM metrics (memory, GC, threads,process_start_time_seconds) are exposed as well unlessjvm.metricsis false, and the usualfilterapplies. See the module's README;docs/metrics_v2.mdlists the new reporter.The module gains dependencies on
prometheus-metrics-exporter-httpserverandprometheus-metrics-instrumentation-jvm(1.9.0, as for the existing Prometheus dependencies), added toDEPENDENCY-LICENSES.Testing
StormMetricsCollectorTest: unit tests for the naming rules.PrometheusStormReporterTest: starts a realStormMetricRegistry, registers a metric of each type (including a map gauge and one of Storm's rate counters), scrapes the HTTP endpoint and checks the samples; also covers the port selection.Known limitations
_sum; use the_meangauge instead.ack-count,process-latency...), which holds for all of Storm's current ones.memory{scope="heap.used"}. The JVM metrics cover the same ground.🤖 Generated with Claude Code