bug: Prometheus shared dict free_space drops periodically when metric expire is enabled

[x1117]

Current Behavior

While using APISIX with the Prometheus plugin enabled, we observed that the free space of the prometheus-metrics shared dict decreases periodically in a step-like pattern.

The metric we monitored is:

apisix_shared_dict_free_space_bytes{name="prometheus-metrics"}

The pattern looks like this:

free_space stays stable for a while
  ↓
drops by one step
  ↓
stays stable again
  ↓
drops by another step

In our environment, this happened approximately once per hour. The active time series count was not very high, but the shared dict free space kept decreasing.

We also dumped the prometheus-metrics shared dict and found many expired KeyIndex entries still present, for example:

__ngx_prom__key_N

One dump showed:

key_count=751496
free=148631552
cap=536870912
phys_used=388239360

Entry summary:

IDX PERM   76
IDX alive  28
IDX STALE  747970

VAL PERM   75
VAL alive  28
VAL STALE  4124
VAL GONE   77

Most stale entries were __ngx_prom__key_N index entries.

We reproduced the same type of behavior locally with APISIX 3.17.0.

During reproduction, prometheus-metrics free space dropped multiple times:

11:43:26  10395648 -> 10330112  drop 65536
11:50:08  10330112 -> 10289152  drop 40960
12:01:27  10289152 -> 8794112   drop 1495040
12:01:42  8794112  -> 8638464   drop 155648
12:08:08  8638464  -> 7581696   drop 1056768

It looks like expired __ngx_prom__key_N entries may remain in the prometheus-metrics shared dict and continue occupying memory, while later traffic creates new KeyIndex slots.

Expected Behavior

When Prometheus metrics are configured with expire > 0, the expired metric value entries and their related KeyIndex entries are expected to be reclaimed effectively, or at least not keep consuming additional physical memory in the prometheus-metrics shared dict.

If the number of active time series is bounded, the free space of prometheus-metrics should eventually become stable instead of continuously dropping step by step.

From our observation, the expired entries may not be physically reclaimed because OpenResty shared dict expiration is lazy. The passive expiration path only checks a small number of entries from the LRU tail, and it can be blocked when permanent entries are at the tail. As a result, expired Prometheus entries can remain in the shared dict and continue occupying slab memory.

If this is expected behavior of the current implementation, could you please suggest the recommended way to avoid continuous prometheus-metrics shared dict memory consumption by expired entries?

Error Logs

There were no fatal APISIX errors during reproduction.

We temporarily added logs around Prometheus KeyIndex operations to understand the behavior.

The KeyIndex slot number kept increasing:

03:43:17  idx=48..161
03:49:57  idx=162..275
04:01:22~04:01:28  idx=276..2555
04:08:04~04:08:07  idx=2556..4721

remove_expired_keys() also reported worker-local cleanup:

remove_expired_keys checked=2280 removed=2280 last=2555

After that, later traffic created new KeyIndex slots.

Steps to Reproduce

1. Run APISIX 3.17.0 via Docker:

apache/apisix:3.17.0-debian

2. Enable the Prometheus plugin and configure metric expiration:

plugin_attr:
  prometheus:
    metrics:
      http_status:
        expire: 120
      http_latency:
        expire: 120
      bandwidth:
        expire: 120
      upstream_status:
        expire: 120

3. Create one or more routes with the Prometheus plugin enabled.

4. Send intermittent traffic. For example:

active traffic for 60s
idle for 330s
active traffic for 60s
idle for 330s
active traffic for 60s

5. Monitor the following metric:

apisix_shared_dict_free_space_bytes{name="prometheus-metrics"}

6. In our local reproduction, to shorten the reproduction time, we temporarily changed Prometheus initialization from:

prometheus = base_prometheus.init("prometheus-metrics", metric_prefix)

to:

prometheus = base_prometheus.init("prometheus-metrics", {
    prefix = metric_prefix,
    remove_expired_keys_interval = 300,
})

This made the local cleanup interval shorter than the default 3600s.

7. Observe that prometheus-metrics free space drops in steps. In our reproduction, we observed:

11:43:26  10395648 -> 10330112  drop 65536
11:50:08  10330112 -> 10289152  drop 40960
12:01:27  10289152 -> 8794112   drop 1495040
12:01:42  8794112  -> 8638464   drop 155648
12:08:08  8638464  -> 7581696   drop 1056768

Environment

• APISIX version: 3.17.0
• Docker image: apache/apisix:3.17.0-debian
• Prometheus plugin: enabled
• Prometheus dependency in APISIX 3.17.0:

nginx-lua-prometheus-api7 = 0.20250302-1

• Operating system for local reproduction:

Linux 6.6.87.2-microsoft-standard-WSL2

• etcd version used in local Docker compose:

bitnamilegacy/etcd:3.5.11

• APISIX Dashboard version: not used
• Plugin runner: not used