libsql-server: fix unbounded memory growth from idle-evicted histogram handles

[raphaelfeitoza]

Summary

• Replace all five live cached histogram handles (once_cell::Lazy<Histogram>) with literal-name histogram! recording functions (record_connection_create_time, record_connection_alive_duration, record_total_response_size_before_lock, record_namespace_load_latency, record_replication_latency), so every sample registers against the exporter's current registry entry, including after idle eviction. Remove five unused cached histogram statics.
• Keep MetricKindMask::ALL, the 120 s idle timeout, all metric names/values, and app/version label behaviour, so per-namespace labelled series are still pruned. Register the existing HELP text once after recorder installation, and document the cached-handle hazard next to the exporter config and the metric definitions.
• Add tests for the detached-handle mechanism, recovery and draining of every converted recording path across eviction, and continued counter/gauge pruning.

Based on the release branch v0.9.30-shopify-patches (production image pin 856b4d20 is an ancestor). The three commits are clean cherry-picks (-x) of the reviewed main-based branch fix/metrics-histogram-idle-eviction-leak, with no conflicts. The release branch's /metrics auth exemption is unchanged, and the cumulative patch is identical to the reviewed one.

Root cause

The admin server sets up metrics-exporter-prometheus with idle_timeout(MetricKindMask::ALL, 120s) (http/admin/mod.rs#L83-L86). Hot-path histograms are cached Lazy<Histogram> statics (metrics.rs#L30, #L45, #L50) that are recorded on every connection create/drop (connection/mod.rs#L363-L364, #L392, #L414).
If a pod has no connection activity for more than 120 s between scrapes, the exporter evicts those keys. However, each static handle keeps its old Arc<AtomicBucket<(f64, Instant)>>, and the exporter only drains (clear_with) buckets for keys still in its registry. From then on, each record() adds 16 B that is never freed, until restart or OOM. We observed 133–187 MiB/day/pod in EU. A previous US pod generation reached 47–55 GiB RSS against 56 GiB limits.

Fixes https://github.com/shop/issues-retail/issues/36220

Test evidence (rebased branch, HEAD 2acafe46ef, Rust 1.98.1 per rust-toolchain.toml)

• cargo test -p libsql-server --lib metrics::tests -- --nocapture: 3 passed
• cargo test -p libsql-server --lib: 103 passed, 0 failed, 2 ignored
• cargo fmt --check: clean. git diff --check origin/v0.9.30-shopify-patches...HEAD: clean.
• The test fails without the fix. Restoring the cached-handle pattern in the five record_* bodies (working tree only) makes production_histograms_survive_idle_eviction fail with libsql_server_connection_create_time did not reappear after idle eviction. This was reproduced on this rebased branch. During review, the same failure was shown for each of the five conversions reverted individually. A mutation that re-counts samples on an idle render trips the new drain (burst) assertion.
• Local macOS builds used a non-committed AR wrapper to work around an ar/ld problem with libsql-ffi's pcre2_internal.h member. This is environmental and pre-existing, and Linux CI is not affected.
• Integration tests (libsql-server/tests) were not run locally. They set disable_metrics = true, and CI runs them.
• During review, one earlier full-lib run timed out in connection::connection_core::test::test_many_concurrent, whose code is unchanged. It did not recur in any later run.

Clippy disclosure (gate waived by owner)

cargo clippy -p libsql-server -- -D warnings exits 101 on this branch. The base branch fails identically. The failure comes from untouched path dependencies, and clippy aborts before it reaches libsql-server:

• libsql-ffi/build.rs: 6 errors (5× needless_borrows_for_generic_args, 1× unnecessary_map_or) on both branch and bare origin/v0.9.30-shopify-patches.
• With --keep-going, both report the identical full set: 26 errors (6 libsql-ffi/build.rs + 20 vendored/sqlite3-parser).
• --no-deps --all-targets: both have 265 (lib) / 306 (lib test) pre-existing libsql-server errors, with identical diagnostics and per-file counts, and 0 in metrics.rs. This branch adds no new lints.

No unrelated lints were fixed, no suppressions were added, and no crates or toolchain were bumped. Clippy is not part of this repo's CI.

Rollout notes (owner actions; nothing deployed by this PR)

1. Cut a new libsql tag from v0.9.30-shopify-patches after merge.
2. Bump the pinned tag in the Shopify/libsql-infrastructure Dockerfile and cut a new image version.
3. Roll out via Shopify/infrastructure (applications/libsql-infrastructure): staging (libsql-stg-us-ce1-jm3) first, then EU (libsql-eu-we1-yk9), then US (libsql-us-ce1-za8).

• The restart caused by the rollout clears buckets that have already leaked. The fix does not drain an existing process retroactively.
• After an idle eviction, a re-registered histogram's _count/_sum restart from zero. Existing macro-based histograms already behave this way. rate()/increase() treat it as a counter reset, so this is fine.
• In staging, let an idle eviction happen across scrapes, then resume traffic. Confirm the connection histogram series return, and watch per-pod RSS for 48–72 h.

Follow-ups (not in this PR)

• Cached Lazy<Counter>/Lazy<Gauge> handles still disappear from /metrics permanently after 120 s idle (e.g. libsql_server_concurrent_connections). This only affects observability and causes no memory growth. It needs a separate change.
• Hot-path cost is unbenchmarked. Each sample now adds a registry shard read-lock and lookup plus a small transient handle allocation. Reviewers judged this acceptable, but it was not measured.
• Upstream tursodatabase/libsql main has the same bug, so this fix is a candidate to upstream.
• The exporter has a race on concurrent renders: overlapping /metrics requests can bring back an evicted series as a frozen "zombie". The effect is bounded and is not a leak. This is a dependency issue, to look at in a future crate bump.
• The reviewed main-based branch fix/metrics-histogram-idle-eviction-leak is kept for a later forward-port to main. origin/main has moved since that review, so it will need re-checking.

---

This change was produced with AI assistance and independently AI-reviewed. Both review findings (drain-assertion strength, comment wording) were addressed. The only remaining review blocker was the strict clippy gate, which the owner has waived as described above.