refactor!: replace shipped RL instruments with a consumer-driven metric registry

[rrs45]

What does this PR do?

Lens stops shipping instruments/rl.py. Metric families whose names belong to a consuming framework (RL series in NeMo-RL) are now declared by that consumer through a generic registry and recorded against it, so lens never has to carry — and the consumer never has to import, extend, or override — a per-consumer instrument module. This applies to metrics the same delegation PR #42 applied to rank/world-size.

Why?

The consuming frameworks already emit metrics and create new ones as they evolve. It is best to manage OTel metrics within the frameworks to maintain consistency — without having to override or extend a lens-side definition and add overhead every time a metric changes. Lens remains an abstraction layer for defining and emitting OTel events, allowing the frameworks to drive naming conventions and standards.

New nemo.lens.instruments API (re-exported at the package level)

• MetricSpec(key, name, kind, unit, description) — one series; kind ∈ {gauge, histogram, counter, up_down_counter}, validated at construction. key is the identifier the caller records under (so it must be a Python identifier); name is the OTel name actually emitted (so it can be dotted, e.g. rl.reward.mean).
• register_metric_group / unregister_metric_group / registered_metric_groups — a process-global, lock-guarded group registry mirroring strategies.py.
• record_metrics(meter, group, values=None, /, *, attributes=None, **kwargs) — lazily creates and per-meter caches instruments, skips None, and never raises into the caller. meter/group/values are positional-only so a consumer may declare metric keys with those names; attributes is the one reserved keyword.

Usage

from nemo.lens.instruments import MetricSpec, record_metrics, register_metric_group

# Once, at startup (in the consumer, e.g. NeMo-RL):
register_metric_group(
    "rl",
    [
        MetricSpec("reward_mean", "rl.reward.mean", "gauge"),
        MetricSpec("generation_duration_ms", "rl.generation.duration_ms", "histogram", unit="ms"),
    ],
)

# Per step:
record_metrics(handle.meter, "rl", reward_mean=0.85)

Never raises into the caller

record_metrics sits in a per-step loop, so every failure mode is logged once and skipped rather than raised: an unregistered group, an unknown key, a None value, a malformed values or attributes argument, an unusable meter, and a failed instrument create or emit. Warnings are deduplicated per (group) and per (group, key) so a misconfigured call cannot flood the log.

Fork safety

os.register_at_fork(before / after_in_parent / after_in_child) holds both module locks across a fork so the child starts from a consistent registry, then replaces them with fresh Lock()s in the child — the same approach CPython's logging module takes. A threading.Lock inherited in a locked state has no thread left to release it and would deadlock the child on its next acquire.

Also

• Removed the rl.* metric-name constants from nemo.lens.semconv — NeMo-RL owns those now, declared via the registry. The rl.* span-attribute names (rl.algorithm, rl.reward, rl.generation.backend, rl.num_rollouts) stay in semconv; this PR is metrics-only.
• Added a reset_metric_registry autouse fixture to isolate the registry across tests.
• Updated docs/user-guide/metrics.mdx, docs/design/semconv.mdx, and the AGENTS.md instruments gotcha.

BREAKING CHANGE

nemo.lens.instruments.rl and record_rl_metrics are removed. Consumers register the "rl" group via register_metric_group and emit with record_metrics(meter, "rl", ...).

Before your PR is "Ready for review"

Pre checks:

• I read the Contributor guidelines.

• All commits are signed off (git commit -s) per the DCO.

• I added or updated tests under tests/ for any behavior change.

• I ran pytest and pre-commit run --all-files locally; both pass (239 passed, registry coverage 98%).

• I updated docs/ where behavior, configuration, or public API changed.

• If this changes a lens public API symbol, I updated the cross-repo _fallbacks.py files in Megatron-LM / RL / Gym.

  Cross-repo note: the removed/added symbols are metric record functions, which are not part of the five-symbol _fallbacks.py surface (trace_fn, managed_span, span_cm, is_span_group_enabled, safe_set_span_attributes), so no consumer _fallbacks.py edit is required. However, NeMo-RL needs a paired PR in RL/nemo_rl/telemetry/ to declare its rl.* group via register_metric_group(...) and replace record_rl_metrics(meter, ...) calls with record_metrics(meter, "rl", ...). Megatron-LM and NeMo-Gym do not call record_rl_metrics and are unaffected.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ahmadki]

/ok to test 193f88d

[ahmadki]

/ok to test 9bdcae8

[copy-pr-bot]

/ok to test 9bdcae8

@ahmadki, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[ahmadki]

/ok to test 1c970e9