Skip to content

Harden NATS messaging telemetry, scaling, and API contracts - #830

Draft
ORESoftware wants to merge 15 commits into
devfrom
agent/nats-messaging-otel-openapi-hardening-20260804
Draft

Harden NATS messaging telemetry, scaling, and API contracts#830
ORESoftware wants to merge 15 commits into
devfrom
agent/nats-messaging-otel-openapi-hardening-20260804

Conversation

@ORESoftware

Copy link
Copy Markdown
Owner

Scope

Audit and harden the deployed Rust NATS HTTP ingress, the always-on durable queue consumer, KEDA lag autoscaling, and the OTEL/Loki/Prometheus/API-contract integration.

The branch is being materialized by a self-validating workflow. Temporary .github/materialize/** files and the materialization workflow delete themselves only after Rust compilation, clippy, unit/live JetStream tests, deterministic OpenAPI generation, observability contract checks, and YAML validation pass.

Intended invariants

  • HTTP ingress remains fail-closed and only publishes configured route/stream pairs after a JetStream ACK.
  • W3C trace context crosses HTTP → NATS → consumer → downstream HTTP/NATS boundaries.
  • Loki receives structured top-level service/severity/event fields.
  • Prometheus directly scrapes ingress and consumer metrics and alerts on dependency loss, overload, handoff failures, fetch errors, and DLQ persistence exhaustion.
  • KEDA keeps one warm listener, scales up quickly, scales down conservatively, and holds replicas when scaler telemetry fails.
  • Ingress OpenAPI is generated from Rust route/schema annotations and CI rejects committed-doc drift.

Deliberately separate follow-up migrations

  • Three-node JetStream R3 storage and broker HA.
  • NATS TLS/accounts/subject permissions via ExternalSecrets.
  • Immutable digest-pinned ingress/consumer images instead of runtime clone/build.
  • Cross-replica idempotency and formal DLQ replay/redaction policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant