Skip to content

feat(tunnel): expose per-controller MaxConcurrentReconciles for source reconcilers#159

Open
kochj23 wants to merge 1 commit into
jacaudi:mainfrom
kochj23:feat/134-per-controller-concurrency
Open

feat(tunnel): expose per-controller MaxConcurrentReconciles for source reconcilers#159
kochj23 wants to merge 1 commit into
jacaudi:mainfrom
kochj23:feat/134-per-controller-concurrency

Conversation

@kochj23

@kochj23 kochj23 commented Jul 1, 2026

Copy link
Copy Markdown

Summary

Every controller in the tunnel bundle (internal/controller/tunnel/setup.go) was built with ctrl.NewControllerManagedBy(mgr).Named(...).For(...).Complete(...) and no .WithOptions(controller.Options{MaxConcurrentReconciles: N}), so controller-runtime defaults each to 1 concurrent reconcile. Concurrent HTTPRoute events serialize behind a single worker.

This PR adds a typed ConcurrencyOptions struct on tunnel.Options and threads a per-controller MaxConcurrentReconciles through all five builders. The zero value preserves current behavior — controller-runtime's controller.New clamps any MaxConcurrentReconciles <= 0 to its default of 1 (pkg/controller/controller.go:244 in the pinned v0.24.1), so an unset ConcurrencyOptions changes nothing.

Root cause / metrics (from #134)

Homelab cluster, v0.19.0, meta+tunnel mode, one external-jacaudi-dev Gateway with three attached HTTPRoutes:

  • 245 reconciles over ~11h on httproute-source, all event-driven (zero requeue_after) — the watch path is hot.
  • Each reconcile ~5ms (p95) — the work itself is fast.
  • Yet workqueue_queue_duration_seconds p95 ~1s — events sit in the queue waiting for the single reconcile worker.

The operator-side dwell is the component we can tighten; it matters most when several HTTPRoutes change in the same window (a multi-app Flux apply).

Design

  • Typed struct, not map[string]int. Named int fields per controller (Tunnel, Service, Gateway, HTTPRoute, TLSRoute) — compile-time safe and IDE-discoverable, matching the existing Options-struct-of-named-fields pattern (DefaultImage, DefaultConnector, ...) and go-standards §1.
  • Per-controller, not a single global knob — load profiles differ; httproute-source is hot while service-source/gateway-source are cooler and tlsroute-source is often disabled.
  • applyOptionDefaults deliberately does not populate Concurrency. Any default bump belongs in the chart, not the Go zero-value default (the issue's "default-bump alternative").
  • Concurrency safety. The source reconcilers are already designed for MaxConcurrentReconciles > 1: their per-instance lazy state is sync.Once-guarded for exactly this reason (see the MaxConcurrentReconciles > 1 notes in source_base.go and each *_source_controller.go), and controller-runtime serializes reconciles per object via the workqueue regardless of the setting.

Scope

This PR is the Go-library change the issue's "Suggested path" centers on (the ConcurrencyOptions struct + threading through the five setup.go builders). The flag parsing (--tunnel-concurrency-*), chart/values.yaml, and meta-operator deployment-args wiring described in the issue are a natural follow-up and can be scoped to the full-vs-HTTPRoute-only variant the issue asks about — happy to add them here or in a follow-up PR, whichever the maintainer prefers.

Tests

Unit tests (internal/controller/tunnel/concurrency_test.go) plus an envtest integration smoke (test/envtest/tunnel_concurrency_envtest_test.go). Category coverage:

  1. Happy pathTestControllerOptions_Passthrough: a positive value maps straight to controller.Options{MaxConcurrentReconciles}.
  2. Boundary / default — same test: zero maps to 0 (documenting controller-runtime's <=0 -> 1 normalization against the pinned version).
  3. Negative input — same test: negative is passed through unchanged (no pre-validation; controller-runtime clamps to 1), matching the issue's note.
  4. Per-field wiringTestConcurrencyOptions_PerFieldIndependence: five distinct field values thread without cross-talk (the payoff of the typed struct).
  5. RegressionTestApplyOptionDefaults_LeavesConcurrencyZero: defaulting must not clobber the zero-value passthrough, and must preserve a caller-supplied value.
  6. IntegrationTestEnvtest_AddToManager_AppliesConcurrency: builds a real manager and calls AddToManager with non-default concurrency on every field, asserting the whole bundle wires without error (proves all five .WithOptions call-sites integrate).
  7. Concurrency (thread-safety) — N/A (placeholder comment): this knob enables parallel reconciles; the reconcilers' safety under it is already guarded by the sync.Once lazy-state init and exercised by the existing tunnel envtest suite. No new race surface.

go build ./..., go vet ./..., and go test ./... pass locally (Go 1.26). The test/envtest suite skips gracefully when KUBEBUILDER_ASSETS is unset in this environment; the integration smoke above should run in maintainer CI where the kube binaries are available. The unit tests fully cover the value/defaulting logic without envtest.

Closes #134

🤖 Generated with Claude Code

Each source controller in internal/controller/tunnel/setup.go was built
with NewControllerManagedBy(mgr).Named(...).For(...).Complete(...) and no
.WithOptions(controller.Options{MaxConcurrentReconciles: N}), so
controller-runtime defaulted every one to 1 concurrent reconcile.
Concurrent HTTPRoute events therefore serialize behind a single worker.

Issue jacaudi#134 measured this in a homelab cluster (v0.19.0, meta+tunnel, one
Gateway with three HTTPRoutes): 245 event-driven reconciles over ~11h,
each ~5ms (p95), yet workqueue p95 dwell on httproute-source ~1s — events
sitting in the queue waiting for the single worker. A multi-app Flux apply
that changes several HTTPRoutes at once is where this dwell hurts.

Add a typed ConcurrencyOptions struct on tunnel.Options with a named int
field per controller (Tunnel/Service/Gateway/HTTPRoute/TLSRoute) and thread
each through its builder via .WithOptions(controllerOptions(...)). Named
fields over a map[string]int keep the API compile-time safe and match the
existing Options-struct-of-named-fields pattern (go-standards §1).

The zero value preserves current behavior: controller-runtime's
controller.New clamps MaxConcurrentReconciles <= 0 to its default of 1
(controller.go:244), so an unset ConcurrencyOptions changes nothing.
applyOptionDefaults deliberately does NOT populate Concurrency — any
default bump belongs in the chart, not the Go zero-value default (see the
issue's "default-bump alternative").

The source reconcilers are already safe under MaxConcurrentReconciles > 1:
their per-instance lazy state is sync.Once-guarded precisely for this (see
the "MaxConcurrentReconciles > 1" notes in source_base.go and the
*_source_controller.go files), and controller-runtime serializes reconciles
per object via the workqueue regardless of the setting.

Closes jacaudi#134

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tunnel: expose per-controller MaxConcurrentReconciles for source reconcilers

1 participant