feat(tunnel): expose per-controller MaxConcurrentReconciles for source reconcilers#159
Open
kochj23 wants to merge 1 commit into
Open
feat(tunnel): expose per-controller MaxConcurrentReconciles for source reconcilers#159kochj23 wants to merge 1 commit into
kochj23 wants to merge 1 commit into
Conversation
Each source controller in internal/controller/tunnel/setup.go was built
with NewControllerManagedBy(mgr).Named(...).For(...).Complete(...) and no
.WithOptions(controller.Options{MaxConcurrentReconciles: N}), so
controller-runtime defaulted every one to 1 concurrent reconcile.
Concurrent HTTPRoute events therefore serialize behind a single worker.
Issue jacaudi#134 measured this in a homelab cluster (v0.19.0, meta+tunnel, one
Gateway with three HTTPRoutes): 245 event-driven reconciles over ~11h,
each ~5ms (p95), yet workqueue p95 dwell on httproute-source ~1s — events
sitting in the queue waiting for the single worker. A multi-app Flux apply
that changes several HTTPRoutes at once is where this dwell hurts.
Add a typed ConcurrencyOptions struct on tunnel.Options with a named int
field per controller (Tunnel/Service/Gateway/HTTPRoute/TLSRoute) and thread
each through its builder via .WithOptions(controllerOptions(...)). Named
fields over a map[string]int keep the API compile-time safe and match the
existing Options-struct-of-named-fields pattern (go-standards §1).
The zero value preserves current behavior: controller-runtime's
controller.New clamps MaxConcurrentReconciles <= 0 to its default of 1
(controller.go:244), so an unset ConcurrencyOptions changes nothing.
applyOptionDefaults deliberately does NOT populate Concurrency — any
default bump belongs in the chart, not the Go zero-value default (see the
issue's "default-bump alternative").
The source reconcilers are already safe under MaxConcurrentReconciles > 1:
their per-instance lazy state is sync.Once-guarded precisely for this (see
the "MaxConcurrentReconciles > 1" notes in source_base.go and the
*_source_controller.go files), and controller-runtime serializes reconciles
per object via the workqueue regardless of the setting.
Closes jacaudi#134
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every controller in the tunnel bundle (
internal/controller/tunnel/setup.go) was built withctrl.NewControllerManagedBy(mgr).Named(...).For(...).Complete(...)and no.WithOptions(controller.Options{MaxConcurrentReconciles: N}), so controller-runtime defaults each to 1 concurrent reconcile. Concurrent HTTPRoute events serialize behind a single worker.This PR adds a typed
ConcurrencyOptionsstruct ontunnel.Optionsand threads a per-controllerMaxConcurrentReconcilesthrough all five builders. The zero value preserves current behavior — controller-runtime'scontroller.Newclamps anyMaxConcurrentReconciles <= 0to its default of 1 (pkg/controller/controller.go:244in the pinned v0.24.1), so an unsetConcurrencyOptionschanges nothing.Root cause / metrics (from #134)
Homelab cluster, v0.19.0, meta+tunnel mode, one
external-jacaudi-devGateway with three attached HTTPRoutes:httproute-source, all event-driven (zerorequeue_after) — the watch path is hot.workqueue_queue_duration_secondsp95 ~1s — events sit in the queue waiting for the single reconcile worker.The operator-side dwell is the component we can tighten; it matters most when several HTTPRoutes change in the same window (a multi-app Flux apply).
Design
map[string]int. Namedintfields per controller (Tunnel,Service,Gateway,HTTPRoute,TLSRoute) — compile-time safe and IDE-discoverable, matching the existingOptions-struct-of-named-fields pattern (DefaultImage,DefaultConnector, ...) and go-standards §1.httproute-sourceis hot whileservice-source/gateway-sourceare cooler andtlsroute-sourceis often disabled.applyOptionDefaultsdeliberately does not populateConcurrency. Any default bump belongs in the chart, not the Go zero-value default (the issue's "default-bump alternative").MaxConcurrentReconciles > 1: their per-instance lazy state issync.Once-guarded for exactly this reason (see theMaxConcurrentReconciles > 1notes insource_base.goand each*_source_controller.go), and controller-runtime serializes reconciles per object via the workqueue regardless of the setting.Scope
This PR is the Go-library change the issue's "Suggested path" centers on (the
ConcurrencyOptionsstruct + threading through the fivesetup.gobuilders). The flag parsing (--tunnel-concurrency-*),chart/values.yaml, and meta-operator deployment-args wiring described in the issue are a natural follow-up and can be scoped to the full-vs-HTTPRoute-only variant the issue asks about — happy to add them here or in a follow-up PR, whichever the maintainer prefers.Tests
Unit tests (
internal/controller/tunnel/concurrency_test.go) plus an envtest integration smoke (test/envtest/tunnel_concurrency_envtest_test.go). Category coverage:TestControllerOptions_Passthrough: a positive value maps straight tocontroller.Options{MaxConcurrentReconciles}.0(documenting controller-runtime's<=0 -> 1normalization against the pinned version).TestConcurrencyOptions_PerFieldIndependence: five distinct field values thread without cross-talk (the payoff of the typed struct).TestApplyOptionDefaults_LeavesConcurrencyZero: defaulting must not clobber the zero-value passthrough, and must preserve a caller-supplied value.TestEnvtest_AddToManager_AppliesConcurrency: builds a real manager and callsAddToManagerwith non-default concurrency on every field, asserting the whole bundle wires without error (proves all five.WithOptionscall-sites integrate).sync.Oncelazy-state init and exercised by the existing tunnel envtest suite. No new race surface.go build ./...,go vet ./..., andgo test ./...pass locally (Go 1.26). Thetest/envtestsuite skips gracefully whenKUBEBUILDER_ASSETSis unset in this environment; the integration smoke above should run in maintainer CI where the kube binaries are available. The unit tests fully cover the value/defaulting logic without envtest.Closes #134
🤖 Generated with Claude Code