feat(cluster): expose broker autoscaling on streamnative_pulsar_cluster

[david-streamlio]

Fixes #158

Motivation

streamnative_pulsar_cluster exposes static capacity only — broker_replicas, compute_unit_per_broker, bookie_replicas, storage_unit_per_bookie. There is no way to declare broker autoscaling, even though the platform has supported it for as long as the pinned cloud-api-server has, and it is settable through the Cloud Console, the cloud API, and snctl.

Because the provider does a read-modify-write on update, a policy set out of band is preserved across a normal terraform apply, so this is not a drift-clobbering bug. The gap is reproducibility: a cluster destroyed and recreated from Terraform comes back without autoscaling, silently, and there is no way to declare it. That matters for teams whose stated workflow is rebuilding clusters cleanly from IaC.

Modifications

Add an optional broker_auto_scaling_policy block with min_replicas and max_replicas, expanded onto Spec.Broker.AutoScalingPolicy on create and update, and read back on refresh.

resource "streamnative_pulsar_cluster" "example" {
  # ...
  broker_auto_scaling_policy {
    min_replicas = 3
    max_replicas = 8
  }
}

min_replicas is Computed. The control plane picks a floor when one is not given, and an omitted value arrives from the SDK as 0; sending that literally would ask for a floor of no brokers. It is left nil instead, and a server-chosen value does not read as drift.

Two failure modes are refused rather than applied silently:

• Serverless instances. The control plane pins the policy for serverless clusters and rejects later changes to it, so a configured block could never be honoured. Create and update refuse it outright, alongside the existing serverless guards on compute_unit and broker_replicas.
• Organizations without the autoscaling feature. The API rejects the field when the feature gate is off. Following the maintenance_window precedent from feat(cloud): add check for maintenance window org-level feature #151, create and update send a dry run first and fail when the policy that comes back is not the one that went in — turning "brokers silently never scale" into a clear error.

broker_replicas gains no diff suppression, deliberately. #158 originally proposed a DiffSuppressFunc or treating the two as mutually exclusive; neither is needed. The controller stops passing a replica count once a policy is set and lets the HPA own it, and the provider never calls d.Set("broker_replicas", ...) during read — it is write-only from configuration, so there is no drift to suppress. That it stops taking effect is documented on the field instead.

Out of scope: bookkeeper autoscaling, per the cloud team. spec.broker.scheduledScaling likewise — it does not exist in the pinned cloud-api-server and would need a dependency bump.

Verifying this change

• go build, go vet, gofmt, and the full unit suite pass.
• Ten new unit tests covering expand, flatten, equality, and the accepted / dropped / overridden validation paths — including the server-defaulted min_replicas case and the serverless override.
• Docs regenerated with tfplugindocs (make docs); the diff is limited to this attribute, with no generator churn elsewhere.

No acceptance test is included. The maintenance_window acceptance tests need TF_ACC and a live organization, and autoscaling additionally needs the org feature gate enabled, so one cannot be verified outside CI. Happy to add one guarded the same way if you would like it in this PR.

Note for reviewers

main does not currently build from a clean Go module cache — a transitive dependency resolves to a revision that no longer exists. It reproduces on unmodified main and is unrelated to this change; it is tracked internally, and cloud-api-server master has already moved past it. I worked around it locally to type-check; go.mod and go.sum are unchanged in this PR.

Worth knowing because the Tests workflow last ran on 2026-06-18 and has not run since, so this PR is likely the first to hit it in CI.

[david-streamlio]

@freeznet — tagging you directly rather than leaving this on the team request, since CODEOWNERS resolves this repo to @streamnative/serverless and you are one of its two members, so your approval is what actually unblocks this one.

Worth separating from the terraform-provider-pulsar PRs you looked at this morning: those are owned by @streamnative/cloud-data-plane, which you are not a member of, so they need a different reviewer. This one you can land.

No urgency if you are mid-something — just making sure it is in front of someone who can merge it rather than sitting on a team-level request.

[freeznet]

Requesting changes for three correctness issues: an omitted minimum disables the downstream HPA, refresh introduces destructive/perpetual drift for unmanaged and serverless policies, and the bounds are not validated as a pair. Details are inline.

[freeznet]

[P1] Do not allow min_replicas to remain unset. At the pinned cloud-api-server commit, broker.go copies a nil minimum into the downstream PulsarBroker; sn-operator v0.18.0-rc.16 then returns a nil policy from GetAutoScalingPolicyWithDefault whenever MinReplicas is nil, so no HPA is built. A configuration containing only max_replicas therefore passes this provider and the cloud API but does not enable autoscaling. Please make min_replicas required, or materialize an explicit and documented minimum before sending the request, and cover the omitted case through the downstream conversion.

[david-streamlio]

Agreed, fixed in 90d417b. min_replicas is now Required — I dropped Optional/Computed rather than materializing a default, since inventing a floor on the practitioner's behalf seemed worse than making them state one.

I confirmed the downstream behaviour in the pinned operator before changing it:

// sn-operator@v0.18.0-rc.16/pkg/hpa/builder.go:74
func GetAutoScalingPolicyWithDefault(policy *v1alpha1.AutoScalingPolicy, ...) {
	if policy == nil || policy.MinReplicas == nil {
		return nil, nil
	}

So a nil minimum builds no HPA at all, exactly as you said.

Two follow-on changes fell out of it. expandBrokerAutoScalingPolicy no longer drops a zero minimum on the floor — the minReplicas > 0 condition is gone, since the validator already bounds the field at 1..15. And brokerAutoScalingPolicyEqual no longer treats a nil expected minimum as "whatever the control plane filled in": that leniency is what let a request with no minimum pass the dry-run acceptance check. There was an existing test asserting that leniency, which I replaced — it was encoding the bug.

[freeznet]

[P1] Avoid importing an unmanaged/serverless policy into an Optional-only block. An API value stored here becomes Terraform state, and when the block is absent from configuration the SDK plans broker_auto_scaling_policy.# from 1 to 0 (reproduced with Resource.SimpleDiff). For a serverless cluster, admission injects the 2..3 policy, this read creates that removal diff, and the update guard below rejects every attempt to apply it. For a dedicated cluster whose policy was set out of band, the same plan silently disables autoscaling, contradicting the preservation described in this PR. Please preserve unmanaged state (for example, only flatten a policy Terraform was already managing and skip the serverless-injected policy), or redesign the schema semantics, and add refresh/plan regression tests.

[david-streamlio]

Agreed, fixed in 90d417b. Refresh now surfaces a policy only when Terraform is already managing one and the instance is not serverless:

func brokerAutoScalingPolicyForState(
	policy *cloudv1alpha1.AutoScalingPolicy, managed bool, serverless bool,
) []interface{} {
	if policy == nil || !managed || serverless {
		return []interface{}{}
	}
	return flattenBrokerAutoScalingPolicy(policy)
}

A nil policy still clears state, so genuine removal drift on a managed policy is detected — it is only the unmanaged and serverless-injected cases that no longer enter state.

Covered by TestBrokerAutoScalingPolicyForState for all four combinations, plus TestBrokerAutoScalingPolicyUnmanagedProducesNoPlan, which uses Resource.SimpleDiff the way you did to assert no broker_auto_scaling_policy.# change is planned.

[freeznet]

[P2] Validate the relationship between the bounds. The two scalar validators independently accept values such as min_replicas = 10 and max_replicas = 5. The pinned cloud-api admission has no cross-field validation and passes both values downstream; Kubernetes rejects the resulting HPA because maxReplicas must be at least minReplicas, leaving this as a late reconciliation failure instead of a Terraform diagnostic. Please add resource-level/CustomizeDiff validation for min_replicas <= max_replicas and a regression test.

[david-streamlio]

Agreed, fixed in 90d417b. validateBrokerAutoScalingPolicyBounds runs at the top of CustomizeDiff, deliberately ahead of the three early returns already in that closure so the check applies on create and update alike rather than only on the paths that fall through.

It is a plain function over the policy list rather than taking the ResourceDiff, which keeps it directly unit-testable. Covered by TestValidateBrokerAutoScalingPolicyBounds for the inverted, equal, ascending, absent and nil-entry cases, and by TestBrokerAutoScalingPolicyBoundsRejectedAtPlan, which drives it through the resource's own SimpleDiff so the wiring is pinned too — not just the helper.

[david-streamlio]

@freeznet all three addressed in 90d417b — replies inline. Thanks for the detail, particularly reproducing the removal diff with SimpleDiff; that made the second one much easier to fix correctly rather than approximately.

Summary:

• P1 min_replicas — now Required. Confirmed GetAutoScalingPolicyWithDefault returns a nil policy on a nil minimum, so an omitted value really did build no HPA.
• P1 unmanaged/serverless refresh — brokerAutoScalingPolicyForState surfaces only a managed, non-serverless policy; removal drift on a managed policy is still detected.
• P2 bounds — validated as a pair at the top of CustomizeDiff, so it applies on create and update.

go build ./..., go vet ./cloud/ and go test ./cloud/ all pass.

One caveat on the verification: this repo does not build here as-is, because cloud-api-server requires unified-rbac/sdk/sdk-go v1.13.2-rc2 and that tag no longer exists upstream (streamnative/cloud-api-server#4860). I compiled and tested behind a temporary replace to v1.13.2, then restored go.mod/go.sum — the changeset is three files and does not touch dependencies. So the results are real, but not against the exact dependency graph CI will resolve. Worth knowing in case CI cannot resolve that tag either.

[freeznet]

Re-reviewed 90d417b. The three prior correctness findings are addressed: min_replicas is required, refresh preserves unmanaged/serverless policy semantics while retaining managed drift detection, and inverted bounds are rejected during planning. I also re-ran go test ./..., go vet ./..., go build ./..., gofmt/diff checks; all passed. No new blocking findings.

[david-streamlio]

@freeznet gentle nudge on this one — it has been approved and CLEAN since 29 Aug and I do not have merge rights on this repo, so it needs you to land it.

Nothing is outstanding that I am aware of: the three threads you raised were addressed in 90d417b on 26 Aug, ahead of your approval. They are still showing as unresolved, but I think that is just the resolve button rather than anything open — happy to resolve them myself if you would prefer that before merging, or to rebase if it has gone stale.

Context on why it matters: this is the min_replicas / max_replicas half of broker autoscaling. The follow-on (#160, the HPA metric target and scaling behaviour) is blocked upstream on the cloud-api-server AutoScalingPolicy type, so this PR is the part that can actually ship today.