feat: Kafka, Backup and DR, APIM and CDN control planes; ECS tag, VPC endpoint and PSC fidelity fixes

[aryanmehrotra]

These are the control planes and fidelity fixes that zopnight's resource-plane families need to run their lifecycle tests against real AWS, GCP and Azure SDKs.

Control planes

• GCP Managed Service for Apache Kafka: managedkafka.googleapis.com v1
• GCP Backup and DR backup vaults: backupdr.googleapis.com v1
• Azure API Management service: Microsoft.ApiManagement/service
• GCP Cloud CDN backendBuckets, on the compute load-balancing handler
• GCP compute addresses / globalAddresses setLabels, with labelFingerprint and label list filters

Fidelity fixes

• ECS: Describe* with include=TAGS now reflects TagResource / UntagResource. The ARN-keyed tag store is the single authority, so the describe calls and ListTagsForResource no longer disagree after the first tag write.
• EC2: VPC endpoint groupSet items carry <groupId>, and <groupName> when the group exists. The bare-string form decoded as an empty GroupId in the AWS SDK. One dangling group id no longer strips the names of the others.
• GCP LB: Private Service Connect consumer forwarding rules:
  • network and subnetwork are stored and returned
  • a PSC rule sent without a scheme reports none, instead of a synthesized EXTERNAL
  • pscConnectionStatus and a stable pscConnectionId are returned
  • a Google APIs bundle target (all-apis / vpc-sc) on a regional rule is refused, as GCP does

Verification (this branch's tip, rebased on development @ dffcf737)

Gate	Result
go build ./... + go vet ./...	✅
go test -count=1 ./...	✅ 501 packages ok. 4 cmd/cloudemu crash/persist tests timed out under a parallel full run; they pass in isolation on both this tip and development (load flake, not this diff)
golangci-lint, --new-from-rev origin/development	✅ 0 issues
zopnight private-endpoint + container-service lifecycles through real SDKs	✅ 95/95 and 75/75 (together with #1335)

Related: #1335 (EC2 tags on Elastic IPs and VPC endpoints) is still needed for the AWS private-endpoint lifecycle.

[NitinKumar004]

Reviewed each area against the real APIs, and ran each one on cloudemu serve with the official SDKs, the CLIs and Terraform.

Verdict: changes requested. Four things block this:

• Managed Kafka's cluster list takes over GKE and AlloyDB lists in the same region.
• Managed Kafka rejects the numeric enums that the official Go client sends.
• Waiting on a Backup and DR vault delete fails in the official Go client.
• APIM says it works with Terraform, but azurerm_api_management cannot be created.

The ECS tag fix, the VPC endpoint fix and the CDN backend bucket work are correct. Every area passes build, vet, race tests, lint and the coverage doc check.

GCP Managed Kafka

High

• server/gcp/managedkafka/handler.go:166-169: claiming the list "when Kafka owns at least one cluster" takes over GKE's (and AlloyDB's) GET /v1/projects/{p}/locations/{l}/clusters. Reproduced on serve: GET returned gke1, then after creating Kafka cluster k9 the same GET returned only .../clusters/k9 and gke1 was gone. That breaks gcloud container clusters list --region, GKE SDK lists and AlloyDB lists whenever the services share a region. The path and shape are identical, so the body can't tell them apart. Options: probe GKE/AlloyDB ownership (wired like SetBucketLister) and claim the list only when Kafka owns clusters there and the siblings own none, or merge the lists by owner. TestFullServerManagedKafkaSharesClustersWithGKE (server/gcp/operations_ownership_test.go:166) checks the GKE list only before any Kafka cluster exists, so it passes on the broken code. It should also assert the GKE list after a Kafka create.
• server/gcp/managedkafka/wire.go:66-67: rebalanceConfig.mode is decoded only as a string. The official cloud.google.com/go/managedkafka v1.2.0 REST client sends enums as numbers (UseEnumNumbers), so CreateCluster with RebalanceConfig{Mode: AUTO_REBALANCE_ON_SCALE_UP} fails with 400 malformed JSON body: cannot unmarshal number into ... rebalanceConfig.mode (same with curl "mode":2). Proto3 JSON allows both forms. Accept a number or a name as server/gcp/privateca/enums.go normalizeEnumNumbers does, and add a gapic-client test. The discovery-client tests send names and pass either way.

Medium

• handler.go:160-164: when a Kafka cluster is missing, item requests fall through to GKE. PATCH .../clusters/k1?updateMask=labels after a delete returns GKE's 405 METHOD_NOT_ALLOWED instead of 404 NOT_FOUND, and GET and DELETE return GKE's cluster "k1" not found in "us-central1" rather than the Kafka error. A PATCH whose body is Kafka-shaped (capacityConfig, gcpConfig, rebalanceConfig, or labels with no cluster wrapper) should be claimed.
• CreateCluster succeeds with an id a GKE cluster already uses in that location. From then on Kafka claims every item route for that id and the GKE cluster can't be reached. This is the same missing cross-owner check as the first High.

Low

• providers/gcp/managedkafka/managedkafka.go:212 and operations.go:188: the library GetOperation reports any unknown name as done, where the API returns 404. The standalone poll also returns done:true with no response. The driver's operation store grows without bound, and in the assembled server the shared lro.Registry owns polls, so this store is unused there.
• Outputs the API returns are missing:
  • kafkaVersion, which defaults to 3.7.x. It is also listed in clusterFixedPaths though the API marks it an optional input.
  • LRO metadata (OperationMetadata).
  • The rebalanceConfig.mode default of NO_REBALANCE.
• tlsConfig, updateOptions and brokerCapacityConfig are silently dropped.
• --async-settle isn't honoured: a new cluster is ACTIVE at once and never CREATING.
• The subnet's region isn't checked against the cluster location, which the API requires.
• Coverage is below the 90% target: provider 80.1%, wire 78.8%.
• anyWithType, formatTime and decodeBody duplicate helpers in sibling packages. decodeBody differs from gcprest.DecodeJSON only in accepting an empty body.

Matches the API:

• Nesting and int64-as-string capacity fields.
• vcpu ≥ 3 and the 1:1 to 1:8 GiB ratio.
• Subnet format with 1 to 10 entries.
• A required updateMask, and rejection of immutable or output-only paths in snake and camel case.
• partitionCount can only grow and replicationFactor is immutable.
• Topic calls are synchronous, and topic delete returns {}.
• A topic's parent must exist, and cluster delete cascades to its topics.
• Errors use the google.rpc.Status shape.

Checked and fine: build, vet, race tests, lint and no coverage doc drift. On serve, curl covered create, poll, get, list, patch, topic CRUD and the error cases. The real cloud.google.com/go/managedkafka REST client passes create (Wait), get, list, update (Wait), topic create/list/delete, delete (Wait) and a 404 get-after-delete, as long as no rebalanceConfig is sent. Terraform hashicorp/google 8.4.0 google_managed_kafka_cluster and google_managed_kafka_topic apply, show a clean plan, update in place (capacityConfig,labels and partitionCount), plan clean again and destroy.

GCP Backup and DR

Scope. Only backupVaults is implemented (create, get, list, patch, delete, plus the LRO). managementServers, backupPlans, backupPlanAssociations, dataSources and backups are not. /backupPlans requests currently fall through to other handlers (see M3).

High

• H1 server/gcp/backupdr/operations.go:165 (and the standalone poll at :185): the delete LRO returns {"done":true} with no response. The cloud.google.com/go/backupdr/apiv1 REST client's DeleteBackupVaultOperation.Wait fails with unsupported result type <nil> (reproduced against serve). The shared poller replays the same shape because the handler registers nil. Return response: {"@type":"type.googleapis.com/google.protobuf.Empty"} as server/gcp/managedkafka/wire.go:18 does, and register that value. The tests only use the discovery client (google.golang.org/api/backupdr/v1), which ignores response, so they pass on the broken code. A delete-and-Wait test with the gapic client would catch it.

Medium

• M1 providers/gcp/backupdr/validate.go:103: the vault ID check is length only. Bad_ID and a/bc are accepted. a/bc is stored as .../backupVaults/a/bc, shows up in list, but is unreachable by get or delete (the path falls through to Firestore and returns 404), leaving an orphan. Enforce ^[a-z][a-z0-9-]{2,62}$.
• M2 validate.go:213 (applyMask) and :154: retention lock is not modelled. On a vault whose effectiveTime has passed (locked), PATCH lowered backupMinimumEnforcedRetentionDuration from 86400s to 1s and moved effectiveTime, and both succeeded. On GCP a locked vault's retention can only increase and the lock time cannot change. checkRetention (:123) also accepts 0s and 1s; the documented range is 1 day to 99 years.
• M3 Parent/child rules: only force versus backupCount is modelled. A vault referenced by a backup plan deletes without an ignoreBackupPlanReferences check. Reproduced with TF google_backup_dr_backup_plan: the plan request was served by the existing gkebackup handler (it answered with a google.cloud.gkebackup.v1.BackupPlan type), then a plain vault DELETE succeeded. The misroute predates this PR, but it means Backup and DR plans silently land in GKE Backup. Worth a ticket or a note in the handler docs.
• M4 providers/gcp/backupdr/backupdr.go:262: SetUsage is a production symbol used only by tests (backupdr_test.go, sdk_roundtrip_test.go), and its doc comment says so. Move it to export_test.go or a test seam.

Low

• L1 operations.go:75: filter and orderBy are silently ignored, so a filtered list returns every vault.
• L2 wire.go writeErr: errors[].reason is upper-case (ABORTED, FAILED_PRECONDITION) where the rest are lower-case, and the etag error drops the vault name because cerrors.Message strips the %w prefix.
• L3 A validateOnly create or delete still records and registers an operation.

Checked and fine: behaviour lives in the provider, RWMutex plus deep clones, Snapshottable round-trips including opSeq, dispatch yields operation polls to the shared poller. Build, vet, race tests and lint are clean and docs/coverage/gcp/backupdr.md has no drift. serve e2e (curl and the gapic REST client) covered create, get, list, patch, LRO poll, 404, duplicate and stale-etag 409, missing-mask 400, allowMissing and validateOnly. Terraform hashicorp/google 8.4.0 google_backup_dr_backup_vault apply, a clean plan, an in-place update, and destroy all pass.

GCP CDN, load balancing, PSC and dependencies

The code that is here is correct and does not change existing GCP routing. The gaps below are operations real users reach for in this area, not regressions.

High (missing operations)

• server/gcp/loadbalancer/backendbuckets.go:132: backendBuckets.addSignedUrlKey and deleteSignedUrlKey return 501, so Terraform google_compute_backend_bucket_signed_url_key fails with "Error 501: backendBuckets.addSignedUrlKey is not implemented", and cdnPolicy.signedUrlKeyNames is never populated. Both actions need adding (take keyName and keyValue, list the name under cdnPolicy.signedUrlKeyNames on GET, never echo the value), plus the same pair on backendServices, which returns 405 today.
• POST .../urlMaps/{m}/invalidateCache returns 405. serviceAttachments does not exist at all: POST .../regions/r/serviceAttachments returns 501 and google_compute_service_attachment fails. So consumerAcceptLists, connectionPreference, connectionLimit and connectedEndpoints are not modelled.

Medium

• forwardingrules_psc.go:28 and :58: every PSC consumer rule reports pscConnectionStatus: ACCEPTED, even for a target like .../serviceAttachments/nosuch. GCP returns PENDING or REJECTED under ACCEPT_MANUAL or the accept/reject lists. Once service attachments exist, the status should come from the producer's lists and limits.
• refvalidation.go:260 and operations.go:1041: a PSC rule with no network is accepted and gets an external 34.x IPAddress. GCP requires the consumer network and an internal address. An explicit loadBalancingScheme: EXTERNAL with a service attachment target is also accepted, but PSC requires an empty scheme. Both should be rejected in validatePSCTarget.
• backendbuckets_validate.go:137: the cdnPolicy cross-field rules are missing. These are all accepted but rejected by GCP: USE_ORIGIN_HEADERS with defaultTtl, maxTtl or clientTtl; FORCE_CACHE_ALL with maxTtl; and clientTtl > maxTtl (maxTtl 100 with clientTtl 5000 inserts fine). defaultTtl is checked only against an explicit maxTtl, not the 86400 default. applyBackendBucketDefaults (line 79) fills in cacheMode but not the documented defaultTtl, maxTtl and clientTtl of 3600, 86400 and 3600.

Low

• operations.go:937 numericID (pre-existing): this returns the full uint64 FNV hash, which can be around 1.6e19 and so above int64. Terraform google_compute_forwarding_rule fails on read with "forwarding_rule_id: expected type 'int', got unconvertible type 'string', value: '16018893947941779668'". It affects every forwarding rule, so the PSC flow cannot complete in Terraform. Masking to 63 bits fixes it.
• server/gcp/vpc/address_labels.go: the labels and labelFingerprint live in the server-side addressStore. They are not in the provider and not in /_cloudemu/snapshot, so they are lost on restore, and they diverge between the library and serve. They belong in the provider.
• The cdnPolicy validation and the url-map in-use delete check sit in the handler, over an opaque driver store (providers/gcp/loadbalancer/backend_buckets.go:50). That matches the existing urlMaps and healthChecks code, but the in-use check at backendbuckets.go:303 races a concurrent url-map insert, as backendServices does.

Dependencies: the only go.mod change is a new armapimanagement/v3 v3.0.0 (MIT). go mod why shows it reached only from server/azure/apimanagement tests. It is not in go list -deps ./cmd/cloudemu, and go mod tidy is clean.

Checked and fine: build, vet, race tests on the loadbalancer, vpc, compute and persist packages, lint, and no coverage doc drift. On serve these work: backend bucket insert, get and patch (cdnPolicy merge), TTL rejection, the url-map dangling-reference 400, the in-use delete 400, snapshot persistence of backend buckets, setLabels 412 and 404, and label filters. Terraform google_compute_backend_bucket with cdn_policy applied, updated in place, gave a clean plan and destroyed.

Azure API Management

Only Microsoft.ApiManagement/service is implemented. apis, operations, products, subscriptions, backends, policies and named values return 404 InvalidResourceType (server/azure/apimanagement/handler.go:80-84), so the service, api, operation and product lifecycle and If-Match cannot be exercised yet.

High

• providers/azure/apimanagement/apimanagement.go:14 says azurerm_api_management sees no drift, but Terraform can't create one. With azurerm v4, apply fails on GET /providers/Microsoft.ApiManagement/locations/eastus/deletedServices/<name> → 501. The provider makes this soft-delete check even with recover_soft_deleted = false. Returning 404 there only moves the failure on: create then fails on GET .../service/<n>/apis, because azurerm lists and deletes the default Echo API after create, and refresh and destroy fail on portalsettings/signin. The minimum needed is deletedServices GET/DELETE (404 when nothing is soft-deleted), the apis list, portalsettings signin/signup, tenant/access and the service policy. Otherwise drop the Terraform claim and document the gap.

Medium

• apimanagement.go:268: the etag is set once and never changes, and PUT and PATCH return the same value. Optimistic concurrency doesn't work, and the child resources will need If-Match on top of it. Rotate the etag on every write.

• validate.go:94: capacity has no upper bound. Developer with capacity 5 returns 201. The Azure limits are Developer 1, Basic 2, Standard 4, Premium 12 and BasicV2/StandardV2 10. Zones are also accepted on tiers other than Premium.

• apimanagement.go:183: the service name is a global DNS label but isn't treated as one. svc1 is created in both rg1 and rg2 with the same svc1.azure-api.net gatewayUrl. Azure returns 409 on the second one. checkNameAvailability returns 501.

• A PUT with a different location returns 200 and silently keeps eastus. ARM rejects this with 409 InvalidResourceLocation.

• Architecture: server/azure/apimanagement/types.go:141-200 builds the defaults and computed properties in the server layer:

  • virtualNetworkType
  • publicNetworkAccess
  • notificationSenderEmail
  • platformVersion
  • the regional gateway URL
  • the Consumption endpoint rules

  The Go library therefore returns a different resource than serve. These belong in the provider.

Low

• apimanagement_discovery.go:32 puts sku and skuCapacity in Properties instead of Attrs.SKU, SKUCapacity and Zones. The Resource Graph row has no top-level sku{name,capacity} and shows properties.sku as a string.
• Validation errors use InvalidParameter where APIM returns ValidationError. List responses have no nextLink paging.

AWS ECS tags and EC2 VPC endpoints

The ECS tag fix and the VPC endpoint groupSet fix are correct. groupSet items now carry groupId plus the resolved groupName (EC2 SecurityGroupIdentifier), and a dangling group still renders with only its id.

Medium

• providers/aws/ecs/services.go:336 (pre-existing, but it undercuts this fix): serviceTaskSpec gives tasks the service's create-time svc.Tags and ignores propagateTags. After untagging s and tagging s2, a --force-new-deployment still launched tasks tagged s=1, and a service with the default propagateTags NONE still put its tags on its tasks. With NONE the tasks should get no tags. SERVICE should use m.liveTags(svc.ARN, …), and TASK_DEFINITION should use the task definition's live tags.
• The fix covers clusters, services, tasks and task definitions only:
  • Container instances: RegisterContainerInstance --tags ci=1 is dropped, and DescribeContainerInstances --include TAGS returns null even after a TagResource.
  • Capacity providers: Create/DescribeCapacityProviders return "unknown ECS operation", yet TagResource on capacity-provider/FARGATE succeeds.

Low (pre-existing)

• server/aws/ecs/taskdefs.go:153 and DescribeServices return tags without include=TAGS. ECS only returns them when asked.
• tags.go:23 TagResource accepts a nonexistent ARN, 51 tags and aws:-prefixed keys. ECS rejects all three with InvalidParameterException.
• Old-format service ARNs (service/s1) don't resolve.

No file overlap with #1335. It touches server/aws/ec2/tags.go and providers/aws/vpc/tags.go, and this PR touches endpoint.go and operations.go. They complement each other: on this branch ec2 create-tags on a VPC endpoint still fails with InvalidID.NotFound, which #1335 fixes.

Checked and fine: build, vet, race tests on 11 packages including persist, lint, and no coverage doc drift. APIM over curl: create, get, PATCH tags (replace), list, the ARG query, and idempotent delete (200 then 204). ECS tag and untag on each resource type with the aws CLI. aws_ecs_cluster, aws_ecs_task_definition and aws_ecs_service: apply, a clean plan, a tag change, a clean plan again, and destroy.

[aryanmehrotra]

Thanks for the thorough review, and for reproducing everything on serve. All four blockers are fixed in b0ebd3d3..db12f2ff, and so are most of the medium and low findings. Where I went a different way than the review suggested, or left something out, I say so below.

Every fix has a test that fails on b0ebd3d3 and passes now. Where you pointed out that the discovery client hid a bug, the new test goes through the official gapic / SDK client.

Blockers

Finding	Fix	Proof
Kafka list takes over GKE/AlloyDB /clusters	aca0460: the handler gets a read-only ClusterSibling probe, wired the way SetBucketLister is. Kafka claims the list only when it owns clusters there and the sibling owns none. I didn't merge the lists, because a GKE ListClustersResponse client would read Kafka entries as broken GKE clusters.	TestFullServerManagedKafkaSharesClustersWithGKE now checks the GKE list after a Kafka create; also …WithAlloyDB
Kafka rejects numeric enums	aca0460: every enum on the wire accepts a number or a name	TestGAPICClusterWithNumericEnums (the cloud.google.com/go/managedkafka REST client, AUTO_REBALANCE_ON_SCALE_UP, Wait)
Backup and DR delete + Wait fails	4686c13: the delete LRO, the standalone poll and the registered value all return google.protobuf.Empty	TestGAPICDeleteWaitSharedPoller, …Standalone (cloud.google.com/go/backupdr/apiv1)
APIM + Terraform	db12f2f: serves deletedServices (soft delete, purge, restore), the apis list/get/delete with the default Echo API, portalsettings signin/signup, tenant/access and the service policy	TestSDKTerraformCreateReadDestroy, which replays azurerm v4's request sequence (api_management_resource.go @ 3cfd078) through armapimanagement v3

⚠️ APIM: I could not run a real terraform apply here, because the emulator's TLS cert can't be trusted from Go on macOS. So the package doc now claims only what the replayed sequence proves, and "no drift" is gone. An empty plan against a real terraform binary is still unproven, and docs/coverage/nongoals/apimanagement.md says so.

Managed Kafka (aca0460, 956451e)

• Missing-cluster fall-through: a Kafka-shaped PATCH now gets 404, not GKE's 405, and a Kafka create with an id GKE or AlloyDB already uses gets 409 (both directions).
  • Partial: GET/DELETE of a cluster nobody owns still gets GKE's 404. It has the same google.rpc.Status shape but GKE's message. A bodiless request carries no hint of which service it is for, and claiming it would give GKE clients Kafka's error.
• Operations: GetOperation 404s an unknown name, and a finished poll carries response and metadata. The library's op store is capped at 1000 entries; the server polls through lro.Registry.
• Outputs and validation:
  • kafkaVersion defaults to 3.7.x and is optional input.
  • rebalanceConfig.mode defaults to NO_REBALANCE.
  • tlsConfig, updateOptions and brokerCapacityConfig are stored.
  • --async-settle gives CREATING then ACTIVE.
  • The subnet region is checked against the cluster location.
• Coverage: provider 97.6%, wire 97.5%. TypedAny, FormatTime and DecodeOptionalJSON are now shared in gcprest.
• The test dependency is managedkafka v1.0.0, not v1.2.0, because v1.2.0 needs go 1.26 and CI is on 1.25. It encodes enums the same way.
• ⚠️ kafkaVersion: the discovery doc this repo pins (v0.291.0) says output-only, and v0.298.0 says optional with default 3.7.x. I went with the newer one, as you suggested.

Backup and DR (4686c13)

• M1: the id format follows the backup vault naming doc: ^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$. A leading digit is allowed and a trailing hyphen is refused, which differs slightly from the regex in the review.
• M2: the retention lock is modelled. Once effectiveTime has passed, the minimum retention can only grow and effectiveTime is frozen. The range is 1 day to 99 years.
  • The error code for a locked vault isn't documented. I used FAILED_PRECONDITION.
• M3: the /backupPlans misroute into gkebackup predates this PR, so I left it. The handler docs now list what's unimplemented and where /backupPlans goes today. Plans and ignoreBackupPlanReferences need that routing fixed first, so they're a follow-up.
• M4: SetUsage moved to export_test.go.
• L1: filter and orderBy work, following artifactregistry/memorystore precedent. Unlike those siblings, an unsupported expression here gets 400 rather than matching everything.
• L2: reasons are lower-case, and the etag error keeps the vault name.
• L3: validateOnly records no operation.
• Also: gcprest now maps aborted to ABORTED, which also fixes kms, where the status was missing.

CDN, LB and PSC (9f7b22e … cefe314)

• Missing operations:
  • signed URL keys on backendBuckets and backendServices: validated, never stored, never echoed; names go in cdnPolicy.signedUrlKeyNames
  • urlMaps.invalidateCache
  • regional serviceAttachments, with consumerAcceptLists / RejectLists, connectionPreference, connectionLimit and connectedEndpoints
• PSC:
  • pscConnectionStatus now comes from the attachment: ACCEPTED, PENDING, REJECTED, or CLOSED once the attachment is deleted.
  • A rule targeting an attachment that doesn't exist is refused.
  • A rule with no network, or with any explicit scheme, is refused.
  • A rule sent without an address gets an internal one.
• cdnPolicy: the cross-field TTL rules are enforced, and the defaults are filled in per cacheMode (3600 / 86400 / 3600 for CACHE_ALL_STATIC).
• numericID: masked to 63 bits. ⚠️ The same defect is in server/gcp/compute/instances.go and server/gcp/vpc/handler.go, which are separate copies I haven't changed. Happy to fold them into a follow-up.
• Address labels: the whole address record now lives in the provider behind GCPAddressStore, because the addresses themselves were missing from the snapshot too. Snapshots taken before this change restore with no addresses.
• Not fixed: the url-map in-use delete race. It's the same pattern backendServices already has, so it's better fixed for both together.

APIM (db12f2f)

• The etag rotates on every write, and a stale If-Match gets 412 on the service and its children.
• Capacity ceilings per SKU, and zones only on Premium.
• The service name is global: 409 across resource groups and subscriptions, a soft-deleted name also blocks it, and checkNameAvailability is served.
• A location change gets 409 InvalidResourceLocation.
• Defaults and computed fields moved into the provider, so the library and serve return the same resource.
• Discovery sets Attrs.SKU, SKUCapacity and Zones.
• Validation errors use ValidationError, and lists page with nextLink through a new shared azurearm.Paginate.
• Two error codes are best guesses: ServiceAlreadyExists and ServiceAlreadyExistsInSoftDeletedState.

ECS (5996269, 676e000)

• propagateTags: NONE (the default) propagates nothing, SERVICE uses the service's live tags, and TASK_DEFINITION uses the task definition's.
• Tags on more resources: container-instance tags are stored and returned. Capacity providers get Create/Describe/Update/Delete, with FARGATE and FARGATE_SPOT built in.
• include=TAGS: DescribeServices, DescribeTaskDefinition and DescribeTasks return tags only when asked.
• TagResource validation: a nonexistent ARN, more than 50 tags, and aws:-prefixed keys or values are all refused.
• Two places where AWS differs from the review:
  • The TagResource reference says tagging a service by its short ARN is an InvalidParameterException, so TagResource/Untag/List now refuse it. Describe and Update already resolve short ARNs, and a test pins that.
  • The docs say the predefined FARGATE / FARGATE_SPOT capacity providers cannot be tagged, so TagResource on capacity-provider/FARGATE now fails. Before, it succeeded, which was a bug.
• ⚠️ Which exception fires for a nonexistent ARN isn't documented. I used InvalidParameterException.

Gates on db12f2ff

• go build / go vet: clean
• go mod tidy: no drift
• go generate coverage docs and compatgen: no drift
• golangci-lint --new-from-rev b0ebd3d3: 0 issues
• go test ./...: 501 packages ok. TestK8sDataPlanePersistsAcrossCrash timed out once in the full parallel run and passes alone on this tip (5.8s). The same 4 crash/persist tests are flaky under load on development too.
• zopnight's resource-plane suites through the real AWS/GCP/Azure SDKs against this branch plus fix(aws-ec2): tag Elastic IPs and VPC endpoints after creation #1335: private-endpoint 95/95, container-service 75/75

[NitinKumar004]

All four blockers are fixed, and I reproduced each fix on serve. Approving. What's left below is small or already existed before this PR.

Earlier findings

GCP Managed Kafka

• Kafka list taking over the GKE list: fixed. With GKE gke1 and then Kafka k9 in the same region, GET .../locations/us-central1/clusters still returns only gke1. In a region with no GKE cluster, the gapic ListClusters returns the Kafka cluster.
• Numeric enums: fixed. cloud.google.com/go/managedkafka CreateCluster with AUTO_REBALANCE_ON_SCALE_UP completes through Wait and reads the mode back. "mode":2 works, and "mode":7 gets a 400.
• PATCH of a missing cluster falling through to GKE: fixed. GET and DELETE of an id nobody owns still get GKE's 404. The reason you gave for that is fair.
• Create reusing an id another service owns: fixed. It returns 409 in both directions.
• Operations: fixed. An unknown operation gets 404, and create returns metadata with an OperationMetadata @type.
• Missing outputs and dropped fields: fixed.
  • kafkaVersion defaults to 3.7.x.
  • rebalanceConfig.mode defaults to NO_REBALANCE.
  • tlsConfig and updateOptions read back.
  • A subnet in another region is refused.
• Coverage and duplicated helpers: fixed.

GCP Backup and DR

• H1, delete and Wait: fixed. The gapic DeleteBackupVaultOperation.Wait returns nil, and so does a rebuilt handle.

• M1, ID format: fixed. Bad_ID, a/bc and pv get 400.

• M2, retention lock: fixed. Once effectiveTime has passed:

  • a lower minimum retention gets 400;
  • a higher one gets 200;
  • moving effectiveTime gets 400.

  1s is refused.

• M3, backup plans routed to gkebackup: documented in the handler. That's reasonable, since the misroute predates this PR.

• M4: fixed. SetUsage is now only in export_test.go.

• L1 to L3: fixed.

GCP CDN, load balancing and PSC

• Signed URL keys: fixed. google_compute_backend_bucket_signed_url_key applies and destroys.
• invalidateCache: served. A missing url map gets 404.
• serviceAttachments: fixed. google_compute_service_attachment with ACCEPT_MANUAL and an accept list applies. Swapping to a reject list turns the consumer rule and the connectedEndpoints entry to REJECTED.
• PSC rule validation: fixed. A PSC rule with no network, or with loadBalancingScheme: EXTERNAL, gets 400.
• cdnPolicy cross-field rules: fixed.
• numericID: fixed. forwarding_rule_id reads back in Terraform.
• Address labels: fixed. They live in the provider now and survive snapshot, reset and restore.
• The url-map in-use delete race is still open. Fixing it later together with backendServices is fine.

Azure API Management

• Terraform create: fixed at the wire level. I replayed azurerm v4's create, read, update and destroy sequence with curl against serve. That covers the soft-delete check, PUT, removing the sample APIs and products, portal settings, tenant access, the policy GET, listSecrets, the PATCH update, DELETE, and the deletedservices purge. Every call returns the status azurerm expects. Dropping the "no drift" claim until a real apply has run was the right call.
• Etag: fixed. It changes on every write, and a stale If-Match gets 412.
• Capacity ceilings: fixed.
• Global name: fixed. It returns 409 ServiceAlreadyExists, and checkNameAvailability answers.
• Location change: fixed. It returns 409 InvalidResourceLocation.
• Defaults moved into the provider: fixed.
• Paging: fixed.
• Persistence: the service restores with its children.

AWS ECS

• propagateTags: fixed. After a tag change and --force-new-deployment, the new task carries v=2.
• include=TAGS: fixed.
• Container-instance tags: fixed.
• Capacity providers: fixed.
• TagResource validation: fixed. It refuses FARGATE, a missing ARN and an aws: key with InvalidParameterException.

New findings (Low)

• golangci-lint --new-from-rev=origin/development reports gocyclo on server/gcp/managedkafka/handler.go:210 (Matches, 14) and server/gcp/managedkafka/wire.go:212 (toDriverCluster, 11). To fix them, pull the list and item branches of Matches into helpers, and move the optional-block conversion into a helper.
• In a location where the sibling owns clusters, the gapic Kafka ListClusters returns gke1 and none of the Kafka clusters. You documented this, and it's the right default. A Kafka client there gets wrong data rather than an error, though, so please add a line to docs/coverage/nongoals.

These predate this PR and aren't blocking, but they're worth tickets:

• server/gcp/loadbalancer/types.go forwardingRuleResponse has no region, and neither does the regional backend service. As a result, the next terraform plan replaces every regional forwarding rule and the service attachment that points at it. The same rule drops allPorts, and an INTERNAL rule gets an external 34.x address.
• server/gcp/loadbalancer/operations.go:862 validateHealthCheckRefs only looks in the rule's own scope. A regional INTERNAL backend service that points at a global health check is refused, but GCP allows it.

Checked

• go build, go vet and go test -race pass on the 25 touched packages, and on persist, server/gcp/gcs and server/grpc/..., which covers the iam and longrunning bumps. coveragegen shows no drift.
• On serve:
  • the gapic Kafka and Backup and DR clients;
  • curl for every earlier repro;
  • snapshot, reset and restore for addresses, Kafka, vaults and APIM.
• Terraform hashicorp/google 6.50.0: Kafka cluster and topic, backup vault, backend bucket with cdn_policy, signed URL key, service attachment, PSC consumer rule and address labels. Each ran apply, update and destroy. Plans were clean except for the forwarding rules above.
• Terraform hashicorp/aws 6.66.0: aws_ecs_cluster, aws_ecs_task_definition and aws_ecs_service with propagate_tags. Each ran apply, a clean plan, a tag update, a clean plan and destroy.