feat(azure): support disk update and App Service plan update (PATCH) - #1336
aryanmehrotra wants to merge 5 commits into
Conversation
NitinKumar004
left a comment
There was a problem hiding this comment.
Disks PATCH and serverfarms PATCH both route, and BeginUpdate/PollUntilDone completes. The findings below need to be addressed before merge.
High
1. Disk PATCH loses concurrent writes and races with attach/detach. server/azure/disks/update.go:40-58, providers/azure/virtualmachines/vm.go:1158-1184
- The handler builds a full
VolumeConfigfrom aDescribeVolumessnapshot and then callsUpdateVolume. Two concurrent PATCHes (one for size, one for tags) overwrite each other. UpdateVolumedoesGet, mutates the stored*VolumeInfoin place with no lock, then callsSet. That is a data race withdescribeResources. It also writes back a stale pointer over a concurrentAttachVolumecopy-on-write.- Repro:
AttachVolume,UpdateVolumeandDescribeVolumesin parallel, 300 rounds with-race. Result: 10 DATA RACE reports at vm.go:1165/1169/1589/1226, and 14/300 disks lost their attachment (state back to available while the VM still lists the disk). - The plan side already does this correctly with
Store.Update. - Fix: add a provider method such as
PatchVolume(ctx, id, VolumeUpdate)that merges and validates insidem.volumes.Update(...)with copy-on-write, the same wayAttachVolumeworks.
2. Azure's disk rules live only in the wire PATCH handler. update.go:117-152
- The grow-only and IOPS/MBps-SKU checks run only on PATCH. A re-PUT on an existing disk still shrinks it: PUT
diskSizeGB:16on a 128 GB disk returned 202, and GET reported 16. - The Go library path (
UpdateVolume) has no checks at all. - Fix: move the validation into the provider (item 1) so PUT, PATCH, the library and
serveshare it.
Medium
3. Wrong error code for a shrink. update.go:18,122, test at update_test.go:213
- Real Azure returns
InvalidParameter(internalInvalidResizeWithName); see Microsoft Learn, "Troubleshoot Azure disk resize failures", error code table. - The PR returns
BadRequest, and the test pins that code. - Fix: return
cerrors.InvalidArgumentfrom the provider.WriteCErralready maps it to 400InvalidParameter.
4. Attached-disk and SAS restrictions are not enforced.
- Resizing a disk attached to a running VM succeeds. Real Azure returns 409
OperationNotAllowed(internalChangeDiskSizeWhileAttachedNotAllowed): "Cannot resize disk X while it is attached to running VM Y. ... requires the virtual machine to be deallocated". The same applies to OS disks and to SKU changes on attached disks. - With an active SAS (
beginGetAccesssucceeded, tracked indiskAccess), a PATCH grew the disk from 16 to 256 GB. Real Azure returnsChangeDiskSizeWhileActiveSasNotAllowed. - Fix: check
vol.State, the attaching instance's power state anddiskAccessin the provider, and returnOperationNotAllowedwith status 409.
5. The App Service plan OS can be flipped. providers/azure/functions/app_service_plan.go:274-276, PUT at server/azure/functions/handler.go:742
- PATCH
{"properties":{"reserved":false}}on a Linux plan returned 200 withreserved:false. PATCH{"kind":"app"}was also accepted. - A re-PUT without
reservedsilently turns a Linux plan into a Windows one (kind:"linux", reserved:false). - Real Azure rejects this: "You cannot change the OS hosting your app at this time. Please recreate your app with the desired OS." (Azure/bicep#5724). azurerm treats
os_typeas ForceNew. TestPatchAppServicePlan(app_service_plan_test.go:233) asserts that the flip succeeds.- Fix: reject a change to
reserved, or to the Linux/Windowskind, on an existing plan in both the PATCH and PUT paths, and change the test to expect the error.
Low
- Plan SKU/capacity are not validated.
sku.capacity:-5andsku.name:"ZZ9"both return 200, and ZZ9 keeps the stale tier "Standard". Capacity 0 dropscapacityfrom the response. Real Azure rejects capacity below 1, capacity above the SKU maximum, unknown SKUs, and moves between Dynamic/ElasticPremium and dedicated tiers. Settingtierexplicitly next to a newnamecan also produce a mismatched pair. - Disk tier handling.
update.go:107copiessku.tier(read-only,Standard/Premium) into the performance tier.properties.tier:"P50"was accepted on an UltraSSD disk and read back assku.tier:"P50". Performance tiers only apply to Premium_LRS. A SKU change should also clear or re-derive the tier. Converting to or from UltraSSD_LRS is accepted without any check. - Missing tests. Nothing covers attached disks, active SAS, a PUT-after-PATCH shrink, concurrent PATCH, or rejecting a plan OS flip.
- Test helpers.
providers/azure/functions/app_service_plan_test.gouses testify; provider tests use hand-rolled helpers. - Merge conflicts with current
developmentindocs/coverage/README.mdanddocs/coverage/azure/README.md. Rebase and rungo generate ./...again.
Verified
- Build, vet and
go test -racepass forserver/azure/{disks,functions},providers/azure/{functions,virtualmachines},internal/coveragegenandpersist. golangci-lint reports 0 new issues.go run ./internal/coveragegenshows no drift on this branch. The new plan fields round-trip through the plans snapshot. - Tags: a PATCH with tags replaces the whole set and omitted tags are kept, for both disks and plans, matching ARM resource-level semantics.
serveover HTTPS with curl: disk grow returns 202 with Azure-AsyncOperation; a missing disk or wrong resource group returns 404; malformed JSON returns 400InvalidRequestContent; plan sku/capacity/tags PATCH returns 200 and re-derives the tier.- Terraform (azurerm v4.81.0):
azurerm_managed_diskresized 32, 64, then 128 and moved from Standard_LRS to Premium_LRS through PATCH, with "No changes" on the follow-up plan.azurerm_service_planB1 to S1 and worker_count 1 to 3 update through PUT (azurerm does not use PATCH for plans), with "No changes" on the follow-up plan. Destroy is clean.
armcompute DisksClient.BeginUpdate against Microsoft.Compute/disks/{name}
returned 501 NotImplemented. PATCH now applies a DiskUpdate body as a
partial update and answers 202 + Azure-AsyncOperation, like CreateOrUpdate,
so the SDK poller completes and its final GET reads the updated disk.
- tags: replaced wholesale when present (ARM Compute PATCH semantics);
internal bookkeeping tags are kept
- sku / tier: switched in place; moving off UltraSSD_LRS/PremiumV2_LRS
drops provisioned IOPS/MBps
- diskSizeGB: grow only; a shrink is a 400 BadRequest
- diskIOPSReadWrite / diskMBpsReadWrite: only on UltraSSD_LRS and
PremiumV2_LRS, otherwise a 400 InvalidParameter
- the volume is updated in place, so id, uniqueId, timeCreated and any
attachment are preserved, and Resource Graph reflects the new size/tags
armappservice PlansClient.Update against Microsoft.Web/serverfarms/{name}
answered 405 MethodNotAllowed; only PUT, GET and DELETE were routed.
PATCH now merges the body into the stored plan and answers 200 with the
updated plan, as the sites PATCH does:
- kind, sku.name/tier/capacity (a new SKU name re-derives the tier)
- tags, replaced wholesale when present
- properties.reserved, perSiteScaling, zoneRedundant and
maximumElasticWorkerCount, which are now also modeled on PUT and
rendered on every read
A PATCH for a missing plan, or one in another resource group, is a 404.
The provider gains PatchAppServicePlan, a read-modify-write under the
store lock.
…e's resize rules What was wrong: - The PATCH handler built a full VolumeConfig from a DescribeVolumes snapshot and UpdateVolume mutated the stored *VolumeInfo in place with no lock, then Set it back. That raced DescribeVolumes and overwrote a concurrent AttachVolume copy-on-write: under -race, 277 of 300 disks lost their attachment when attach and update ran together. Two concurrent PATCHes (size, tags) also overwrote each other. - Grow-only and the IOPS/MBps-SKU rule lived only in the wire PATCH handler, so a re-PUT (and the Go library) could shrink a disk. - A shrink answered 400 BadRequest; real Azure answers 400 InvalidParameter (InvalidResizeWithName, "Troubleshoot Azure disk resize failures"). - Attached-disk and active-SAS restrictions were not enforced, sku.tier was copied into the performance tier, a tier was accepted on any SKU, and conversions to/from UltraSSD_LRS were accepted. What the real API does, and what this does now: - New optional driver capability AzureDiskPatcher.PatchVolume. The merge and every rule run inside m.volumes.Update with copy-on-write, like AttachVolume. UpdateVolume (the re-PUT) goes through the same path. - Shrink -> InvalidArgument (400 InvalidParameter); size above the SKU maximum (32767 GiB, 65536 for Ultra/PremiumV2) -> 400. - Active, unexpired SAS (beginGetAccess) -> 409 ChangeDiskSizeWhileActiveSasNotAllowed for a resize, 409 OperationNotAllowed for other property changes; tags still update. - Attached disk (Microsoft Learn troubleshoot-disk-resize, expand-disks, disks-convert-types): an OS disk is only resized, and any disk only converted, while its VM is deallocated (409 OperationNotAllowed); data disks still grow online; a Standard/Premium disk of 4 TiB or less cannot grow past 4 TiB while attached (409 InvalidResizeForLargeDisks). - SKU conversion: Ultra is neither source nor target, PremiumV2 cannot be converted away, an OS disk cannot become PremiumV2, unknown SKUs are 400. - Performance tier (disks-change-performance): Premium SSD only, not below the size's baseline, P60-P80 only above 4 TiB; a move to Premium sets the baseline tier, a move off clears it, a grow raises it to the new baseline. sku.tier is read-only and ignored on PATCH and re-PUT. - The wire layer echoes the provider's ARM code via *driver.AzureDiskError.
… SKU/capacity on update
What was wrong:
- PATCH {"properties":{"reserved":false}} on a Linux plan, or {"kind":"app"},
was accepted, and a re-PUT without reserved silently turned a Linux plan
into a Windows one. TestPatchAppServicePlan pinned the flip as success.
- PATCH accepted sku.capacity -5 and 0, an unknown sku.name (keeping the
stale tier), moves between Consumption/Elastic Premium and dedicated
tiers, and an explicit tier that did not match the name.
What the real API does, and what this does now:
- Real Azure answers 400 BadRequest "You cannot change the OS hosting your
app at this time. Please recreate your app with the desired OS."
(Azure/bicep#5724; azurerm treats os_type as ForceNew). PATCH and re-PUT
now share checkPlanUpdate under the store lock, and refuse a change of
reserved or of a Linux/Windows kind with that 400 BadRequest.
- On a change: capacity must be >= 1 and <= the tier maximum (Free/Shared 1,
Basic 3, Standard 10, Premium 30, Isolated 100), the SKU must be known,
the hosting family (Consumption, Elastic Premium, Workflow Standard,
dedicated) cannot change, and the tier must match the name. A create with
a negative capacity is 400.
- The re-PUT is now an atomic update-or-insert rather than a blind Set.
- New provider tests use hand-rolled helpers instead of testify.
go generate picks up the new optional driver interface. The azure/disks wire-op list keeps its existing line and adds Update on its own line with a goconst nolint: wireops.go is a table of SDK operation names and already carries 11 goconst hits on development (min-occurrences is 2), so any first occurrence added to it trips the linter.
5b9e55e to
6710f0e
Compare
|
Thanks for the review, and for the race repro. The fixes are in
Medium 4: one departure, with sources. Microsoft Learn says data disks can be expanded without deallocating the VM (expand-disks). Online resize is unsupported only for OS disks, shared disks, and Standard/Premium disks of 4 TiB or less growing past 4 TiB (troubleshoot-disk-resize). So:
Was your live 409 on an OS disk, a shared disk, or one crossing 4 TiB? If it was a plain data disk under 4 TiB, I'd like to see it.
Not in this PR, suggest follow-ups:
Gates on
|
NitinKumar004
left a comment
There was a problem hiding this comment.
The disk and plan fixes hold up: the races are gone, the rules now live in the provider, and the error codes match Azure. Two new issues need fixing before merge. A plan re-PUT is refused as an OS change even when nothing changed, and there is one lint failure.
Earlier findings
High 1. Disk PATCH lost writes and raced attach/detach: fixed. PatchVolume merges inside m.volumes.Update with copy-on-write, and the handler no longer rebuilds a full config from a snapshot. I ran PatchVolume(size), PatchVolume(tags), AttachVolume and DescribeVolumes concurrently over 60 disks with -race -count=50. There were no races, and every disk kept its size, tags and attachment. The lock order is volumes, then instances and diskAccess read locks. No instances callback takes the volumes lock, so there is no inversion.
High 2. Rules only on the wire PATCH: fixed. UpdateVolume goes through patchVolume. Over serve, a re-PUT of diskSizeGB:16 on a 128 GB disk now returns 400 InvalidParameter, and GET still reports 128.
Medium 3. Shrink error code: fixed. Shrinking through PATCH or re-PUT returns 400 InvalidParameter.
Medium 4. Attached disk and SAS: fixed. The data-disk departure is fine, because Microsoft documents online expansion for data disks. Verified over serve:
- A data disk grows online on a running VM (202).
- A SKU change on a running VM returns 409
OperationNotAllowed. - An OS disk resize returns 409
OperationNotAllowedboth while the VM is running and while it is stopped but still allocated. - Crossing 4 TiB while attached returns 409
InvalidResizeForLargeDisks. - After deallocate, the OS disk resize and the SKU change both succeed.
- With an active SAS, a resize returns 409
ChangeDiskSizeWhileActiveSasNotAllowedon both PATCH and re-PUT. AfterendGetAccessit goes through.
Medium 5. Plan OS flip: fixed. PATCH reserved:false and PATCH kind:"app" on a Linux plan both return 400 BadRequest with the Azure message. A re-PUT that omits reserved is refused the same way. New finding 1 below is a false positive on this same check.
Low, plan SKU/capacity: fixed. Out-of-range capacity is refused, and so is an unknown SKU. So are moves between the dedicated, Elastic Premium and Consumption families, and a mismatched sku.tier. The tier is re-derived from the name.
Low, disk tier: fixed.
Low, tests: fixed. This is moot anyway, because the other tests in both packages already use testify.
Low, merge conflicts: fixed. The coverage docs have been regenerated.
New findings
1. A Linux plan with no stored kind can't be re-PUT as Linux (providers/azure/functions/app_service_plan_rules.go:81, app_service_plan.go:322).
checkPlanOS compares linuxKind(cur.Kind) with linuxKind(next.Kind) whenever next.Kind is set, so an empty stored kind reads as Windows.
- Repro: PUT
{"sku":{"name":"B1"},"properties":{"reserved":true}}, then PUT the same body with"kind":"linux"added. The second call returns 400BadRequest"You cannot change the OS hosting your app", even though nothing changed. - PATCH
{"kind":""}is accepted and blanks the kind on a Linux plan. From then on, every re-PUT withkind:"linux"andreserved:truefails the same way. - A plan restored from a snapshot taken before this PR has
Reserved=false, even when its kind islinux.
Suggested fix:
- Treat
reservedas the single source of truth for the OS, since Azure deriveskindfrom it. - Skip the kind comparison when either side is empty.
- Ignore an empty
kindin PATCH. - Optionally, derive
Reservedfrom a Linux kind on create and on restore.
2. Lint gate (internal/coveragegen/wireops.go:67). golangci-lint run --new-from-rev=origin/development reports nolintlint: the //nolint:goconst on "Update" is no longer used. Drop the directive.
3. (Low) Disk errors name internal IDs (providers/azure/virtualmachines/disk_update.go:353,371,381). The messages read Cannot change the SKU of disk "/subscriptions/sub/resourceGroups/rg/.../disks/disk-1" while it is attached to running VM "vm-00000001". Those are the driver's placeholder IDs, not the caller's disk (d1 in rg1) or VM (vm1). Format the message with the ARM name tag instead, or leave the ID out and let the handler add the resource name.
4. (Low) vmAllocated adds a reader to an existing PowerState race (disk_update.go:421). If you add Stop, Deallocate and Start to the concurrent test, you get DATA RACE reports at vm.go:709/714 (transitionInstances) against vmAllocated, and also against DescribeInstances, which predates this PR. A follow-up is fine, but this race is now on the disk update path too.
5. (Low, pre-existing, fine as follow-ups)
- diskState only reports
UnattachedandAttached. Azure also reportsActiveSASduring a grant andReservedfor a disk on a deallocated VM, and these new rules depend on those states. - A re-PUT that changes
locationsilently moves a disk or plan. sku.tierechoes the performance tier (P15) after a PATCH.- Plan SKU names keep the caller's casing.
- The create path doesn't apply the per-tier capacity maximum (B1 with capacity 50 is accepted).
Checked
go build ./....go vetandgo test -raceon providers/azure/{virtualmachines,functions}, server/azure/{disks,functions}, services/compute, internal/coveragegen and persist.- Concurrent disk and plan PATCH tests with
-race -count=50. - Over serve with curl:
- Disk PATCH and re-PUT rules.
- Attached and deallocated OS and data disks.
- SAS grant and revoke.
- 404 and malformed JSON.
- Tags REPLACE on PATCH, and
tagsabsent or null keeping the existing set, for both disks and plans. - Plan PATCH for OS, SKU, capacity, tier and property fields.
- Snapshot round-trip of the new plan fields.
What failed
armcompute/v5DisksClient.BeginUpdatePATCH .../Microsoft.Compute/disks/{name}armappservice/v3PlansClient.UpdatePATCH .../Microsoft.Web/serverfarms/{name}What changed
Disks (
server/azure/disks)DiskUpdatebody as a partial update. It answers 202 + Azure-AsyncOperation, the same wayCreateOrUpdatedoes, soBeginUpdate(...).PollUntilDonecompletes and its final GET returns the updated disk.tags: when present they replace the tag set wholesale (ARM Compute PATCH semantics, same as virtualMachines). The internal bookkeeping tags are kept.sku/tier: switched in place. Moving offUltraSSD_LRS/PremiumV2_LRSdrops the provisioned IOPS/MBps.diskSizeGB: can only grow. A shrink returns 400BadRequest("Disk size can only be increased...").diskIOPSReadWrite/diskMBpsReadWrite: accepted only onUltraSSD_LRS/PremiumV2_LRS. Any other SKU returns 400InvalidParameter.AzureDiskUpdater.UpdateVolume, so id,uniqueId,timeCreatedand any attachment are kept. Resource Graph reads the same volume, so it reports the new size and tags.coveragegenwire-op list gainsUpdateforazure/disks.docs/coveragehas been regenerated.App Service plans (
server/azure/functions,providers/azure/functions)kind,sku.name/tier/capacity(a new SKU name re-derives the tier, as create does),tags(replaced wholesale when present), andproperties.reserved,perSiteScaling,zoneRedundant,maximumElasticWorkerCount.PatchAppServicePlan, a read-modify-write done under the memstore lock (Store.Update).armappservice.PlanPatchResourcehas noskuortagsfields, so the SDK cannot send them. The handler still accepts them from a raw ARM PATCH, and a raw-HTTP test covers that path.docs/sdk-server.mdAzure rows are updated.How it was tested
Real SDK clients point at an
httptestTLS server:server/azure/disks/update_test.go:BeginUpdate+PollUntilDone: grow, SKU change to PremiumV2 with IOPS/MBps, tag replace, and identity preservedarmresourcegraph) shows the patcheddiskSizeGBand tagsserver/azure/functions/plan_patch_sdk_test.go:PlansClient.Updatesets maximumElasticWorkerCount/reserved/zoneRedundant and keeps sku/kind/tagsproviders/azure/functions/app_service_plan_test.go:TestPatchAppServicePlan.Lint reports 15
goconstfindings, andorigin/developmentreports the same 15 for these packages. This PR adds no new findings.