Skip to content

feat: run aks node controller at boot time faster by 30s#8082

Draft
awesomenix wants to merge 2 commits intomainfrom
nishp/nocse
Draft

feat: run aks node controller at boot time faster by 30s#8082
awesomenix wants to merge 2 commits intomainfrom
nishp/nocse

Conversation

@awesomenix
Copy link
Contributor

@awesomenix awesomenix commented Mar 12, 2026

Summary

  • move scriptless AKS node controller startup earlier by switching generated custom data to cloud-boothook
  • Savings of 30s
  • update the baked aks-node-controller.service ordering to match the earlier-start model while keeping the VHD enable path intact
  • update the e2e hack path to mirror the same boothook-driven startup pattern

Details

  • aks-node-controller/pkg/nodeconfigutils/utils.go now writes the config from boothook and starts aks-node-controller.service immediately
  • parts/linux/cloud-init/artifacts/aks-node-controller.service now waits on network-online.target and stays active (exited) after the one-shot run
  • e2e/vmss.go switches the hack flow from runcmd to a boothook-dropped service and wrapper
  • generate-testdata was run to refresh generated snapshot data impacted by the pkg change

Timings

  - +3s — cloud-init init-local (10:28:08)
   - +6s — network configured (10:28:11)
   - +8s — cloud-init init (10:28:13)
   - +12s — guest agent start (10:28:17)
   - +13.241s — aks-node-controller provision start (10:28:18.240921)
   - +29s — cloud-init modules:final (10:28:34)
   - +29.563s — aks-node-controller provision-wait start (10:28:34.562877)
   - +39s — kubelet start (10:28:44)
   - +40.296s — node registered
   - +40.544s — NodeReady
   - +41.893s — provision-wait finished

Copilot AI review requested due to automatic review settings March 12, 2026 02:18
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts the e2e VMSS provisioning flow to run the aks-node-controller hack in the foreground (synchronously) and alters how provisioning status is validated during scenario setup.

Changes:

  • Run /opt/azure/bin/aks-node-controller-hack provision ... synchronously in cloud-init instead of backgrounding it.
  • Stop setting the VMSS CustomScript commandToExecute when provisioning via AKSNodeConfig (commented out).
  • Disable the post-create Custom Script Extension status check (commented out).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
e2e/vmss.go Runs aks-node-controller-hack provision synchronously; comments out CSE command wiring when using AKSNodeConfig.
e2e/test_helpers.go Comments out the VMSS Custom Script Extension status validation after VMSS creation.
Comments suppressed due to low confidence (1)

e2e/vmss.go:158

  • cse is no longer set when s.Runtime.AKSNodeConfig != nil. In the DisableScriptLessCompilation path this results in generating CustomData that only writes the aks-node-controller config file, but does not execute aks-node-controller (the VMSS CustomScript extension is skipped because cseCmd is empty). This will prevent the node from being provisioned. Restore wiring so scriptless mode still executes /opt/azure/containers/aks-node-controller provision-wait (or alternatively ensure CustomDataWithHack runs the equivalent of provision-wait). Note: leaving cse empty can also lead to a nil dereference in getBaseVMSSModel for Windows, which assumes an ExtensionProfile exists.
	if s.Runtime.AKSNodeConfig != nil {
		//cse = nodeconfigutils.CSE
		customData = func() string {
			if config.Config.DisableScriptLessCompilation {
				data, err := nodeconfigutils.CustomData(s.Runtime.AKSNodeConfig)
				require.NoError(s.T, err, "failed to generate custom data from AKSNodeConfig")

You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +304 to +305
// err = getCustomScriptExtensionStatus(s, scenarioVM.VM)
// require.NoError(s.T, err)
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These commented-out lines disable getCustomScriptExtensionStatus validation entirely. That removes early detection of CSE failures (non-zero exit code / provisioning state) and disables saving Windows CSE output to the scenario log directory, making failures harder to debug and potentially letting scenarios proceed past provisioning errors. Consider re-enabling this check with a retry/polling loop (or gating it on whether a CustomScript extension was configured for the VMSS).

Copilot generated this review using guidance from repository custom instructions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.


You can also share your feedback on Copilot code review. Take the survey.

[Install]
WantedBy=cloud-init.target
WantedBy=oem-cloudinit.service
RemainAfterExit=yes
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unit file no longer has an [Install] section (e.g., WantedBy=...) after this change. VHD build currently runs systemctl enable aks-node-controller.service (vhdbuilder/packer/pre-install-dependencies.sh:79-81), which will fail with "no installation config" when [Install] is missing. Re-add an appropriate [Install] section so the baked enable path continues to work (and aligns with the PR description).

Suggested change
RemainAfterExit=yes
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

Copilot uses AI. Check for mistakes.
var cse, customData string
if s.Runtime.AKSNodeConfig != nil {
cse = nodeconfigutils.CSE
//cse = nodeconfigutils.CSE
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving cse = nodeconfigutils.CSE commented out is effectively dead code and makes it unclear whether AKSNodeConfig-based scenarios are supposed to use the VMSS CustomScript extension or not. Please either remove the commented line and add an explicit assignment/comment explaining why cse must be empty for this flow, or restore a real cse value if the extension should still run.

Suggested change
//cse = nodeconfigutils.CSE
// AKSNodeConfig-based scenarios rely solely on cloud-init/custom data and
// intentionally do not configure a VMSS CustomScript extension.
// Keep cse explicitly empty to reflect that no extension should be added.
cse = ""

Copilot uses AI. Check for mistakes.
Comment on lines +47 to +48
runcmd:
- ` + CSE + `
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CustomData() now injects a #cloud-config part that runs the CSE command via runcmd (/opt/azure/containers/aks-node-controller provision-wait). That means cloud-init will execute provision-wait during boot, which is a behavior change from “CustomData writes config; CSE runs provision-wait” and may block cloud-init completion until provisioning finishes (and potentially duplicate the command if a caller still runs CSE via the CustomScript extension). If the intent is only to write config + start the service earlier, consider removing this runcmd and keeping CSE as a separate command for callers/VM extension.

Suggested change
runcmd:
- ` + CSE + `

Copilot uses AI. Check for mistakes.
Comment on lines 162 to 166
return &aksnodeconfigv1.Configuration{
Version: "v1",
BootstrappingConfig: bootstrappingConfig,
DisableCustomData: nbc.AgentPoolProfile.IsFlatcar() || nbc.AgentPoolProfile.IsACL(),
DisableCustomData: true,
LinuxAdminUsername: "azureuser",
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoding DisableCustomData: true changes behavior from the previous Flatcar/ACL-only disable. If this field is meant to reflect whether custom data should be used for bootstrapping, setting it true for all converted configs may make AKSNodeConfig e2e scenarios diverge from real configurations and reduce coverage of the custom-data-enabled path. Consider restoring the previous conditional (or add a brief rationale if the field is intentionally always true for AKSNodeConfig e2e).

Copilot uses AI. Check for mistakes.
Key components:

1. `aks-node-controller.service`: systemd unit that is triggered once cloud-init is complete (guaranteeing that config is present on disk) and then kickstarts bootstrapping.
1. `aks-node-controller.service`: systemd unit that can be started directly by cloud-boothook as soon as the config file is written, while remaining enabled on the VHD as a fallback boot hook.
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The README states aks-node-controller.service remains enabled on the VHD as a fallback boot hook, but this PR’s unit file change removes the [Install] section (so systemctl enable aks-node-controller.service fails during VHD build). Either update this doc to match the new enable/start model, or restore an enable-able unit definition so the fallback claim is accurate.

Suggested change
1. `aks-node-controller.service`: systemd unit that can be started directly by cloud-boothook as soon as the config file is written, while remaining enabled on the VHD as a fallback boot hook.
1. `aks-node-controller.service`: systemd unit that is started directly by cloud-boothook as soon as the config file is written; it is started explicitly by the provisioning flow rather than being persistently enabled on the VHD as a fallback boot hook.

Copilot uses AI. Check for mistakes.
@awesomenix awesomenix changed the title feat: run aks node controller in sync feat: run aks node controller at boot time Mar 12, 2026
@awesomenix awesomenix changed the title feat: run aks node controller at boot time feat: run aks node controller at boot time faster by 30s Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants