From 704cf75f56237cc1b195e9ac4b2188fa8d341678 Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Mon, 17 Aug 2026 22:08:20 +0000 Subject: [PATCH 01/11] storm/aclagent: add trident-acl-agent E2E test harness, test images, and pipeline Adds the storm-trident E2E scenario ("aclagent") that validates trident-acl-agent end-to-end against real tridentd (while mocking kubelet/Nebraska) on a VM, testing stage/finalize/rollback/commit through real reboots, plus the storm code, pipeline wiring, test images, and development documentation to run these tests locally. Storm code (tools/storm/aclagent/): - proxies/: fake apiserver serving the Node annotation protocol (including a real K8s-compatible watch stream), a fake Nebraska/Omaha endpoint backed by the real github.com/flatcar/nebraska server package (not a hand-rolled mock), a fake image server for serving update COSIs, a minimal kubelet shim, and an RP client that drives scenarios and polls status annotations. - tests/: run-ab-update (stage/finalize/commit through a real reboot), run-rollback (rollback stage/finalize/commit through a real reboot, plus a regression test that a second rollback against an empty rollback chain is a servicing_kind no-op rather than a false success), plus VM lifecycle and log-collection helpers. - README.md documents local usage. Registered in tools/cmd/storm-trident/main.go alongside storm-trident's other scenarios. Test images (tests/images/): baseimg-acl-agent.yaml/ updateimg-acl-agent.yaml VM image configs for the base and post-update ACL test images, wired into testimages.py. Pipeline: trident-acl-agent-test.yml stage wired into e2e-template.yml. Docs: docs/Development/Testing/TridentAclAgent-Tests.md documents the suite; Testing.md links to it. Harness evolution as trident-acl-agent's config/annotation surface changed (all folded into this one commit, tree matches the final state of PR #731's original commit history): - goal_source renamed to "annotations"; --validate-connection checked before/after config delivery. - Nebraska mock rebuilt on the real flatcar/nebraska server package, modeling event/in-progress instance state instead of approximating it. - Config/kubelet persistence made durable across the A/B reboot; SSH stability required before post-reboot reconfigure. - Commit status read from its own annotation key rather than conflated with the operation status. - trident-acl-agent's static TOML config (then TRIDENT_ACL_AGENT_* env vars) progressively emptied out as Nebraska endpoint/app_id/track moved to per-request overrides on the update-request annotation (server/appId/track fields) instead: prepareVmForAclAgent no longer writes any config to the VM at all, and expectValidateConnection can inject one-off TRIDENT_ACL_AGENT_NEBRASKA_* env vars to exercise both the success and failure paths of that override mechanism directly. - kubernetes.node_name derived from the VM image's own Image Customizer hostname instead of a config override, matching trident-acl-agent's hostname-based default. - Assorted fixes: CI flag passthrough, Copilot-flagged review issues, Makefile go-tools dependency restoration, logrus formatting, and removal of a dead trident-acl-agent.conf collection step once the config file stopped existing. Verified: all 6 storm aclagent test cases (deploy-vm, check-deployment, run-ab-update, run-rollback, collect-logs, cleanup-vm) pass end-to-end against a real QEMU VM with real A/B update and rollback reboot cycles. --- .pipelines/templates/e2e-template.yml | 5 + .../trident-acl-agent-test.yml | 137 +++++ Makefile | 21 + docs/Development/Testing/Testing.md | 4 + .../Testing/TridentAclAgent-Tests.md | 203 +++++++ tests/images/testimages.py | 8 + tests/images/trident-vm-testimage/README.md | 33 +- .../base/baseimg-acl-agent.yaml | 202 +++++++ .../base/updateimg-acl-agent.yaml | 211 +++++++ tools/cmd/storm-trident/main.go | 4 + tools/go.mod | 62 +- tools/go.sum | 183 ++++-- tools/storm/aclagent/README.md | 107 ++++ tools/storm/aclagent/proxies/apiserver.go | 357 ++++++++++++ tools/storm/aclagent/proxies/constants.go | 11 + tools/storm/aclagent/proxies/imageserver.go | 57 ++ tools/storm/aclagent/proxies/kubelet.go | 183 ++++++ tools/storm/aclagent/proxies/nebraska.go | 426 ++++++++++++++ tools/storm/aclagent/proxies/nebraska_test.go | 167 ++++++ tools/storm/aclagent/proxies/rp.go | 179 ++++++ tools/storm/aclagent/proxies/scenario.go | 105 ++++ tools/storm/aclagent/tests/logs.go | 11 + tools/storm/aclagent/tests/rollback.go | 135 +++++ tools/storm/aclagent/tests/update.go | 548 ++++++++++++++++++ tools/storm/aclagent/tests/vm.go | 43 ++ tools/storm/aclagent/trident.go | 92 +++ tools/storm/aclagent/utils/config/config.go | 19 + tools/storm/servicing/tests/update.go | 2 +- tools/storm/utils/vm/qemu/qemu.go | 7 +- 29 files changed, 3448 insertions(+), 74 deletions(-) create mode 100644 .pipelines/templates/stages/testing_acl_agent/trident-acl-agent-test.yml create mode 100644 docs/Development/Testing/TridentAclAgent-Tests.md create mode 100644 tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml create mode 100644 tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml create mode 100644 tools/storm/aclagent/README.md create mode 100644 tools/storm/aclagent/proxies/apiserver.go create mode 100644 tools/storm/aclagent/proxies/constants.go create mode 100644 tools/storm/aclagent/proxies/imageserver.go create mode 100644 tools/storm/aclagent/proxies/kubelet.go create mode 100644 tools/storm/aclagent/proxies/nebraska.go create mode 100644 tools/storm/aclagent/proxies/nebraska_test.go create mode 100644 tools/storm/aclagent/proxies/rp.go create mode 100644 tools/storm/aclagent/proxies/scenario.go create mode 100644 tools/storm/aclagent/tests/logs.go create mode 100644 tools/storm/aclagent/tests/rollback.go create mode 100644 tools/storm/aclagent/tests/update.go create mode 100644 tools/storm/aclagent/tests/vm.go create mode 100644 tools/storm/aclagent/trident.go create mode 100644 tools/storm/aclagent/utils/config/config.go diff --git a/.pipelines/templates/e2e-template.yml b/.pipelines/templates/e2e-template.yml index a0654303a4..f06dc752c7 100644 --- a/.pipelines/templates/e2e-template.yml +++ b/.pipelines/templates/e2e-template.yml @@ -270,6 +270,11 @@ stages: dependsOnStage: ${{ parameters.baseImageArtifactStage }} testSecureBoot: ${{ parameters.testSecureBoot }} + # Validate trident-acl-agent (storm A/B update scenario against a real tridentd) + - template: stages/testing_acl_agent/trident-acl-agent-test.yml + parameters: + dependsOnStage: ${{ parameters.baseImageArtifactStage }} + # TESTING stages for PRERELEASE - ${{ if eq(parameters.stageType, 'pre') }}: # Functional Testing diff --git a/.pipelines/templates/stages/testing_acl_agent/trident-acl-agent-test.yml b/.pipelines/templates/stages/testing_acl_agent/trident-acl-agent-test.yml new file mode 100644 index 0000000000..b35e8ce472 --- /dev/null +++ b/.pipelines/templates/stages/testing_acl_agent/trident-acl-agent-test.yml @@ -0,0 +1,137 @@ +parameters: + - name: dependsOnStage + type: string + default: "" + + - name: micBuildType + displayName: MIC Build Type + type: string + values: + - dev + - preview + - release + default: release + + - name: micVersion + displayName: MIC Version + type: string + default: "*.*.*" + + - name: baseimgAzlVersion + displayName: Base Image AZL version + type: string + default: "3.0" + + - name: verboseLogging + displayName: "Enable verbose logging" + type: boolean + default: false + +stages: + - stage: BuildImagesAclAgent + displayName: Build Base and Update Images for trident-acl-agent + dependsOn: + - PrepareSSHKeys + - GetTridentBinaries_rpms_amd64 + - ${{ if ne(parameters.dependsOnStage, '') }}: + - ${{ parameters.dependsOnStage }} + + jobs: + - template: ../trident_images/build-image.yml + parameters: + label: "acl-agent-base" + makeTarget: "artifacts/trident-vm-acl-agent-testimage.qcow2" + baseimgType: qemu_guest + baseimgAzlVersion: ${{ parameters.baseimgAzlVersion }} + micBuildType: ${{ parameters.micBuildType }} + micVersion: ${{ parameters.micVersion }} + useStagedSshKeys: true + + - template: ../trident_images/build-image.yml + parameters: + label: "acl-agent-update" + makeTarget: "artifacts/trident-vm-acl-agent-update-testimage.cosi" + baseimgType: qemu_guest + baseimgAzlVersion: ${{ parameters.baseimgAzlVersion }} + micBuildType: ${{ parameters.micBuildType }} + micVersion: ${{ parameters.micVersion }} + useStagedSshKeys: true + + - stage: TridentAclAgentTest + displayName: Validate trident-acl-agent + dependsOn: + - BuildingTools + - BuildImagesAclAgent + + jobs: + - job: AclAgentStormTest + displayName: Run storm aclagent scenario + timeoutInMinutes: 30 + pool: + type: linux + name: trident-ubuntu-1es-pool-eastus2 + hostArchitecture: amd64 + + variables: + ob_outputDirectory: /tmp/output + ob_artifactBaseName: "aclagent-storm-test" + + steps: + - template: ../common_tasks/checkout_trident.yml + - template: ../common_tasks/avoid-pypi-usage.yml + + - task: DownloadPipelineArtifact@2 + inputs: + buildType: current + artifactName: image-acl-agent-base + targetPath: "$(Build.ArtifactStagingDirectory)" + displayName: Download Base Image (qcow2) + + - task: DownloadPipelineArtifact@2 + inputs: + buildType: current + artifactName: image-acl-agent-update + targetPath: "$(Build.ArtifactStagingDirectory)" + displayName: Download Update Image (cosi) + + - task: DownloadPipelineArtifact@2 + inputs: + buildType: current + artifactName: ssh-keys + targetPath: "$(Build.ArtifactStagingDirectory)/ssh" + displayName: Download SSH Keys + + - task: DownloadPipelineArtifact@2 + displayName: "Download go-tools" + inputs: + buildType: current + artifactName: "go-tools" + patterns: | + storm-trident + targetPath: "$(TRIDENT_SOURCE_DIR)/bin" + + - bash: | + set -eux + chmod +x $(TRIDENT_SOURCE_DIR)/bin/storm-trident + cp $(Build.ArtifactStagingDirectory)/ssh/id_rsa* ~/.ssh/ + chmod -R 700 ~/.ssh/ + mkdir -p $(ob_outputDirectory) + displayName: Set up SSH keys and output directory + workingDirectory: $(TRIDENT_SOURCE_DIR) + + - bash: | + set -eux + ls -la $(Build.ArtifactStagingDirectory)/ + displayName: List downloaded image artifacts + + - bash: | + set -eux + + FLAGS="-a --verbose" + + sudo ./bin/storm-trident run aclagent $FLAGS \ + --output-path $(ob_outputDirectory) \ + --artifacts-dir $(Build.ArtifactStagingDirectory) \ + --ssh-private-key-path ~/.ssh/id_rsa + displayName: "๐Ÿงช Run trident-acl-agent A/B update + rollback scenario" + workingDirectory: $(TRIDENT_SOURCE_DIR) diff --git a/Makefile b/Makefile index 8638d018f4..bc90adb1f4 100644 --- a/Makefile +++ b/Makefile @@ -1220,6 +1220,27 @@ artifacts/trident-vm-usr-verity-testimage.qcow2: \ --output-image-format qcow2 \ --config-file /repo/$(VM_IMAGE_PATH_PREFIX)/baseimg-usr-verity.yaml +artifacts/trident-vm-acl-agent-testimage.qcow2: \ + $(QEMU_GUEST_IMAGE) \ + $(TRIDENT_VM_DEPENDENCIES) \ + $(VM_IMAGE_PATH_PREFIX)/baseimg-acl-agent.yaml \ + $(VM_IMAGE_PATH_PREFIX)/files/id_rsa.pub \ + artifacts/rpm-overrides + @echo "Building $@ from $<" + docker run --rm \ + --privileged \ + -v ".:/repo:z" \ + -v "/dev:/dev" \ + ${MIC_CONTAINER_IMAGE} \ + --log-level debug \ + --rpm-source /repo/bin/RPMS \ + --rpm-source /repo/artifacts/rpm-overrides \ + --build-dir /build \ + --image-file /repo/$< \ + --output-image-file /repo/$@ \ + --output-image-format qcow2 \ + --config-file /repo/$(VM_IMAGE_PATH_PREFIX)/baseimg-acl-agent.yaml + artifacts/trident-vm-grub-verity-azure-testimage.vhd: \ $(CORE_SELINUX_IMAGE) \ $(TRIDENT_VM_DEPENDENCIES) \ diff --git a/docs/Development/Testing/Testing.md b/docs/Development/Testing/Testing.md index 9812dd4b70..22d318d9d3 100644 --- a/docs/Development/Testing/Testing.md +++ b/docs/Development/Testing/Testing.md @@ -54,6 +54,10 @@ manual rollback chains without using `netlaunch` or an installer ISO. rollback via `storm-trident run servicing` - [Rollback Tests](Rollback-Tests.md) โ€” full rollback chain (A/B + runtime updates) via `storm-trident run rollback` +- [Trident ACL Agent Tests](TridentAclAgent-Tests.md) โ€” validates + `trident-acl-agent`'s label-driven update protocol against fake + Kubernetes API server and Nebraska/Omaha endpoints via + `storm-trident run aclagent` ## Code Coverage diff --git a/docs/Development/Testing/TridentAclAgent-Tests.md b/docs/Development/Testing/TridentAclAgent-Tests.md new file mode 100644 index 0000000000..a3004c6a59 --- /dev/null +++ b/docs/Development/Testing/TridentAclAgent-Tests.md @@ -0,0 +1,203 @@ +--- +sidebar_position: 9 +--- + +# Trident ACL Agent Tests + +`storm-trident run aclagent` is the single supported validation entrypoint for +the label-driven `trident-acl-agent` protocol described in the ACL AKS +node-label design. Unlike [Servicing Tests](Servicing-Tests.md), which drive +Trident's own `stage`/`finalize` gRPC calls directly, this scenario validates +`trident-acl-agent` itself: it deploys a VM, starts fake in-process test +doubles for the Kubernetes API server and the Nebraska/Omaha update server, +seeds bootstrap node labels, and lets the real `trident-acl-agent` binary +running inside the VM drive a full A/B update against those fakes. + +There is intentionally no fake `tridentd` โ€” the scenario talks to the real +`tridentd` and real `trident-acl-agent` running inside the VM. + +## What It Validates + +- `trident-acl-agent` watching its own Kubernetes Node object for + RP-authored label changes (via `kube::runtime::watcher()`) +- Reading the update image URL/hash from labels and triggering a real + Trident `stage` + `finalize` A/B update through the normal gRPC path +- Patching back observed-state labels/annotations as the update progresses +- Resuming correctly after a real reboot (see [Reboot Choice](#reboot-choice)) + +## VM Image Contents + +The VM image used by this scenario must already contain: + +- `tridentd.socket` installed and enabled (starts `tridentd.service` on + demand) +- `trident-acl-agent` package installed, but **`trident-acl-agent.service` + left disabled** โ€” it must not start before a config file exists +- the same SSH user/key setup expected by the [servicing](Servicing-Tests.md) + scenario + +Both the enabled/disabled state of `trident-acl-agent.service` and +`/etc/trident/trident-acl-agent.conf` live under `/etc`, which is not part of +the A/B-swapped `/usr`/root volume pair in this usr-verity image layout. That +makes it safe for the scenario to write the config and enable the service +once, after `deploy-vm`, rather than baking enablement into the image โ€” the +state persists across `run-ab-update`'s finalize the same way the config file +does. + +## Prerequisites + +- **Linux host** with root access +- **libvirt and QEMU** installed and configured +- **Docker** (for building images with Image Customizer) +- **Go 1.24+** (for building Go tools) +- **Rust** (latest stable, for building Trident and `trident-acl-agent`) + +See [Dependencies](../Building/Dependencies.md) for full build dependency +details. + +## Building Dependencies + +### 1. Build Trident, `trident-acl-agent`, and RPMs + +Always build through `make`, not a raw `cargo build`, when the RPM tarball +needs to reflect a source change โ€” `make` injects the dev version string +(`TRIDENT_VERSION`) that the RPM spec's `%check` step verifies against. A +plain `cargo build` skips that and produces an RPM build failure. + +```bash +make target/release/trident target/release/trident-acl-agent +make bin/trident-rpms.tar.gz +``` + +### 2. Build Go Tools + +```bash +make bin/storm-trident +``` + +### 3. Generate SSH Keys + +```bash +make artifacts/id_rsa +``` + +:::note +The VM images below bake in the public key from `artifacts/id_rsa.pub` (via +the `files/id_rsa.pub` Makefile rule), **not** `~/.ssh/id_rsa.pub`. Always +pass `--ssh-private-key-path artifacts/id_rsa` when running the scenario +locally โ€” using your personal `~/.ssh/id_rsa` doesn't fail fast, it just +hangs/retries during `check-deployment`'s SSH auth. +::: + +### 4. Download the qemu_guest Base Image + +Same base image as the servicing tests โ€” see [Servicing Tests, step +4](Servicing-Tests.md#4-download-the-qemu_guest-base-image) for details. + +### 5. Build the Base and Update VM Images + +The scenario needs two images built from the current source: + +```bash +# Base image: trident-acl-agent installed but disabled +make artifacts/trident-vm-acl-agent-testimage.qcow2 + +# Update image: what the agent updates the VM to +make artifacts/trident-vm-acl-agent-update-testimage.cosi +``` + +:::caution Rebuild after any `trident-acl-agent` change +Both image targets embed the RPM built in step 1. If you only rebuild the +Rust binary and re-run the scenario without rebuilding these images, you are +still testing the **old** binary baked into the existing qcow2/cosi files โ€” +the failure (or fix) you're trying to observe silently won't reproduce. Clear +stale artifacts first if you're not sure they're current: + +```bash +rm -f artifacts/trident-vm-acl-agent-testimage.qcow2 \ + artifacts/trident-vm-acl-agent-update-testimage.cosi +``` +::: + +## Running the ACL Agent Scenario + +The scenario requires root access for VM creation via `virt-install`: + +```bash +sudo bin/storm-trident run aclagent \ + --artifacts-dir ./artifacts \ + --output-path /tmp/aclagent-output \ + --ssh-private-key-path ./artifacts/id_rsa \ + --verbose +``` + +### Test Cases + +The scenario runs these test cases in order: + +1. **deploy-vm** โ€” Copies the base qcow2 image and creates a QEMU VM +2. **check-deployment** โ€” Verifies the VM booted and is accessible via SSH; + writes `/etc/trident/trident-acl-agent.conf` pointing at the + `localhost:` endpoints storm reverse-SSH-forwards into the VM, then + runs `systemctl enable --now trident-acl-agent.service` +3. **run-ab-update** โ€” Starts the fake apiserver and fake Nebraska/Omaha + endpoints in-process, seeds bootstrap node labels, patches the desired + update-image label, and waits for `trident-acl-agent` to drive a real + Trident A/B update to completion (including a real reboot) +4. **collect-logs** โ€” Fetches `trident-acl-agent` and Trident logs from the + VM via SSH; also runs automatically (with a `journalctl` dump for + `trident-acl-agent.service`) if `run-ab-update` times out waiting for the + service to become active, to make crash-loops self-diagnosing +5. **cleanup-vm** โ€” Destroys the QEMU VM + +### Flags + +| Flag | Description | Default | +|------|-------------|---------| +| `--artifacts-dir` | Directory containing VM images | `/tmp` | +| `--output-path` | Output directory for logs | `./output` | +| `--platform` | `qemu` or `azure` | `qemu` | +| `--ssh-private-key-path` | Path to SSH private key | `~/.ssh/id_rsa` | +| `--api-server-port` | Port for the fake Kubernetes API server | `18080` | +| `--nebraska-port` | Port for the fake Nebraska/Omaha server | `18081` | +| `--verbose` | Enable verbose logging | `false` | +| `--test-case-to-run` | Run a specific test case only | `all` | + +## Reboot Choice + +This scenario uses a real VM reboot rather than a shim: `trident-acl-agent` +issues a genuine `systemctl reboot` on finalize, and the scenario polls SSH +until it goes unreachable (confirming the reboot actually happened) and then +reachable again (confirming the VM came back up), exercising the agent's +real post-reboot resume logic end to end. This is slower than a shim-based +approach, but it validates the real reboot path instead of a simulation of +it. + +## Debugging Failures + +If `trident-acl-agent.service` gets stuck reporting `activating` and the test +times out, that almost always means a **crash-restart loop**, not a slow +start โ€” the unit has no explicit `Type=`, so `Type=simple` (the implicit +default) is used, and systemd marks such units active immediately on +`fork`/`exec` with no readiness signal. A persistent `activating` state for +the full wait window can only mean `Restart=on-failure` (`RestartSec=5`) is +cycling the service. + +`run-ab-update`'s wait-for-active check captures `journalctl -u +trident-acl-agent.service --no-pager -n 200` on timeout and includes it in +the test failure, so the actual crash reason (e.g. a panic, a fatal error +from the Kubernetes client, or a config problem) should be visible directly +in the CI log or local output without a separate log-collection step. + +The fake Kubernetes API server (`tools/storm/aclagent/proxies/apiserver.go`) +is a minimal, hand-rolled HTTP handler โ€” it does not implement the full +Kubernetes API surface. If `trident-acl-agent` is changed to make a new kind +of API call (a different verb, a new field selector, list pagination, +etc.), the fake apiserver's routing may need a corresponding update or the +call will simply 404 and (since `trident-acl-agent` treats such client +errors as fatal) crash-loop the service. For example, migrating node-watching +from polling to `kube::runtime::watcher()` introduced an initial **LIST** +call to the collection endpoint (`GET /api/v1/nodes?fieldSelector=...`) that +the fake server didn't originally route, only the singular +`/api/v1/nodes/` path โ€” surfacing as exactly this crash-loop symptom +until the collection route was added. diff --git a/tests/images/testimages.py b/tests/images/testimages.py index 9ab341cba9..a5137a2114 100755 --- a/tests/images/testimages.py +++ b/tests/images/testimages.py @@ -155,6 +155,14 @@ requires_ukify=True, ssh_key="files/id_rsa.pub", ), + ImageConfig( + "trident-vm-acl-agent-update-testimage", + base_image=BaseImage.QEMU_GUEST, + config="trident-vm-testimage", + config_file="base/updateimg-acl-agent.yaml", + requires_ukify=True, + ssh_key="files/id_rsa.pub", + ), ImageConfig( "trident-vm-grub-verity-azure-testimage", base_image=BaseImage.CORE_SELINUX, diff --git a/tests/images/trident-vm-testimage/README.md b/tests/images/trident-vm-testimage/README.md index e527ae04ee..51d1d3c3b3 100644 --- a/tests/images/trident-vm-testimage/README.md +++ b/tests/images/trident-vm-testimage/README.md @@ -10,8 +10,19 @@ Two sets of images are available: - regular - with verity - -For both, a set of corresponding update images is available. +- with UKI usr-verity for ACL-agent-driven A/B update testing + +For both, a set of corresponding update images is available. The ACL-agent +variant reuses the servicing-style VM image layout but additionally installs +`trident-acl-agent` so storm ACL-agent scenarios can drive a real in-guest +agent talking to `tridentd`. The base (qcow2) image installs the package but +leaves `trident-acl-agent.service` disabled -- the storm scenario enables and +starts it itself after seeding a real config. Only the update image enables +`trident-acl-agent.service` by default, since after a real A/B update boots +into it there is no test harness left to `systemctl enable --now` it. Neither +image preseeds /etc/trident/trident-acl-agent.conf with runner-specific +tunnel ports; the test scenario should SSH in after boot and write the real +localhost proxy endpoints for Nebraska and the Kubernetes API server. ## Additional Prerequisites @@ -23,15 +34,17 @@ For both, a set of corresponding update images is available. To build the base image, run: -| Image type | Make command | Output path | -| ---------------------------- | --------------------------------------------------- | ---------------------------------------------- | -| Regular | `make artifacts/trident-vm-grub-testimage.qcow2` | `artifacts/trident-vm-grub-testimage.qcow2` | -| With verity `qcow2` | `make artifacts/trident-vm-grub-verity-testimage.qcow2` | `artifacts/trident-vm-grub-verity-testimage.qcow2` | -| With verity fixed size `vhd` | `make artifacts/trident-vm-grub-verity-testimage.vhd` | `artifacts/trident-vm-grub-verity-testimage.vhd` | +| Image type | Make command | Output path | +| ---------- | ------------ | ----------- | +| Regular | `make artifacts/trident-vm-grub-testimage.qcow2` | `artifacts/trident-vm-grub-testimage.qcow2` | +| With verity `qcow2` | `make artifacts/trident-vm-grub-verity-testimage.qcow2` | `artifacts/trident-vm-grub-verity-testimage.qcow2` | +| With verity fixed size `vhd` | `make artifacts/trident-vm-grub-verity-testimage.vhd` | `artifacts/trident-vm-grub-verity-testimage.vhd` | +| ACL-agent UKI usr-verity `qcow2` | `make artifacts/trident-vm-acl-agent-testimage.qcow2` | `artifacts/trident-vm-acl-agent-testimage.qcow2` | To build the update images, run: -| Image type | Make command | Output path | -| ----------- | --------------------------------------- | ----------------------------------- | -| Regular | `make trident-vm-grub-testimage` | `artifacts/trident-vm-grub-testimage/*` | +| Image type | Make command | Output path | +| ---------- | ------------ | ----------- | +| Regular | `make trident-vm-grub-testimage` | `artifacts/trident-vm-grub-testimage/*` | | With verity | `make trident-vm-grub-verity-testimage` | `artifacts/trident-vm-grub-testimage/*` | +| ACL-agent UKI usr-verity | `make artifacts/trident-vm-acl-agent-update-testimage.cosi` | `artifacts/trident-vm-acl-agent-update-testimage.cosi` | diff --git a/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml b/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml new file mode 100644 index 0000000000..4215e1cbdd --- /dev/null +++ b/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml @@ -0,0 +1,202 @@ +# UKI usr-verity VM test image for validating trident-acl-agent's +# label-driven A/B update protocol against a real tridentd. +storage: + bootType: efi + + disks: + - partitionTableType: gpt + partitions: + - id: esp + type: esp + label: esp + size: 512M + + - id: boot-a + size: 256M + + - id: boot-b + size: 256M + + - id: root-a + size: 4G + + - id: root-b + size: 4G + + - id: usr-a + size: 1G + + - id: usr-b + size: 1G + + - id: usr-hash-a + size: 128M + + - id: usr-hash-b + size: 128M + + - id: trident + label: trident + size: 512M + + - id: trident-acl-agent + label: trident-acl-agent + size: 128M + + - id: kubelet + label: kubelet + size: 128M + + - id: etc-trident + label: etc-trident + size: 128M + + - id: home + label: home + size: 1G + + - id: srv + label: srv + size: 128M + + verity: + - id: usrverity + name: usr + dataDeviceId: usr-a + hashDeviceId: usr-hash-a + dataDeviceMountIdType: uuid + hashDeviceMountIdType: uuid + + filesystems: + - deviceId: esp + type: fat32 + mountPoint: + idType: part-label + path: /boot/efi + options: umask=0077 + + - deviceId: boot-a + type: ext4 + mountPoint: + idType: uuid + path: /boot + + - deviceId: usrverity + type: ext4 + mountPoint: + path: /usr + options: defaults,ro + + - deviceId: root-a + type: ext4 + mountPoint: + idType: uuid + path: / + + - deviceId: trident + type: ext4 + mountPoint: + idType: part-label + path: /var/lib/trident + + - deviceId: trident-acl-agent + type: ext4 + mountPoint: + idType: part-label + path: /var/lib/trident-acl-agent + + - deviceId: kubelet + type: ext4 + mountPoint: + idType: part-label + path: /var/lib/kubelet + + - deviceId: etc-trident + type: ext4 + mountPoint: + idType: part-label + path: /etc/trident + + - deviceId: home + type: ext4 + mountPoint: + idType: part-label + path: /home + + - deviceId: srv + type: ext4 + mountPoint: + idType: part-label + path: /srv + +os: + bootloader: + resetType: hard-reset + hostname: trident-acl-agent-testimg + + selinux: + mode: disabled + + uki: + mode: create + kernelCommandLine: + # Replicates BM base image settings, that would otherwise be lost + extraCommandLine: + - console=tty0 + - console=tty1 + - console=ttyS0 + - rd.debug + - loglevel=6 + - log_buf_len=1M + - systemd.journald.forward_to_console=1 + - rd.hostonly=0 + + packages: + install: + - device-mapper + - dnf + - efibootmgr + - iproute + - iptables + - jq + - kexec-tools + - lvm2 + - openssh-server + - systemd-boot + - systemd-udev + - trident-acl-agent + - veritysetup + - vim + - netplan + remove: + - grub2-efi-binary + + additionalFiles: + - source: files/99-dhcp-eth0.network + destination: /etc/systemd/network/99-dhcp-eth0.network + - source: files/sudoers-wheel + destination: /etc/sudoers.d/wheel + + services: + enable: + - kdump + - tridentd.socket + + users: + - name: testuser + sshPublicKeyPaths: + - files/id_rsa.pub + secondaryGroups: + - wheel + +scripts: + postCustomization: + - path: scripts/post-install.sh + - path: scripts/update-host-status.sh + - path: scripts/prepare-update-config-verity.sh + arguments: + - uki + - path: scripts/duid-type-to-link-layer.sh + +previewFeatures: + - uki diff --git a/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml b/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml new file mode 100644 index 0000000000..fbf15e0374 --- /dev/null +++ b/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml @@ -0,0 +1,211 @@ +# UKI usr-verity VM test image for validating trident-acl-agent's +# label-driven A/B update protocol against a real tridentd. +# +# This is the "update" counterpart to baseimg-acl-agent.yaml: it is built as +# a .cosi and served to the initial qcow2-booted VM as the A/B update target, +# not booted directly. Unlike the qcow2 base image, this image enables +# trident-acl-agent.service by default, since after a real A/B update boots +# into this image's root, there is no SSH session available anymore to +# `systemctl enable --now` it as the storm test harness does for the initial +# qcow2 boot. +storage: + bootType: efi + + disks: + - partitionTableType: gpt + partitions: + - id: esp + type: esp + label: esp + size: 512M + + - id: boot-a + size: 256M + + - id: boot-b + size: 256M + + - id: root-a + size: 4G + + - id: root-b + size: 4G + + - id: usr-a + size: 1G + + - id: usr-b + size: 1G + + - id: usr-hash-a + size: 128M + + - id: usr-hash-b + size: 128M + + - id: trident + label: trident + size: 512M + + - id: trident-acl-agent + label: trident-acl-agent + size: 128M + + - id: kubelet + label: kubelet + size: 128M + + - id: etc-trident + label: etc-trident + size: 128M + + - id: home + label: home + size: 1G + + - id: srv + label: srv + size: 128M + + verity: + - id: usrverity + name: usr + dataDeviceId: usr-a + hashDeviceId: usr-hash-a + dataDeviceMountIdType: uuid + hashDeviceMountIdType: uuid + + filesystems: + - deviceId: esp + type: fat32 + mountPoint: + idType: part-label + path: /boot/efi + options: umask=0077 + + - deviceId: boot-a + type: ext4 + mountPoint: + idType: uuid + path: /boot + + - deviceId: usrverity + type: ext4 + mountPoint: + path: /usr + options: defaults,ro + + - deviceId: root-a + type: ext4 + mountPoint: + idType: uuid + path: / + + - deviceId: trident + type: ext4 + mountPoint: + idType: part-label + path: /var/lib/trident + + - deviceId: trident-acl-agent + type: ext4 + mountPoint: + idType: part-label + path: /var/lib/trident-acl-agent + + - deviceId: kubelet + type: ext4 + mountPoint: + idType: part-label + path: /var/lib/kubelet + + - deviceId: etc-trident + type: ext4 + mountPoint: + idType: part-label + path: /etc/trident + + - deviceId: home + type: ext4 + mountPoint: + idType: part-label + path: /home + + - deviceId: srv + type: ext4 + mountPoint: + idType: part-label + path: /srv + +os: + bootloader: + resetType: hard-reset + hostname: trident-acl-agent-testimg + + selinux: + mode: disabled + + uki: + mode: create + kernelCommandLine: + # Replicates BM base image settings, that would otherwise be lost + extraCommandLine: + - console=tty0 + - console=tty1 + - console=ttyS0 + - rd.debug + - loglevel=6 + - log_buf_len=1M + - systemd.journald.forward_to_console=1 + - rd.hostonly=0 + + packages: + install: + - device-mapper + - dnf + - efibootmgr + - iproute + - iptables + - jq + - kexec-tools + - lvm2 + - openssh-server + - systemd-boot + - systemd-udev + - trident-acl-agent + - veritysetup + - vim + - netplan + remove: + - grub2-efi-binary + + additionalFiles: + - source: files/99-dhcp-eth0.network + destination: /etc/systemd/network/99-dhcp-eth0.network + - source: files/sudoers-wheel + destination: /etc/sudoers.d/wheel + + services: + enable: + - kdump + - tridentd.socket + - trident-acl-agent.service + + users: + - name: testuser + sshPublicKeyPaths: + - files/id_rsa.pub + secondaryGroups: + - wheel + +scripts: + postCustomization: + - path: scripts/post-install.sh + - path: scripts/update-host-status.sh + - path: scripts/prepare-update-config-verity.sh + arguments: + - uki + - path: scripts/duid-type-to-link-layer.sh + +previewFeatures: + - uki diff --git a/tools/cmd/storm-trident/main.go b/tools/cmd/storm-trident/main.go index ef550475fd..eb0fd1e308 100644 --- a/tools/cmd/storm-trident/main.go +++ b/tools/cmd/storm-trident/main.go @@ -1,6 +1,7 @@ package main import ( + "tridenttools/storm/aclagent" "tridenttools/storm/e2e" "tridenttools/storm/helpers" "tridenttools/storm/rollback" @@ -34,6 +35,9 @@ func main() { // Add Trident servicing scenario storm.AddScenario(&servicing.TridentServicingScenario{}) + // Add Trident ACL agent scenario + storm.AddScenario(&aclagent.TridentAclAgentScenario{}) + // Add Trident rollback scenario storm.AddScenario(&rollback.TridentRollbackScenario{}) diff --git a/tools/go.mod b/tools/go.mod index 0cef69be17..089afab795 100644 --- a/tools/go.mod +++ b/tools/go.mod @@ -6,17 +6,21 @@ require ( github.com/bmc-toolbox/bmclib/v2 v2.0.1-0.20230530141715-da28e42c453f github.com/digitalocean/go-libvirt v0.0.0-20250512231903-57024326652b github.com/dustin/go-humanize v1.0.1 - github.com/fatih/color v1.18.0 + github.com/fatih/color v1.19.0 + github.com/flatcar/nebraska/backend v0.0.0-20260806113018-30a488d8d300 github.com/google/uuid v1.6.0 github.com/knqyf263/go-rpmdb v0.1.1 github.com/microsoft/storm v0.4.0-alpha1 github.com/pkg/errors v0.9.1 github.com/pkg/sftp v1.13.9 - github.com/sirupsen/logrus v1.9.3 - github.com/spf13/cobra v1.8.1 + github.com/sirupsen/logrus v1.9.4 + github.com/spf13/cobra v1.10.2 github.com/spf13/viper v1.19.0 google.golang.org/grpc v1.82.1 + gopkg.in/guregu/null.v4 v4.0.0 gopkg.in/yaml.v2 v2.4.0 + k8s.io/api v0.34.1 + k8s.io/apimachinery v0.34.1 libvirt.org/go/libvirtxml v1.11007.0 libvirt.org/libvirt-go-xml v7.4.0+incompatible modernc.org/sqlite v1.20.3 @@ -24,20 +28,39 @@ require ( ) require ( + github.com/blang/semver/v4 v4.0.0 // indirect + github.com/doug-martin/goqu/v9 v9.19.0 // indirect + github.com/flatcar/go-omaha v0.0.2-0.20251014063321-4998849bd39c // indirect + github.com/fxamacker/cbor/v2 v2.9.0 // indirect + github.com/go-gorp/gorp/v3 v3.1.0 // indirect + github.com/gogo/protobuf v1.3.2 // indirect + github.com/jackc/pgpassfile v1.0.0 // indirect + github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect + github.com/jackc/pgx/v5 v5.10.0 // indirect + github.com/jackc/puddle/v2 v2.2.2 // indirect + github.com/jmoiron/sqlx v1.4.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect github.com/jstemmer/go-junit-report/v2 v2.1.0 // indirect github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect + github.com/lib/pq v1.12.3 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect github.com/opencontainers/go-digest v1.0.0 // indirect github.com/opencontainers/image-spec v1.1.1 // indirect github.com/remyoudompheng/bigfft v0.0.0-20230126093431-47fa9a501578 // indirect - github.com/rogpeppe/go-internal v1.12.0 // indirect - github.com/stretchr/testify v1.11.1 // indirect - golang.org/x/mod v0.35.0 // indirect + github.com/rs/zerolog v1.35.1 // indirect + github.com/rubenv/sql-migrate v1.8.1 // indirect + github.com/x448/float16 v0.8.4 // indirect + go.yaml.in/yaml/v2 v2.4.2 // indirect + golang.org/x/mod v0.37.0 // indirect golang.org/x/sync v0.22.0 // indirect - golang.org/x/tools v0.44.0 // indirect + golang.org/x/tools v0.47.0 // indirect golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478 // indirect google.golang.org/protobuf v1.36.11 // indirect - gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect + gopkg.in/inf.v0 v0.9.1 // indirect + k8s.io/klog/v2 v2.130.1 // indirect + k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect lukechampine.com/uint128 v1.2.0 // indirect modernc.org/cc/v3 v3.40.0 // indirect modernc.org/ccgo/v3 v3.16.13 // indirect @@ -47,6 +70,9 @@ require ( modernc.org/opt v0.1.3 // indirect modernc.org/strutil v1.1.3 // indirect modernc.org/token v1.0.1 // indirect + sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect + sigs.k8s.io/randfill v1.0.0 // indirect + sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect ) require ( @@ -66,30 +92,30 @@ require ( github.com/klauspost/compress v1.17.10 github.com/kr/fs v0.1.0 // indirect github.com/magiconair/properties v1.8.7 // indirect - github.com/mattn/go-colorable v0.1.13 // indirect - github.com/mattn/go-isatty v0.0.20 // indirect + github.com/mattn/go-colorable v0.1.15 // indirect + github.com/mattn/go-isatty v0.0.23 // indirect github.com/mitchellh/mapstructure v1.5.0 // indirect - github.com/pelletier/go-toml/v2 v2.2.3 // indirect + github.com/pelletier/go-toml/v2 v2.3.1 // indirect github.com/sagikazarmark/locafero v0.6.0 // indirect github.com/sagikazarmark/slog-shim v0.1.0 // indirect github.com/satori/go.uuid v1.2.0 // indirect github.com/seancfoley/bintree v1.3.1 // indirect github.com/seancfoley/ipaddress-go v1.7.1 github.com/sourcegraph/conc v0.3.0 // indirect - github.com/spf13/afero v1.11.0 // indirect + github.com/spf13/afero v1.15.0 // indirect github.com/spf13/cast v1.7.0 // indirect - github.com/spf13/pflag v1.0.5 // indirect + github.com/spf13/pflag v1.0.10 // indirect github.com/stmcginnis/gofish v0.19.0 github.com/subosito/gotenv v1.6.0 // indirect github.com/vishvananda/netlink v1.3.0 github.com/vishvananda/netns v0.0.4 // indirect go.uber.org/multierr v1.11.0 // indirect - golang.org/x/crypto v0.52.0 + golang.org/x/crypto v0.54.0 golang.org/x/exp v0.0.0-20240823005443-9b4947da3948 - golang.org/x/net v0.55.0 // indirect - golang.org/x/sys v0.45.0 // indirect - golang.org/x/term v0.43.0 // indirect - golang.org/x/text v0.37.0 // indirect + golang.org/x/net v0.57.0 // indirect + golang.org/x/sys v0.47.0 // indirect + golang.org/x/term v0.45.0 // indirect + golang.org/x/text v0.40.0 // indirect gopkg.in/ini.v1 v1.67.0 // indirect gopkg.in/yaml.v3 v3.0.1 ) diff --git a/tools/go.sum b/tools/go.sum index b4fc3f278b..32526d1fec 100644 --- a/tools/go.sum +++ b/tools/go.sum @@ -1,3 +1,7 @@ +filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA= +filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4= +github.com/DATA-DOG/go-sqlmock v1.5.0 h1:Shsta01QNfFxHCfpW6YH2STWB0MudeXXEWMr20OEh60= +github.com/DATA-DOG/go-sqlmock v1.5.0/go.mod h1:f/Ixk793poVmq4qj/V1dPUg2JEAKC73Q5eFN3EC/SaM= github.com/Jeffail/gabs/v2 v2.7.0 h1:Y2edYaTcE8ZpRsR2AtmPu5xQdFDIthFG0jYhu5PY8kg= github.com/Jeffail/gabs/v2 v2.7.0/go.mod h1:dp5ocw1FvBBQYssgHsG7I1WYsiLRtkUaB1FEtSwvNUw= github.com/VictorLowther/simplexml v0.0.0-20180716164440-0bff93621230 h1:t95Grn2mOPfb3+kPDWsNnj4dlNcxnvuR72IjY8eYjfQ= @@ -10,6 +14,8 @@ github.com/alecthomas/kong v1.12.1 h1:iq6aMJDcFYP9uFrLdsiZQ2ZMmcshduyGv4Pek0MQPW github.com/alecthomas/kong v1.12.1/go.mod h1:p2vqieVMeTAnaC83txKtXe8FLke2X07aruPWXyMPQrU= github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc= github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4= +github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= +github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ= github.com/bmc-toolbox/bmclib/v2 v2.0.1-0.20230530141715-da28e42c453f h1:5xXPluhAvpNCmrgVhQesx+GcpPX1pXRyDFtj3JmxT+g= github.com/bmc-toolbox/bmclib/v2 v2.0.1-0.20230530141715-da28e42c453f/go.mod h1:a3Ra0ce/LV3wAj7AHuphlHNTx5Sg67iQqtLGr1zoqio= github.com/bmc-toolbox/common v0.0.0-20240806132831-ba8adc6a35e3 h1:/BjZSX/sphptIdxpYo4wxAQkgMLyMMgfdl48J9DKNeE= @@ -18,33 +24,51 @@ github.com/bombsimon/logrusr/v2 v2.0.1 h1:1VgxVNQMCvjirZIYaT9JYn6sAVGVEcNtRE0y4m github.com/bombsimon/logrusr/v2 v2.0.1/go.mod h1:ByVAX+vHdLGAfdroiMg6q0zgq2FODY2lc5YJvzmOJio= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/denisenkom/go-mssqldb v0.10.0/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU= github.com/digitalocean/go-libvirt v0.0.0-20250512231903-57024326652b h1:o/RoLbHmKtibc3lMpuPcYGUjnboEORpLFnqtC89tfqY= github.com/digitalocean/go-libvirt v0.0.0-20250512231903-57024326652b/go.mod h1:B2R8mtJc0BNx0NvvfOajL5no+MaFDumyD5sHsxll62g= +github.com/doug-martin/goqu/v9 v9.19.0 h1:PD7t1X3tRcUiSdc5TEyOFKujZA5gs3VSA7wxSvBx7qo= +github.com/doug-martin/goqu/v9 v9.19.0/go.mod h1:nf0Wc2/hV3gYK9LiyqIrzBEVGlI8qW3GuDCEobC4wBQ= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= -github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM= -github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU= +github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w= +github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE= +github.com/flatcar/go-omaha v0.0.2-0.20251014063321-4998849bd39c h1:Zv3WUF6IfhWr0jA2xseIdFFPTNUeI0pUew8ikP6LLOA= +github.com/flatcar/go-omaha v0.0.2-0.20251014063321-4998849bd39c/go.mod h1:ntSHoD5lo8beYqjwPd98MPFaVOFtQ7ezRcnvMvnPwkw= +github.com/flatcar/nebraska/backend v0.0.0-20260806113018-30a488d8d300 h1:Dw2IJA6ATBMHLtMB3Y7Vcc6tf0W66sF11RKqsFnWEjA= +github.com/flatcar/nebraska/backend v0.0.0-20260806113018-30a488d8d300/go.mod h1:c42gCbeVkb5mPXTLr5RCWgEnz3CY3wv394yc/GpqPTc= github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= +github.com/fxamacker/cbor/v2 v2.9.0 h1:NpKPmjDBgUfBms6tr6JZkTHtfFGcMKsw3eGcmD/sapM= +github.com/fxamacker/cbor/v2 v2.9.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ= github.com/glebarez/go-sqlite v1.20.3 h1:89BkqGOXR9oRmG58ZrzgoY/Fhy5x0M+/WV48U5zVrZ4= github.com/glebarez/go-sqlite v1.20.3/go.mod h1:u3N6D/wftiAzIOJtZl6BmedqxmmkDfH3q+ihjqxC9u0= +github.com/go-gorp/gorp/v3 v3.1.0 h1:ItKF/Vbuj31dmV4jxA1qblpSwkl9g1typ24xoe70IGs= +github.com/go-gorp/gorp/v3 v3.1.0/go.mod h1:dLEjIyyRNiXvNZ8PSmzpt1GsWAUK8kjVhEpjH8TixEw= github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-sql-driver/mysql v1.6.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg= +github.com/go-sql-driver/mysql v1.8.1 h1:LedoTUt/eveggdHS9qUFC1EFSa8bU2+1pZjSRpvNJ1Y= +github.com/go-sql-driver/mysql v1.8.1/go.mod h1:wEBSXgmK//2ZFJyE+qWnIsVGmvmEKlqwuVSjsCm7DZg= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0= github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/pprof v0.0.0-20221118152302-e6195bd50e26 h1:Xim43kblpZXfIBQsbuBVKCudVG457BR2GZFIz3uw3hQ= github.com/google/pprof v0.0.0-20221118152302-e6195bd50e26/go.mod h1:dDKJzRmX4S37WGHujM7tX//fmj1uioxKzKxz3lo4HJo= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= @@ -60,46 +84,69 @@ github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUq github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= +github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= +github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= +github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0= +github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= +github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= +github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jacobweinstock/iamt v0.0.0-20230502042727-d7cdbe67d9ef h1:G4k02HGmBUfJFSNu3gfKJ+ki+B3qutKsYzYndkqqKc4= github.com/jacobweinstock/iamt v0.0.0-20230502042727-d7cdbe67d9ef/go.mod h1:FgmiLTU6cJewV4Xgrq6m5o8CUlTQOJtqzaFLGA0mG+E= github.com/jacobweinstock/registrar v0.4.7 h1:s4dOExccgD+Pc7rJC+f3Mc3D+NXHcXUaOibtcEsPxOc= github.com/jacobweinstock/registrar v0.4.7/go.mod h1:PWmkdGFG5/ZdCqgMo7pvB3pXABOLHc5l8oQ0sgmBNDU= +github.com/jmoiron/sqlx v1.4.0 h1:1PLqN7S1UYp5t4SrVVnt4nUVNemrDAtxlulVe+Qgm3o= +github.com/jmoiron/sqlx v1.4.0/go.mod h1:ZrZ7UsYB/weZdl2Bxg6jCRO9c3YHl8r3ahlKmRT4JLY= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/jstemmer/go-junit-report/v2 v2.1.0 h1:X3+hPYlSczH9IMIpSC9CQSZA0L+BipYafciZUWHEmsc= github.com/jstemmer/go-junit-report/v2 v2.1.0/go.mod h1:mgHVr7VUo5Tn8OLVr1cKnLuEy0M92wdRntM99h7RkgQ= github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNUXsshfwJMBgNA0RU6/i7WVaAegv3PtuIHPMs= github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:CzGEWj7cYgsdH8dAjBGEr58BoE7ScuLd+fwFZ44+/x8= +github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/klauspost/compress v1.17.10 h1:oXAz+Vh0PMUvJczoi+flxpnBEPxoER1IaAnU/NMPtT0= github.com/klauspost/compress v1.17.10/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0= github.com/knqyf263/go-rpmdb v0.1.1 h1:oh68mTCvp1XzxdU7EfafcWzzfstUZAEa3MW0IJye584= github.com/knqyf263/go-rpmdb v0.1.1/go.mod h1:9LQcoMCMQ9vrF7HcDtXfvqGO4+ddxFQ8+YF/0CVGDww= github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8= github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg= -github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= -github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= -github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= +github.com/lib/pq v1.10.1/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o= +github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o= +github.com/lib/pq v1.12.3 h1:tTWxr2YLKwIvK90ZXEw8GP7UFHtcbTtty8zsI+YjrfQ= +github.com/lib/pq v1.12.3/go.mod h1:/p+8NSbOcwzAEI7wiMXFlgydTwcgTr3OSKMsD2BitpA= github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY= github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0= -github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= -github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= -github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= -github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= -github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= -github.com/mattn/go-sqlite3 v1.14.15 h1:vfoHhTN1af61xCRSWzFIWzx2YskyMTwHLrExkBOjvxI= -github.com/mattn/go-sqlite3 v1.14.15/go.mod h1:2eHXhiwb8IkHr+BDWZGa96P6+rkvnG63S2DGjv9HUNg= +github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY= +github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= +github.com/mattn/go-isatty v0.0.23 h1:cYwCQTQf3HB6xUC+BtyCLZNr7IzbOmoZbmssVNzSyiQ= +github.com/mattn/go-isatty v0.0.23/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A= +github.com/mattn/go-sqlite3 v1.14.7/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU= +github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU= +github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y= github.com/microsoft/storm v0.4.0-alpha1 h1:U4Bn6rZQW33xrAq+s4pVT7D+RG7crWYTcOPboGysAyo= github.com/microsoft/storm v0.4.0-alpha1/go.mod h1:QMLHpLhA/rI2bmMFor4Vxd6N99k1eadlrXv6lPLbXks= github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY= github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee h1:W5t00kpgFdJifH4BDsTlE89Zl93FEloxaWZfGcifgq8= +github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M= -github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M= -github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc= +github.com/pelletier/go-toml/v2 v2.3.1 h1:MYEvvGnQjeNkRF1qUuGolNtNExTDwct51yp7olPtrEc= +github.com/pelletier/go-toml/v2 v2.3.1/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/sftp v1.13.9 h1:4NGkvGudBL7GteO3m6qnaQ4pC0Kvf0onSVc9gR3EWBw= @@ -107,11 +154,17 @@ github.com/pkg/sftp v1.13.9/go.mod h1:OBN7bVXdstkFFN/gdnHPUb5TE8eb8G1Rp9wCItqjkk github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/poy/onpar v1.1.2 h1:QaNrNiZx0+Nar5dLgTVp5mXkyoVFIbepjyEoGSnhbAY= +github.com/poy/onpar v1.1.2/go.mod h1:6X8FLNoxyr9kkmnlqpK6LSoiOtrO6MICtWwEuWkLjzg= github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= github.com/remyoudompheng/bigfft v0.0.0-20230126093431-47fa9a501578 h1:VstopitMQi3hZP0fzvnsLmzXZdQGc4bEcgu24cp+d4M= github.com/remyoudompheng/bigfft v0.0.0-20230126093431-47fa9a501578/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= -github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= -github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4= +github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= +github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= +github.com/rs/zerolog v1.35.1 h1:m7xQeoiLIiV0BCEY4Hs+j2NG4Gp2o2KPKmhnnLiazKI= +github.com/rs/zerolog v1.35.1/go.mod h1:EjML9kdfa/RMA7h/6z6pYmq1ykOuA8/mjWaEvGI+jcw= +github.com/rubenv/sql-migrate v1.8.1 h1:EPNwCvjAowHI3TnZ+4fQu3a915OpnQoPAjTXCGOy2U0= +github.com/rubenv/sql-migrate v1.8.1/go.mod h1:BTIKBORjzyxZDS6dzoiw6eAFYJ1iNlGAtjn4LGeVjS8= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/sagikazarmark/locafero v0.6.0 h1:ON7AQg37yzcRPU69mt7gwhFEBwxI6P9T4Qu3N51bwOk= github.com/sagikazarmark/locafero v0.6.0/go.mod h1:77OmuIc6VTraTXKXIs/uvUxKGUXjE1GbemJYHqdNjX0= @@ -123,24 +176,28 @@ github.com/seancfoley/bintree v1.3.1 h1:cqmmQK7Jm4aw8gna0bP+huu5leVOgHGSJBEpUx3E github.com/seancfoley/bintree v1.3.1/go.mod h1:hIUabL8OFYyFVTQ6azeajbopogQc2l5C/hiXMcemWNU= github.com/seancfoley/ipaddress-go v1.7.1 h1:fDWryS+L8iaaH5RxIKbY0xB5Z+Zxk8xoXLN4S4eAPdQ= github.com/seancfoley/ipaddress-go v1.7.1/go.mod h1:TQRZgv+9jdvzHmKoPGBMxyiaVmoI0rYpfEk8Q/sL/Iw= -github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= -github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= +github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w= +github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g= github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo= github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0= -github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8= -github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY= +github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I= +github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg= github.com/spf13/cast v1.7.0 h1:ntdiHjuueXFgm5nzDRdOS4yfT43P5Fnud6DH50rz/7w= github.com/spf13/cast v1.7.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo= -github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM= -github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y= -github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= -github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU= +github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4= +github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= +github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/viper v1.19.0 h1:RWq5SEjt8o25SROyN3z2OrDB9l7RPd3lwTWU8EcEdcI= github.com/spf13/viper v1.19.0/go.mod h1:GQUN9bilAbhU/jgc1bKs99f/suXKeUMct8Adx5+Ntkg= github.com/stmcginnis/gofish v0.19.0 h1:fmxdRZ5WHfs+4ExArMYoeRfoh+SAxLELKtmoVplBkU4= github.com/stmcginnis/gofish v0.19.0/go.mod h1:lq2jHj2t8Krg0Gx02ABk8MbK7Dz9jvWpO/TGnVksn00= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY= +github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= @@ -152,6 +209,10 @@ github.com/vishvananda/netlink v1.3.0 h1:X7l42GfcV4S6E4vHTsw48qbrV+9PVojNfIhZcwQ github.com/vishvananda/netlink v1.3.0/go.mod h1:i6NetklAujEcC6fK0JPjT8qSwWyO0HLn4UKG+hGqeJs= github.com/vishvananda/netns v0.0.4 h1:Oeaw1EM2JMxD51g9uhtC0D7erkIjgmj8+JZc26m1YX8= github.com/vishvananda/netns v0.0.4/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM= +github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= +github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= @@ -169,24 +230,36 @@ go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A= go.uber.org/goleak v1.2.1/go.mod h1:qlT2yGI9QafXHhZZLxlSuNsMw3FFLxBr+tBRlmO1xH4= go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= +go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= +go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= +go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190325154230-a5d413f7728c/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc= golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU= golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8= golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk= -golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988= -golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc= +golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw= +golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk= golang.org/x/exp v0.0.0-20240823005443-9b4947da3948 h1:kx6Ds3MlpiUHKj7syVnbp57++8WpuKPcR5yjLBjvLEA= golang.org/x/exp v0.0.0-20240823005443-9b4947da3948/go.mod h1:akd2r19cwCdwSwWeIdzYQGa/EZZyqcOdwWiwj5L5eKQ= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= -golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= -golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= +golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ= +golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= @@ -194,9 +267,11 @@ golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk= golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM= -golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= -golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= +golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE= +golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y= @@ -206,23 +281,22 @@ golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= -golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= @@ -232,8 +306,8 @@ golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU= golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk= golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY= golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM= -golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4= -golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk= +golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0= +golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= @@ -243,17 +317,22 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ= -golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= -golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38= +golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs= +golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58= golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk= -golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c= -golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI= +golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q= +golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk= golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= @@ -269,6 +348,10 @@ gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntN gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/go-playground/assert.v1 v1.2.1 h1:xoYuJVE7KT85PYWrN730RguIQO0ePzVRfFMXadIrXTM= gopkg.in/go-playground/assert.v1 v1.2.1/go.mod h1:9RXL0bg/zibRAgZUYszZSwO/z8Y/a8bDuhia5mkpMnE= +gopkg.in/guregu/null.v4 v4.0.0 h1:1Wm3S1WEA2I26Kq+6vcW+w0gcDo44YKYD7YIEJNHDjg= +gopkg.in/guregu/null.v4 v4.0.0/go.mod h1:YoQhUrADuG3i9WqesrCmpNRwm1ypAgSHYqoOcTu/JrI= +gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= +gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= @@ -276,6 +359,14 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +k8s.io/api v0.34.1 h1:jC+153630BMdlFukegoEL8E/yT7aLyQkIVuwhmwDgJM= +k8s.io/api v0.34.1/go.mod h1:SB80FxFtXn5/gwzCoN6QCtPD7Vbu5w2n1S0J5gFfTYk= +k8s.io/apimachinery v0.34.1 h1:dTlxFls/eikpJxmAC7MVE8oOeP1zryV7iRyIjB0gky4= +k8s.io/apimachinery v0.34.1/go.mod h1:/GwIlEcWuTX9zKIg2mbw0LRFIsXwrfoVxn+ef0X13lw= +k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk= +k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= +k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y= +k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= libvirt.org/go/libvirtxml v1.11007.0 h1:SNc8wjOprLl0nsR0H1T0qs2pRD3j/0cVrUTZiP931zI= libvirt.org/go/libvirtxml v1.11007.0/go.mod h1:7Oq2BLDstLr/XtoQD8Fr3mfDNrzlI3utYKySXF2xkng= libvirt.org/libvirt-go-xml v7.4.0+incompatible h1:NaCRjbtz//xuTZOp1nDHbe0eu5BQlhIy5PPuc09EWtU= @@ -310,3 +401,11 @@ modernc.org/z v1.7.0 h1:xkDw/KepgEjeizO2sNco+hqYkU12taxQFqPEmgm1GWE= modernc.org/z v1.7.0/go.mod h1:hVdgNMh8ggTuRG1rGU8x+xGRFfiQUIAw0ZqlPy8+HyQ= oras.land/oras-go/v2 v2.6.2 h1:N04RXngAp1LJKTG6ifz3xHPipasEkWr+hFmInja5YKo= oras.land/oras-go/v2 v2.6.2/go.mod h1:PlTtg4JTDJkDe8yVHpM2wz7/YDc00GVas+i4jAW2TZ4= +sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE= +sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg= +sigs.k8s.io/randfill v1.0.0 h1:JfjMILfT8A6RbawdsK2JXGBR5AQVfd+9TbzrlneTyrU= +sigs.k8s.io/randfill v1.0.0/go.mod h1:XeLlZ/jmk4i1HRopwe7/aU3H5n1zNUcX6TM94b3QxOY= +sigs.k8s.io/structured-merge-diff/v6 v6.3.0 h1:jTijUJbW353oVOd9oTlifJqOGEkUw2jB/fXCbTiQEco= +sigs.k8s.io/structured-merge-diff/v6 v6.3.0/go.mod h1:M3W8sfWvn2HhQDIbGWj3S099YozAsymCo/wrT5ohRUE= +sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs= +sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4= diff --git a/tools/storm/aclagent/README.md b/tools/storm/aclagent/README.md new file mode 100644 index 0000000000..d872b51938 --- /dev/null +++ b/tools/storm/aclagent/README.md @@ -0,0 +1,107 @@ +# Trident ACL agent storm scenario + +`storm-trident aclagent` is the single supported validation entrypoint for the +label-driven trident ACL agent protocol. + +## What it does + +- deploys a QEMU or Azure VM using the existing storm VM helpers +- starts the fake single-node Kubernetes apiserver in-process inside the storm binary +- starts the fake Nebraska/Omaha endpoint in-process inside the storm binary +- seeds bootstrap node annotations and simulated Ready flips with an in-process kubelet helper +- talks to the real `tridentd` and real `trident-acl-agent` running inside the VM +- lets `trident-acl-agent` issue a real `systemctl reboot` on finalize, then polls SSH until it drops and comes back up to confirm the reboot actually happened + +There is intentionally no fake `tridentd`. + +## Test cases + +- `deploy-vm` +- `check-deployment` +- `run-ab-update` +- `run-rollback` +- `collect-logs` +- `cleanup-vm` + +## Expected image contents + +The VM image used by this scenario must already contain: + +- `tridentd.socket` installed and enabled (starts `tridentd.service` on demand) +- `trident-acl-agent` package installed, but **`trident-acl-agent.service` + left disabled** -- it must not start before the fake kubeconfig exists +- the same SSH user/key setup expected by the existing storm servicing scenario + +Both the enabled/disabled state of `trident-acl-agent.service` +(`/etc/systemd/system/multi-user.target.wants/...`) and the fake kubeconfig +at `/var/lib/kubelet/kubeconfig` live under paths that are not part of the +A/B-swapped `/usr`/root volume pair in this usr-verity layout. That makes it +safe for the scenario to deliver the kubeconfig and enable the service once, +after `deploy-vm`, rather than baking enablement into the image: the state +persists across `run-ab-update`'s finalize the same way the kubeconfig does. + +`prepareVmForAclAgent` never writes a `trident-acl-agent.conf` at all - the +agent's compiled-in defaults already cover everything it needs (see the +function's own doc comment): `nebraska.app_id`/`nebraska.endpoint` are +supplied per-request via the update-request annotation's `appId`/`server` +fields instead (see `RunABUpdate`'s `PatchStep.AppId`/`PatchStep.Server`), +`kubernetes.node_name` defaults to the node's real hostname (which the VM +image's Image Customizer config sets to match `TestConfig.NodeName`), and +`kubernetes.api_server` is left unset so the fake kubeconfig's own `server:` +field is used as-is. `prepareVmForAclAgent` writes only that fake +kubeconfig, then runs `systemctl enable --now trident-acl-agent.service`. +Before that runs, the service simply isn't started -- no crash-looping, no +log noise. + +## Local usage + +The VM image (`make artifacts/trident-vm-acl-agent-testimage.qcow2`) bakes in +the public key from `artifacts/id_rsa.pub`, not `~/.ssh/id_rsa.pub` -- pass +`--ssh-private-key-path` pointing at `artifacts/id_rsa` (the matching private +key) or `check-deployment` will hang/fail trying to authenticate with the +wrong key. + +```bash +make bin/storm-trident +./bin/storm-trident run aclagent deploy-vm \ + --artifacts-dir --ssh-private-key-path /id_rsa +./bin/storm-trident run aclagent check-deployment \ + --artifacts-dir --ssh-private-key-path /id_rsa +./bin/storm-trident run aclagent run-ab-update \ + --artifacts-dir --ssh-private-key-path /id_rsa +./bin/storm-trident run aclagent run-rollback \ + --artifacts-dir --ssh-private-key-path /id_rsa +./bin/storm-trident run aclagent collect-logs \ + --artifacts-dir --ssh-private-key-path /id_rsa +./bin/storm-trident run aclagent cleanup-vm \ + --artifacts-dir --ssh-private-key-path /id_rsa +``` + +Common overrides mirror other storm VM scenarios, for example: + +```bash +./bin/storm-trident run aclagent run-ab-update \ + --platform qemu \ + --artifacts-dir ./artifacts \ + --output-path ./output/aclagent \ + --ssh-private-key-path ./artifacts/id_rsa \ + --api-server-port 18080 \ + --nebraska-port 18081 +``` + +## Reboot choice + +This scenario keeps the shim-based reboot interception from the old tester. +That is less realistic than a full VM reboot, but it keeps the test deterministic +and lets the storm runner hold the reverse SSH tunnels and in-process fake services +steady while the agent drives the finalize path. + +## `run-rollback` + +`run-rollback` exercises the `rollback` annotation end-to-end against +tridentd's stable `RollbackService` gRPC API (`RollbackStage`/ +`RollbackFinalize`), followed by a real reboot and post-reboot commit - +mirroring `run-ab-update`'s stage/finalize/commit flow. It must run *after* +`run-ab-update` in the same VM lifetime, since rollback re-activates the +volume that was active before `run-ab-update`'s finalize and there is +nothing to roll back to on a freshly-deployed VM. diff --git a/tools/storm/aclagent/proxies/apiserver.go b/tools/storm/aclagent/proxies/apiserver.go new file mode 100644 index 0000000000..73edf1ba96 --- /dev/null +++ b/tools/storm/aclagent/proxies/apiserver.go @@ -0,0 +1,357 @@ +package proxies + +import ( + "context" + "encoding/json" + "fmt" + "net" + "net/http" + "strings" + "sync" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" +) + +type NodeStore struct { + mu sync.RWMutex + node *corev1.Node + watchers map[int]chan *corev1.Node + nextID int + resourceVersion int64 +} + +func NewSeedNode(name string, labels map[string]string) *corev1.Node { + seed := &corev1.Node{ + TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "Node"}, + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Labels: map[string]string{}, + Annotations: map[string]string{}, + }, + } + for key, value := range labels { + seed.Labels[key] = value + } + return seed +} + +func LoadSeedNode(data []byte) (*corev1.Node, error) { + var node corev1.Node + if err := json.Unmarshal(data, &node); err != nil { + return nil, fmt.Errorf("failed to parse seed node json: %w", err) + } + if node.Name == "" { + return nil, fmt.Errorf("seed node json must set metadata.name") + } + if node.APIVersion == "" { + node.APIVersion = "v1" + } + if node.Kind == "" { + node.Kind = "Node" + } + if node.Labels == nil { + node.Labels = map[string]string{} + } + if node.Annotations == nil { + node.Annotations = map[string]string{} + } + return &node, nil +} + +func NewNodeStore(seed *corev1.Node) *NodeStore { + node := seed.DeepCopy() + store := &NodeStore{node: node, watchers: map[int]chan *corev1.Node{}, resourceVersion: 1} + node.ResourceVersion = "1" + return store +} + +// bumpLocked increments the store's resourceVersion counter and stamps it +// onto the current node object. Every real Kubernetes object always carries +// metadata.resourceVersion, and kube-rs's watcher() rejects watch events +// (and the LIST used to bootstrap a watch) that omit it, so this must be +// set on every mutation. Callers must hold s.mu for writing. +func (s *NodeStore) bumpLocked() { + s.resourceVersion++ + s.node.ResourceVersion = fmt.Sprintf("%d", s.resourceVersion) +} + +func (s *NodeStore) Snapshot() *corev1.Node { + s.mu.RLock() + defer s.mu.RUnlock() + return s.node.DeepCopy() +} + +func (s *NodeStore) MergePatch(raw []byte) (*corev1.Node, error) { + var patch metadataPatch + if err := json.Unmarshal(raw, &patch); err != nil { + return nil, fmt.Errorf("failed to parse merge patch: %w", err) + } + + s.mu.Lock() + defer s.mu.Unlock() + applyOptionalStringMap(s.node.Labels, patch.Metadata.Labels) + applyOptionalStringMap(s.node.Annotations, patch.Metadata.Annotations) + if patch.Status.Conditions != nil { + s.node.Status.Conditions = append([]corev1.NodeCondition(nil), (*patch.Status.Conditions)...) + } + s.bumpLocked() + s.broadcastLocked() + return s.node.DeepCopy(), nil +} + +func (s *NodeStore) PatchLabels(labels map[string]string) *corev1.Node { + s.mu.Lock() + defer s.mu.Unlock() + for key, value := range labels { + s.node.Labels[key] = value + } + s.bumpLocked() + s.broadcastLocked() + return s.node.DeepCopy() +} + +func (s *NodeStore) PatchAnnotations(annotations map[string]string) *corev1.Node { + s.mu.Lock() + defer s.mu.Unlock() + for key, value := range annotations { + s.node.Annotations[key] = value + } + s.bumpLocked() + s.broadcastLocked() + return s.node.DeepCopy() +} + +func (s *NodeStore) SetReadyCondition(ready bool) *corev1.Node { + s.mu.Lock() + defer s.mu.Unlock() + status := corev1.ConditionFalse + message := "Simulated reboot in progress" + reason := "TridentACLAgentTesterReboot" + if ready { + status = corev1.ConditionTrue + message = "Node ready" + reason = "TridentACLAgentTesterReady" + } + s.node.Status.Conditions = []corev1.NodeCondition{{ + Type: corev1.NodeReady, + Status: status, + LastHeartbeatTime: metav1.Now(), + LastTransitionTime: metav1.Now(), + Reason: reason, + Message: message, + }} + s.bumpLocked() + s.broadcastLocked() + return s.node.DeepCopy() +} + +func (s *NodeStore) Subscribe() (int, <-chan *corev1.Node, *corev1.Node) { + s.mu.Lock() + defer s.mu.Unlock() + id := s.nextID + s.nextID++ + ch := make(chan *corev1.Node, 8) + s.watchers[id] = ch + return id, ch, s.node.DeepCopy() +} + +func (s *NodeStore) Unsubscribe(id int) { + s.mu.Lock() + defer s.mu.Unlock() + if ch, ok := s.watchers[id]; ok { + delete(s.watchers, id) + close(ch) + } +} + +func (s *NodeStore) broadcastLocked() { + snapshot := s.node.DeepCopy() + for _, ch := range s.watchers { + select { + case ch <- snapshot.DeepCopy(): + default: + } + } +} + +type APIServer struct { + nodeName string + store *NodeStore +} + +func NewAPIServer(nodeName string, store *NodeStore) *APIServer { + return &APIServer{nodeName: nodeName, store: store} +} + +func (s *APIServer) Handler() http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/v1/nodes": + // Collection endpoint. kube-rs's watcher always performs an + // initial LIST here (optionally filtered by fieldSelector) before + // switching to a watch on the same collection; both must be + // served or the watcher treats the 404 as fatal and the process + // exits, taking down the whole service. + if r.Method != http.MethodGet { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + if r.URL.Query().Get("watch") == "true" { + s.handleWatch(w, r) + return + } + s.handleList(w, r) + return + case "/api/v1/nodes/" + s.nodeName: + if r.Method == http.MethodGet && r.URL.Query().Get("watch") == "true" { + s.handleWatch(w, r) + return + } + switch r.Method { + case http.MethodGet: + s.handleGet(w, r) + case http.MethodPatch: + s.handlePatch(w, r) + default: + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + } + return + default: + http.NotFound(w, r) + } + }) +} + +func (s *APIServer) ListenAndServe(ctx context.Context, listenAddr string) (net.Listener, error) { + listener, err := net.Listen("tcp", listenAddr) + if err != nil { + return nil, fmt.Errorf("failed to listen on %s: %w", listenAddr, err) + } + server := &http.Server{Handler: s.Handler()} + go func() { + <-ctx.Done() + _ = server.Shutdown(context.Background()) + }() + go func() { _ = server.Serve(listener) }() + return listener, nil +} + +func (s *APIServer) handleGet(w http.ResponseWriter, _ *http.Request) { + writeJSON(w, http.StatusOK, s.store.Snapshot()) +} + +// handleList serves the collection endpoint's plain (non-watch) LIST +// request. kube-rs's watcher() issues this before it ever opens a watch +// stream, so it must return a well-formed NodeList (including +// metadata.resourceVersion) even though this fake only ever tracks one node. +func (s *APIServer) handleList(w http.ResponseWriter, r *http.Request) { + node := s.store.Snapshot() + items := []corev1.Node{} + if selector := r.URL.Query().Get("fieldSelector"); selector != "" { + if selector == "metadata.name="+s.nodeName { + items = append(items, *node) + } + } else { + items = append(items, *node) + } + list := corev1.NodeList{ + TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "NodeList"}, + ListMeta: metav1.ListMeta{ResourceVersion: node.ResourceVersion}, + Items: items, + } + writeJSON(w, http.StatusOK, &list) +} + +func (s *APIServer) handlePatch(w http.ResponseWriter, r *http.Request) { + if contentType := r.Header.Get("Content-Type"); contentType != "" && !strings.Contains(contentType, "merge-patch+json") { + http.Error(w, "expected application/merge-patch+json", http.StatusUnsupportedMediaType) + return + } + defer r.Body.Close() + body := json.NewDecoder(r.Body) + body.DisallowUnknownFields() + var raw map[string]any + if err := body.Decode(&raw); err != nil { + http.Error(w, fmt.Sprintf("invalid patch body: %v", err), http.StatusBadRequest) + return + } + bytes, err := json.Marshal(raw) + if err != nil { + http.Error(w, fmt.Sprintf("failed to re-marshal patch: %v", err), http.StatusInternalServerError) + return + } + updated, err := s.store.MergePatch(bytes) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + writeJSON(w, http.StatusOK, updated) +} + +func (s *APIServer) handleWatch(w http.ResponseWriter, r *http.Request) { + flusher, ok := w.(http.Flusher) + if !ok { + http.Error(w, "streaming unsupported", http.StatusInternalServerError) + return + } + watchID, ch, current := s.store.Subscribe() + defer s.store.Unsubscribe(watchID) + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + if err := writeWatchEvent(w, "ADDED", current); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + flusher.Flush() + for { + select { + case <-r.Context().Done(): + return + case node, ok := <-ch: + if !ok { + return + } + if err := writeWatchEvent(w, "MODIFIED", node); err != nil { + return + } + flusher.Flush() + } + } +} + +func writeWatchEvent(w http.ResponseWriter, eventType string, node *corev1.Node) error { + raw, err := json.Marshal(node) + if err != nil { + return err + } + event := metav1.WatchEvent{Type: eventType, Object: runtime.RawExtension{Raw: raw}} + return json.NewEncoder(w).Encode(&event) +} + +func writeJSON(w http.ResponseWriter, status int, value any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _ = json.NewEncoder(w).Encode(value) +} + +type metadataPatch struct { + Metadata struct { + Labels map[string]*string `json:"labels"` + Annotations map[string]*string `json:"annotations"` + } `json:"metadata"` + Status struct { + Conditions *[]corev1.NodeCondition `json:"conditions"` + } `json:"status"` +} + +func applyOptionalStringMap(target map[string]string, patch map[string]*string) { + for key, value := range patch { + if value == nil { + delete(target, key) + continue + } + target[key] = *value + } +} diff --git a/tools/storm/aclagent/proxies/constants.go b/tools/storm/aclagent/proxies/constants.go new file mode 100644 index 0000000000..f72063ec16 --- /dev/null +++ b/tools/storm/aclagent/proxies/constants.go @@ -0,0 +1,11 @@ +package proxies + +const ( + UpdateRequestAnnotation = "acl.azure.com/update-request" + UpdateStatusAnnotation = "acl.azure.com/update-status" + UpdateCommitStatusAnnotation = "acl.azure.com/update-commit-status" + NodeImageVersionLabel = "kubernetes.azure.com/node-image-version" + + DefaultNodeName = "trident-node" + DefaultMarkerFile = "./trident-acl-agent-reboot-signal" +) diff --git a/tools/storm/aclagent/proxies/imageserver.go b/tools/storm/aclagent/proxies/imageserver.go new file mode 100644 index 0000000000..94da81076c --- /dev/null +++ b/tools/storm/aclagent/proxies/imageserver.go @@ -0,0 +1,57 @@ +package proxies + +import ( + "context" + "fmt" + "net" + "net/http" + "path/filepath" +) + +// ImageServer serves a single OS update image (e.g. a .cosi file) over plain +// HTTP so trident-acl-agent's fake Nebraska endpoint can point tridentd at a +// real, downloadable artifact during A/B update staging. tridentd downloads +// the image itself (not the acl-agent), so this just needs to serve the raw +// bytes at a stable path. +type ImageServer struct { + // ImagePath is the local filesystem path to the image file to serve. + ImagePath string +} + +// Handler returns an http.Handler that serves ImagePath at the request path's +// base name (e.g. "/acl.cosi"), regardless of the requested path, so it works +// whether the caller mounts it at "/" or "/images/". +func (s *ImageServer) Handler() http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet && r.Method != http.MethodHead { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + http.ServeFile(w, r, s.ImagePath) + }) +} + +// ListenAndServe starts the image server on listenAddr and serves the image +// at every path under the given package name (so codebase + package name +// joins correctly regardless of trailing slash handling). +func (s *ImageServer) ListenAndServe(ctx context.Context, listenAddr string) (net.Listener, error) { + listener, err := net.Listen("tcp", listenAddr) + if err != nil { + return nil, fmt.Errorf("failed to listen on %s: %w", listenAddr, err) + } + mux := http.NewServeMux() + mux.Handle("/", s.Handler()) + server := &http.Server{Handler: mux} + go func() { + <-ctx.Done() + _ = server.Shutdown(context.Background()) + }() + go func() { _ = server.Serve(listener) }() + return listener, nil +} + +// PackageBaseName returns the file name portion of ImagePath, used as both +// the Nebraska package name and the served URL path segment. +func (s *ImageServer) PackageBaseName() string { + return filepath.Base(s.ImagePath) +} diff --git a/tools/storm/aclagent/proxies/kubelet.go b/tools/storm/aclagent/proxies/kubelet.go new file mode 100644 index 0000000000..36522987c0 --- /dev/null +++ b/tools/storm/aclagent/proxies/kubelet.go @@ -0,0 +1,183 @@ +package proxies + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http" + "os" + "path/filepath" + "strings" + "time" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +const RebootStateAnnotation = "trident-acl-agent/reboot-state" + +type KubeletProxy struct { + HTTPClient *http.Client + APIServerURL string + NodeName string + NodeStore *NodeStore + BootstrapLabels map[string]string + MarkerFile string + RebootDuration time.Duration +} + +func (k *KubeletProxy) Run(ctx context.Context) error { + if k.NodeStore != nil { + if len(k.BootstrapLabels) > 0 { + k.NodeStore.PatchLabels(k.BootstrapLabels) + } + k.NodeStore.SetReadyCondition(true) + } else { + if len(k.BootstrapLabels) > 0 { + if err := patchStringMap(ctx, k.client(), k.nodeURL(), "labels", k.BootstrapLabels); err != nil { + return err + } + } + if err := patchReadyCondition(ctx, k.client(), k.nodeURL(), true); err != nil { + return err + } + } + + if k.MarkerFile == "" { + k.MarkerFile = DefaultMarkerFile + } + if k.RebootDuration <= 0 { + k.RebootDuration = 30 * time.Second + } + + ticker := time.NewTicker(1 * time.Second) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-ticker.C: + if _, err := os.Stat(k.MarkerFile); err == nil { + if k.NodeStore != nil { + k.NodeStore.SetReadyCondition(false) + k.NodeStore.PatchAnnotations(map[string]string{RebootStateAnnotation: "not-ready"}) + } else { + if err := patchReadyCondition(ctx, k.client(), k.nodeURL(), false); err != nil { + return err + } + if err := patchStringMap(ctx, k.client(), k.nodeURL(), "annotations", map[string]string{RebootStateAnnotation: "not-ready"}); err != nil { + return err + } + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(k.RebootDuration): + } + if k.NodeStore != nil { + k.NodeStore.SetReadyCondition(true) + k.NodeStore.PatchAnnotations(map[string]string{RebootStateAnnotation: "ready"}) + } else { + if err := patchReadyCondition(ctx, k.client(), k.nodeURL(), true); err != nil { + return err + } + if err := patchStringMap(ctx, k.client(), k.nodeURL(), "annotations", map[string]string{RebootStateAnnotation: "ready"}); err != nil { + return err + } + } + if err := os.Remove(k.MarkerFile); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("failed to remove reboot marker %s: %w", k.MarkerFile, err) + } + } + } + } +} + +func WriteRebootMarker(markerFile string) error { + if markerFile == "" { + markerFile = DefaultMarkerFile + } + if err := os.MkdirAll(filepath.Dir(markerFile), 0o755); err != nil { + return fmt.Errorf("failed to create reboot marker directory: %w", err) + } + return os.WriteFile(markerFile, []byte("reboot-requested\n"), 0o644) +} + +func (k *KubeletProxy) nodeURL() string { + return strings.TrimRight(k.APIServerURL, "/") + "/api/v1/nodes/" + k.NodeName +} + +func (k *KubeletProxy) client() *http.Client { + if k.HTTPClient != nil { + return k.HTTPClient + } + return http.DefaultClient +} + +func patchStringMap(ctx context.Context, client *http.Client, nodeURL string, field string, values map[string]string) error { + body, err := json.Marshal(map[string]any{ + "metadata": map[string]any{field: values}, + }) + if err != nil { + return err + } + request, err := http.NewRequestWithContext(ctx, http.MethodPatch, nodeURL, bytes.NewReader(body)) + if err != nil { + return err + } + request.Header.Set("Content-Type", "application/merge-patch+json") + response, err := client.Do(request) + if err != nil { + return err + } + defer response.Body.Close() + if response.StatusCode >= 300 { + return fmt.Errorf("fake apiserver patch failed with %s", response.Status) + } + return nil +} + +func patchReadyCondition(ctx context.Context, client *http.Client, nodeURL string, ready bool) error { + status := corev1.ConditionFalse + message := "Simulated reboot in progress" + reason := "TridentACLAgentTesterReboot" + if ready { + status = corev1.ConditionTrue + message = "Node ready" + reason = "TridentACLAgentTesterReady" + } + + condition := corev1.NodeCondition{ + Type: corev1.NodeReady, + Status: status, + LastHeartbeatTime: metav1.Now(), + LastTransitionTime: metav1.Now(), + Reason: reason, + Message: message, + } + + body, err := json.Marshal(map[string]any{ + "status": map[string]any{ + "conditions": []corev1.NodeCondition{condition}, + }, + }) + if err != nil { + return err + } + request, err := http.NewRequestWithContext(ctx, http.MethodPatch, nodeURL, bytes.NewReader(body)) + if err != nil { + return err + } + request.Header.Set("Content-Type", "application/merge-patch+json") + response, err := client.Do(request) + if err != nil { + return err + } + defer response.Body.Close() + if response.StatusCode >= 300 { + return fmt.Errorf("fake apiserver status patch failed with %s", response.Status) + } + return nil +} diff --git a/tools/storm/aclagent/proxies/nebraska.go b/tools/storm/aclagent/proxies/nebraska.go new file mode 100644 index 0000000000..a9f024320c --- /dev/null +++ b/tools/storm/aclagent/proxies/nebraska.go @@ -0,0 +1,426 @@ +package proxies + +import ( + "bytes" + "context" + "database/sql" + "fmt" + "net" + "net/http" + "os" + "os/exec" + "strings" + "time" + + "github.com/flatcar/nebraska/backend/pkg/api" + "github.com/flatcar/nebraska/backend/pkg/api/admin" + "github.com/flatcar/nebraska/backend/pkg/omaha" + "gopkg.in/guregu/null.v4" + "gopkg.in/yaml.v3" +) + +// nebraskaTrack is the group's Track this proxy seeds. Now supplied to +// trident-acl-agent per-request via the update-request annotation's `track` +// field (see Track() and RunABUpdate's PatchStep.Track) rather than needing +// to match DEFAULT_NEBRASKA_TRACK, since the agent no longer has any +// config-file fallback for it. +const nebraskaTrack = "west-us" + +// defaultTeamID is the team seeded by Nebraska's own db migrations +// (0005_default_team_id.sql); application rows have a NOT NULL team_id FK. +const defaultTeamID = "d89342dc-9214-441d-a4af-bdd837a3b239" + +// noUpdateVersion is a sentinel package version used when Scenario.Available +// is false. Leaving a channel with no package at all makes real Nebraska +// return ErrNoPackageFound, which maps to "error-noPackageFound" - a hard +// error trident-acl-agent's client treats as a failure, not "no update +// available". Seeding a package this far below any real image version +// instead exercises Nebraska's actual semver-comparison path in +// GetUpdatePackage and still cleanly yields "noupdate", since the +// instance's real reported version can never be lower than 0.0.1. +const noUpdateVersion = "0.0.1" + +type NebraskaScenario struct { + Available bool `yaml:"available"` + Version string `yaml:"version,omitempty"` + URL string `yaml:"url,omitempty"` + SHA384 string `yaml:"sha384,omitempty"` + PackageName string `yaml:"package-name,omitempty"` +} + +func LoadNebraskaScenario(path string) (*NebraskaScenario, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("failed to read Nebraska scenario %s: %w", path, err) + } + var scenario NebraskaScenario + if err := yaml.Unmarshal(data, &scenario); err != nil { + return nil, fmt.Errorf("failed to parse Nebraska scenario yaml: %w", err) + } + return &scenario, nil +} + +// NebraskaProxy wraps the real github.com/flatcar/nebraska/backend Omaha +// server, backed by an ephemeral Postgres container that this proxy manages +// itself, instead of a hand-rolled fake. This exercises trident-acl-agent +// against Nebraska's actual instance/event/update-grant state machine +// (RegisterInstance, RegisterEvent, GetUpdatePackage's in-progress gating +// and semver-driven grant/completion logic) rather than an approximation of +// it, at the cost of needing Docker plus a couple seconds of container +// startup/migration time per run. +type NebraskaProxy struct { + Scenario *NebraskaScenario + + containerID string + dbURL string + api *api.API + handler *omaha.Handler + appID string +} + +// ExpectedUpdateStatusSequence is the ordered sequence of Nebraska instance +// statuses trident-acl-agent's real update flow is expected to drive the +// seeded instance through end-to-end: GetUpdatePackage grants the update +// (UpdateGranted) on the first check, stage reports DownloadStarted/ +// DownloadFinished (Downloading/Downloaded), finalize reports Installed +// (Installed), and the post-reboot commit reports the terminal Completed +// event (Complete). Pass to ValidateStatusHistory after a full run-ab-update +// scenario completes. +var ExpectedUpdateStatusSequence = []int{ + api.InstanceStatusUpdateGranted, + api.InstanceStatusDownloading, + api.InstanceStatusDownloaded, + api.InstanceStatusInstalled, + api.InstanceStatusComplete, +} + +// AppID returns the DB-generated application ID trident-acl-agent must be +// configured with. Real Nebraska generates application IDs server-side +// (admin.Service.AddApp does not accept a caller-supplied ID), so callers +// must read this back after ListenAndServe seeds the app, rather than +// hardcoding an app_id string as the old fake mock allowed. +func (p *NebraskaProxy) AppID() string { + return p.appID +} + +// Track returns the group Track trident-acl-agent must be configured with +// (via the update-request annotation's `track` field), matching the group +// this proxy seeded in seed(). +func (p *NebraskaProxy) Track() string { + return nebraskaTrack +} + +func (p *NebraskaProxy) Handler() http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + defer r.Body.Close() + ip := r.RemoteAddr + if host, _, err := net.SplitHostPort(r.RemoteAddr); err == nil { + ip = host + } + var buf bytes.Buffer + if err := p.handler.Handle(r.Body, &buf, ip); err != nil { + http.Error(w, fmt.Sprintf("nebraska omaha handler error: %v", err), http.StatusInternalServerError) + return + } + w.Header().Set("Content-Type", "application/xml") + _, _ = w.Write(buf.Bytes()) + }) +} + +// ListenAndServe starts an ephemeral Postgres container, applies Nebraska's +// db migrations against it, seeds an application/package/channel/group +// matching p.Scenario, and starts serving the real Nebraska Omaha handler on +// listenAddr. The Postgres container and http server are both torn down +// when ctx is cancelled. +func (p *NebraskaProxy) ListenAndServe(ctx context.Context, listenAddr string) (net.Listener, error) { + dbURL, containerID, err := startEphemeralPostgres(ctx) + if err != nil { + return nil, fmt.Errorf("failed to start ephemeral Postgres for Nebraska: %w", err) + } + p.containerID = containerID + p.dbURL = dbURL + + // api.New only reads NEBRASKA_DB_URL from the environment - there is no + // functional option to set a custom DSN - so this is the only way to + // point it at our ephemeral container. Safe here because exactly one + // NebraskaProxy is ever instantiated per storm test process. + if err := os.Setenv("NEBRASKA_DB_URL", dbURL); err != nil { + stopEphemeralPostgres(containerID) + return nil, fmt.Errorf("failed to set NEBRASKA_DB_URL: %w", err) + } + + a, err := api.NewWithMigrations(api.OptionInitDB) + if err != nil { + stopEphemeralPostgres(containerID) + return nil, fmt.Errorf("failed to initialize Nebraska API against ephemeral Postgres: %w", err) + } + p.api = a + p.handler = omaha.NewHandler(a) + + if err := p.seed(); err != nil { + stopEphemeralPostgres(containerID) + return nil, fmt.Errorf("failed to seed Nebraska scenario: %w", err) + } + + listener, err := net.Listen("tcp", listenAddr) + if err != nil { + stopEphemeralPostgres(containerID) + return nil, fmt.Errorf("failed to listen on %s: %w", listenAddr, err) + } + server := &http.Server{Handler: p.Handler()} + go func() { + <-ctx.Done() + _ = server.Shutdown(context.Background()) + stopEphemeralPostgres(containerID) + }() + go func() { _ = server.Serve(listener) }() + return listener, nil +} + +// seed creates the application/package/channel/group real Nebraska needs to +// answer update checks, per p.Scenario. The package version is the +// scenario's target version when an update should be offered, or the +// noUpdateVersion sentinel otherwise (see its doc comment). +func (p *NebraskaProxy) seed() error { + svc := admin.NewService(p.api.Reads()) + + app, err := svc.AddApp(&api.Application{ + Name: "trident-acl-agent-storm-test", + TeamID: defaultTeamID, + }) + if err != nil { + return fmt.Errorf("failed to seed application: %w", err) + } + p.appID = app.ID + + version := p.Scenario.Version + if version == "" { + version = "1.0.0" + } + if !p.Scenario.Available { + version = noUpdateVersion + } + packageName := p.Scenario.PackageName + if packageName == "" { + packageName = "acl.cosi" + } + baseURL := p.Scenario.URL + if baseURL == "" { + baseURL = "https://example.invalid/images/" + } + + // The package.hash column is varchar(64) (sized for base64 SHA1 or hex + // SHA256, what real Nebraska/Omaha packages normally carry), but + // Scenario.SHA384 is a 96-character hex string and would overflow it. + // trident-acl-agent's Package wire struct doesn't even parse this field + // (see crates/trident-acl-agent/src/nebraska/wire.rs) - image integrity + // is checked via the COSI metadata instead - so it's safe to leave + // unset here rather than truncate it into something misleading. + pkg, err := svc.AddPackage(&api.Package{ + Type: api.PkgTypeOther, + URL: baseURL, + Version: version, + Filename: null.StringFrom(packageName), + ApplicationID: app.ID, + Arch: api.ArchAMD64, + }) + if err != nil { + return fmt.Errorf("failed to seed package: %w", err) + } + + channel, err := svc.AddChannel(&api.Channel{ + Name: "storm", + ApplicationID: app.ID, + PackageID: null.StringFrom(pkg.ID), + Arch: api.ArchAMD64, + }) + if err != nil { + return fmt.Errorf("failed to seed channel: %w", err) + } + + if _, err := svc.AddGroup(&api.Group{ + Name: "storm", + ApplicationID: app.ID, + ChannelID: null.StringFrom(channel.ID), + Track: nebraskaTrack, + PolicyUpdatesEnabled: true, + PolicyPeriodInterval: "15 minutes", + PolicyMaxUpdatesPerPeriod: 100, + PolicyUpdateTimeout: "60 minutes", + }); err != nil { + return fmt.Errorf("failed to seed group: %w", err) + } + + return nil +} + +// StatusHistory returns every status the seeded application's instance(s) +// have transitioned through, in chronological order, read directly from +// Nebraska's instance_status_history table. There's no dbreads query for +// "history across all instances of an app" (only a single-instance one that +// takes an instance ID, which storm doesn't predict - it's derived from +// hashing the VM's /etc/machine-id, see crates/trident-acl-agent/src/lib.rs), +// so this queries the table by application_id directly instead, using a +// fresh connection to the same ephemeral Postgres container ListenAndServe +// started. +func (p *NebraskaProxy) StatusHistory() ([]int, error) { + db, err := sql.Open("pgx", p.dbURL) + if err != nil { + return nil, fmt.Errorf("failed to open Nebraska db for status history query: %w", err) + } + defer db.Close() + + rows, err := db.Query( + `select status from instance_status_history where application_id = $1 order by created_ts asc, id asc`, + p.appID, + ) + if err != nil { + return nil, fmt.Errorf("failed to query instance_status_history: %w", err) + } + defer rows.Close() + + var statuses []int + for rows.Next() { + var status int + if err := rows.Scan(&status); err != nil { + return nil, fmt.Errorf("failed to scan instance_status_history row: %w", err) + } + statuses = append(statuses, status) + } + return statuses, rows.Err() +} + +// ValidateStatusHistory asserts that the seeded application's real Nebraska +// instance_status_history exactly matches want, in order. Nebraska only ever +// appends a new history row when an instance's status actually changes (see +// updateInstanceData in the vendored api package), so this is a stable, +// duplicate-free ordering to assert against - it fails if trident-acl-agent +// skips a status transition, reports one out of order, or a status update +// silently gets rejected/ignored by Nebraska (e.g. sent while no update is +// in progress). +func (p *NebraskaProxy) ValidateStatusHistory(want []int) error { + got, err := p.StatusHistory() + if err != nil { + return fmt.Errorf("failed to validate Nebraska instance status history: %w", err) + } + if len(got) != len(want) { + return fmt.Errorf("unexpected Nebraska instance status history: want %s, got %s", formatStatuses(want), formatStatuses(got)) + } + for i := range want { + if got[i] != want[i] { + return fmt.Errorf("unexpected Nebraska instance status history: want %s, got %s", formatStatuses(want), formatStatuses(got)) + } + } + return nil +} + +// statusName renders a Nebraska instance status int using its api.InstanceStatus* +// name, for readable ValidateStatusHistory error messages. +func statusName(status int) string { + switch status { + case api.InstanceStatusUndefined: + return "Undefined" + case api.InstanceStatusUpdateGranted: + return "UpdateGranted" + case api.InstanceStatusError: + return "Error" + case api.InstanceStatusComplete: + return "Complete" + case api.InstanceStatusInstalled: + return "Installed" + case api.InstanceStatusDownloaded: + return "Downloaded" + case api.InstanceStatusDownloading: + return "Downloading" + case api.InstanceStatusOnHold: + return "OnHold" + default: + return fmt.Sprintf("Unknown(%d)", status) + } +} + +func formatStatuses(statuses []int) string { + names := make([]string, len(statuses)) + for i, s := range statuses { + names[i] = statusName(s) + } + return "[" + strings.Join(names, " -> ") + "]" +} + +// startEphemeralPostgres starts a disposable Postgres container for this +// Nebraska instance to use, waits for it to accept connections, and returns +// a connection URL for it plus its container ID (for teardown). +func startEphemeralPostgres(ctx context.Context) (dbURL string, containerID string, err error) { + out, err := exec.CommandContext(ctx, "docker", "run", "-d", "--rm", + "-e", "POSTGRES_PASSWORD=nebraska", + "-e", "POSTGRES_DB=nebraska", + "-p", "127.0.0.1::5432", + "postgres:16-alpine", + ).Output() + if err != nil { + return "", "", fmt.Errorf("docker run postgres: %w", err) + } + containerID = strings.TrimSpace(string(out)) + + portOut, err := exec.CommandContext(ctx, "docker", "port", containerID, "5432/tcp").Output() + if err != nil { + stopEphemeralPostgres(containerID) + return "", "", fmt.Errorf("docker port: %w", err) + } + // docker port prints e.g. "127.0.0.1:32771" (or several such lines); + // take the port from the last colon-separated field of the first line. + firstLine := strings.SplitN(strings.TrimSpace(string(portOut)), "\n", 2)[0] + fields := strings.Split(firstLine, ":") + port := fields[len(fields)-1] + + dbURL = fmt.Sprintf("postgres://postgres:nebraska@127.0.0.1:%s/nebraska?sslmode=disable&connect_timeout=10", port) + + if err := waitForPostgres(ctx, dbURL); err != nil { + stopEphemeralPostgres(containerID) + return "", "", err + } + + return dbURL, containerID, nil +} + +// waitForPostgres polls dbURL until it accepts connections or ctx/deadline +// expires. Relies on the "pgx" driver already being registered with +// database/sql as a side effect of importing github.com/flatcar/nebraska's +// api package (which blank-imports github.com/jackc/pgx/v5/stdlib). +func waitForPostgres(ctx context.Context, dbURL string) error { + deadline := time.Now().Add(30 * time.Second) + var lastErr error + for time.Now().Before(deadline) { + if err := pingOnce(dbURL); err != nil { + lastErr = err + } else { + return nil + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(500 * time.Millisecond): + } + } + return fmt.Errorf("timed out waiting for ephemeral Postgres to accept connections: %w", lastErr) +} + +func pingOnce(dbURL string) error { + db, err := sql.Open("pgx", dbURL) + if err != nil { + return err + } + defer db.Close() + return db.Ping() +} + +func stopEphemeralPostgres(containerID string) { + if containerID == "" { + return + } + _ = exec.Command("docker", "rm", "-f", containerID).Run() +} diff --git a/tools/storm/aclagent/proxies/nebraska_test.go b/tools/storm/aclagent/proxies/nebraska_test.go new file mode 100644 index 0000000000..03484613ac --- /dev/null +++ b/tools/storm/aclagent/proxies/nebraska_test.go @@ -0,0 +1,167 @@ +package proxies + +import ( + "context" + "fmt" + "io" + "net/http" + "os/exec" + "strings" + "testing" + + "github.com/flatcar/nebraska/backend/pkg/api" +) + +// requireDocker skips the test when Docker isn't available, so this test +// doesn't hard-fail on machines/CI runners without it. NebraskaProxy always +// needs a real ephemeral Postgres container - there is no in-memory +// alternative for github.com/flatcar/nebraska/backend. +func requireDocker(t *testing.T) { + t.Helper() + if _, err := exec.LookPath("docker"); err != nil { + t.Skip("docker not available; skipping test that needs an ephemeral Postgres container") + } +} + +func postUpdateCheck(t *testing.T, addr, appID, machineID string) string { + t.Helper() + req := fmt.Sprintf(` + + + + +`, appID, machineID) + + resp, err := http.Post(fmt.Sprintf("http://%s/", addr), "text/xml", strings.NewReader(req)) + if err != nil { + t.Fatalf("POST updatecheck: %v", err) + } + defer resp.Body.Close() + body, _ := io.ReadAll(resp.Body) + if resp.StatusCode != http.StatusOK { + t.Fatalf("expected 200, got %d: %s", resp.StatusCode, body) + } + return string(body) +} + +// postEvent posts a bare progress/terminal event, mirroring the (eventtype, +// eventresult) requests trident-acl-agent's nebraska::Client sends during +// stage/finalize/post-reboot-commit (see +// crates/trident-acl-agent/src/nebraska/event.rs for the whitelisted pairs). +func postEvent(t *testing.T, addr, appID, machineID string, eventType, eventResult int) string { + t.Helper() + req := fmt.Sprintf(` + + + + +`, appID, machineID, eventType, eventResult) + + resp, err := http.Post(fmt.Sprintf("http://%s/", addr), "text/xml", strings.NewReader(req)) + if err != nil { + t.Fatalf("POST event(%d,%d): %v", eventType, eventResult, err) + } + defer resp.Body.Close() + body, _ := io.ReadAll(resp.Body) + if resp.StatusCode != http.StatusOK { + t.Fatalf("expected 200, got %d: %s", resp.StatusCode, body) + } + return string(body) +} + +// TestNebraskaProxy exercises NebraskaProxy against the real +// github.com/flatcar/nebraska/backend Omaha handler and an ephemeral +// Postgres container (started/torn down per subtest), rather than a +// hand-rolled fake. Each subtest takes roughly 10-15s due to container +// startup and db migrations - slow for a unit test, but this is what +// validates the mock's seeding logic (app/package/channel/group, track +// matching, semver-driven grant/noupdate) independently of the full +// storm-trident VM suite, which additionally exercises the real +// trident-acl-agent binary end-to-end. +func TestNebraskaProxy(t *testing.T) { + requireDocker(t) + + t.Run("update-available", func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + + p := &NebraskaProxy{Scenario: &NebraskaScenario{ + Available: true, + Version: "5.0.0", + URL: "http://example.invalid/images/", + SHA384: "deadbeef", + PackageName: "acl.cosi", + }} + listener, err := p.ListenAndServe(ctx, "127.0.0.1:0") + if err != nil { + t.Fatalf("ListenAndServe: %v", err) + } + + if p.AppID() == "" { + t.Fatal("expected non-empty AppID after seeding") + } + + body := postUpdateCheck(t, listener.Addr().String(), p.AppID(), "smoke-test-machine") + if !strings.Contains(body, `status="ok"`) { + t.Fatalf("expected an ok updatecheck offering the package, got: %s", body) + } + if !strings.Contains(body, "5.0.0") { + t.Fatalf("expected manifest version 5.0.0 in response, got: %s", body) + } + }) + + t.Run("no-update", func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + + p := &NebraskaProxy{Scenario: &NebraskaScenario{Available: false}} + listener, err := p.ListenAndServe(ctx, "127.0.0.1:0") + if err != nil { + t.Fatalf("ListenAndServe: %v", err) + } + + body := postUpdateCheck(t, listener.Addr().String(), p.AppID(), "smoke-test-machine-2") + if !strings.Contains(body, `status="noupdate"`) { + t.Fatalf("expected noupdate status, got: %s", body) + } + }) + + t.Run("status-history-matches-full-update-sequence", func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + + p := &NebraskaProxy{Scenario: &NebraskaScenario{ + Available: true, + Version: "5.0.0", + URL: "http://example.invalid/images/", + SHA384: "deadbeef", + PackageName: "acl.cosi", + }} + listener, err := p.ListenAndServe(ctx, "127.0.0.1:0") + if err != nil { + t.Fatalf("ListenAndServe: %v", err) + } + addr := listener.Addr().String() + machineID := "status-history-machine" + + // Drive the instance through exactly the same request sequence a + // real trident-acl-agent run-ab-update does: an updatecheck grants + // the update, then stage/finalize/post-reboot-commit each report one + // progress or terminal event (see event.rs's whitelisted pairs). + postUpdateCheck(t, addr, p.AppID(), machineID) + postEvent(t, addr, p.AppID(), machineID, 13, 1) // DownloadStarted + postEvent(t, addr, p.AppID(), machineID, 14, 1) // DownloadFinished + postEvent(t, addr, p.AppID(), machineID, 800, 1) // Installed + postEvent(t, addr, p.AppID(), machineID, 3, 2) // Completed (success+reboot) + + if err := p.ValidateStatusHistory(ExpectedUpdateStatusSequence); err != nil { + t.Fatalf("ValidateStatusHistory: %v", err) + } + + // A truncated/reordered history must be rejected, not silently + // accepted - otherwise ValidateStatusHistory would be a no-op check. + if err := p.ValidateStatusHistory([]int{api.InstanceStatusUpdateGranted}); err == nil { + t.Fatal("expected ValidateStatusHistory to reject a truncated status sequence") + } + }) +} diff --git a/tools/storm/aclagent/proxies/rp.go b/tools/storm/aclagent/proxies/rp.go new file mode 100644 index 0000000000..2d61ed0df4 --- /dev/null +++ b/tools/storm/aclagent/proxies/rp.go @@ -0,0 +1,179 @@ +package proxies + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http" + "strings" + "time" + + corev1 "k8s.io/api/core/v1" +) + +type RPClient struct { + HTTPClient *http.Client + APIServerURL string + NodeName string +} + +type updateRequest struct { + SchemaVersion string `json:"schemaVersion"` + NodeUpdateID string `json:"nodeUpdateId"` + OperationID string `json:"operationId"` + Operation string `json:"operation"` + TargetVersion string `json:"targetVersion,omitempty"` + Server string `json:"server,omitempty"` + AppId string `json:"appId,omitempty"` + Track string `json:"track,omitempty"` +} + +type updateStatus struct { + SchemaVersion string `json:"schemaVersion"` + NodeUpdateID string `json:"nodeUpdateId"` + OperationID string `json:"operationId"` + Operation string `json:"operation"` + Code string `json:"code"` +} + +func (c *RPClient) RunScenario(ctx context.Context, scenario *Scenario) (*ScenarioReport, error) { + report := &ScenarioReport{Steps: make([]StepReport, 0, len(scenario.Steps)), Passed: true} + for index, step := range scenario.Steps { + started := time.Now() + stepReport, err := c.runStep(ctx, index, step) + if err != nil { + return nil, err + } + stepReport.ElapsedMS = time.Since(started).Milliseconds() + report.Steps = append(report.Steps, *stepReport) + report.Passed = report.Passed && stepReport.Passed + } + return report, nil +} + +func (c *RPClient) runStep(ctx context.Context, index int, step ScenarioStep) (*StepReport, error) { + switch { + case step.Patch != nil: + if err := c.patchNodeRequest(ctx, step.Patch); err != nil { + return nil, err + } + return &StepReport{Index: index, Kind: "patch", Passed: true, Message: "patched fake Node request annotation"}, nil + case step.Expect != nil: + return c.expectStatus(ctx, index, step.Expect) + default: + return nil, fmt.Errorf("step %d had no recognized action", index) + } +} + +func (c *RPClient) expectStatus(ctx context.Context, index int, step *ExpectStep) (*StepReport, error) { + deadline := time.Now().Add(step.Timeout) + pollInterval := 500 * time.Millisecond + annotationKey := UpdateStatusAnnotation + if step.Operation == "commit" { + annotationKey = UpdateCommitStatusAnnotation + } + var lastObserved map[string]string + matched := false + for time.Now().Before(deadline) { + node, err := c.getNode(ctx) + if err != nil { + return nil, err + } + status, _ := decodeStatus(node, annotationKey) + if status != nil { + lastObserved = map[string]string{"operation-id": status.OperationID, "operation": status.Operation, "code": status.Code} + if status.Code == step.Code && (step.OperationID == "" || status.OperationID == step.OperationID) && (step.Operation == "" || status.Operation == step.Operation) { + matched = true + break + } + } + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-time.After(pollInterval): + } + } + passed := matched + message := "observed expected status" + if step.ExpectTimeout { + passed = !matched + if passed { + message = "timed out as expected" + } else { + message = "expected no matching status before timeout, but status matched" + } + } else if !passed { + message = "status expectation failed" + } + return &StepReport{Index: index, Kind: "expect", Passed: passed, Message: message, Expected: map[string]any{"operation-id": step.OperationID, "operation": step.Operation, "code": step.Code, "timeout": step.Timeout.String()}, Actual: lastObserved}, nil +} + +func (c *RPClient) patchNodeRequest(ctx context.Context, step *PatchStep) error { + request := updateRequest{SchemaVersion: "1.0", NodeUpdateID: step.NodeUpdateID, OperationID: step.OperationID, Operation: step.Operation, TargetVersion: step.TargetOSImageVersion, Server: step.Server, AppId: step.AppId, Track: step.Track} + raw, err := json.Marshal(request) + if err != nil { + return err + } + body, err := json.Marshal(map[string]any{"metadata": map[string]any{"annotations": map[string]string{UpdateRequestAnnotation: string(raw)}}}) + if err != nil { + return err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPatch, c.nodeURL(), bytes.NewReader(body)) + if err != nil { + return err + } + req.Header.Set("Content-Type", "application/merge-patch+json") + resp, err := c.client().Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode >= 300 { + return fmt.Errorf("fake apiserver patch failed with %s", resp.Status) + } + return nil +} + +func decodeStatus(node *corev1.Node, annotationKey string) (*updateStatus, error) { + raw := node.Annotations[annotationKey] + if raw == "" { + return nil, nil + } + var status updateStatus + if err := json.Unmarshal([]byte(raw), &status); err != nil { + return nil, err + } + return &status, nil +} + +func (c *RPClient) getNode(ctx context.Context) (*corev1.Node, error) { + request, err := http.NewRequestWithContext(ctx, http.MethodGet, c.nodeURL(), nil) + if err != nil { + return nil, err + } + response, err := c.client().Do(request) + if err != nil { + return nil, err + } + defer response.Body.Close() + if response.StatusCode >= 300 { + return nil, fmt.Errorf("fake apiserver get failed with %s", response.Status) + } + var node corev1.Node + if err := json.NewDecoder(response.Body).Decode(&node); err != nil { + return nil, fmt.Errorf("failed to decode fake Node response: %w", err) + } + return &node, nil +} + +func (c *RPClient) nodeURL() string { + return strings.TrimRight(c.APIServerURL, "/") + "/api/v1/nodes/" + c.NodeName +} + +func (c *RPClient) client() *http.Client { + if c.HTTPClient != nil { + return c.HTTPClient + } + return http.DefaultClient +} diff --git a/tools/storm/aclagent/proxies/scenario.go b/tools/storm/aclagent/proxies/scenario.go new file mode 100644 index 0000000000..0cec949df5 --- /dev/null +++ b/tools/storm/aclagent/proxies/scenario.go @@ -0,0 +1,105 @@ +package proxies + +import ( + "fmt" + "os" + "time" + + "gopkg.in/yaml.v3" +) + +type Scenario struct { + Steps []ScenarioStep `yaml:"steps"` +} + +type ScenarioStep struct { + Patch *PatchStep `yaml:"patch,omitempty"` + Expect *ExpectStep `yaml:"expect,omitempty"` +} + +type PatchStep struct { + NodeUpdateID string `yaml:"node-update-id,omitempty" json:"nodeUpdateId,omitempty"` + OperationID string `yaml:"operation-id,omitempty" json:"operationId,omitempty"` + Operation string `yaml:"operation,omitempty" json:"operation,omitempty"` + TargetOSImageVersion string `yaml:"target-os-image-version,omitempty" json:"targetVersion,omitempty"` + // Server overrides trident-acl-agent's configured Nebraska endpoint for + // this request. The agent no longer has a config file at all (see + // prepareVmForAclAgent), so any request that reaches Nebraska (stage, + // finalize) must set this field. + Server string `yaml:"server,omitempty" json:"server,omitempty"` + // AppId overrides trident-acl-agent's configured Nebraska app_id for + // this request, for the same reason Server does: there is no config + // file to source it from. + AppId string `yaml:"app-id,omitempty" json:"appId,omitempty"` + // Track overrides trident-acl-agent's configured Nebraska track for + // this request, for the same reason Server/AppId do: there is no + // config file to source it from. + Track string `yaml:"track,omitempty" json:"track,omitempty"` +} + +type ExpectStep struct { + OperationID string `yaml:"operation-id,omitempty" json:"operationId,omitempty"` + Operation string `yaml:"operation,omitempty" json:"operation,omitempty"` + Code string `yaml:"code" json:"code"` + Timeout time.Duration `yaml:"-" json:"timeoutSeconds"` + TimeoutRaw string `yaml:"timeout,omitempty" json:"-"` + ExpectTimeout bool `yaml:"expect-timeout,omitempty" json:"expectTimeout,omitempty"` +} + +type ScenarioReport struct { + Passed bool `json:"passed"` + Steps []StepReport `json:"steps"` +} + +type StepReport struct { + Index int `json:"index"` + Kind string `json:"kind"` + Passed bool `json:"passed"` + ElapsedMS int64 `json:"elapsedMs"` + Message string `json:"message"` + Expected any `json:"expected,omitempty"` + Actual any `json:"actual,omitempty"` +} + +func LoadScenario(path string) (*Scenario, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("failed to read scenario %s: %w", path, err) + } + var scenario Scenario + if err := yaml.Unmarshal(data, &scenario); err != nil { + return nil, fmt.Errorf("failed to parse scenario yaml: %w", err) + } + if err := scenario.Validate(); err != nil { + return nil, err + } + return &scenario, nil +} + +func (s *Scenario) Validate() error { + for index := range s.Steps { + step := &s.Steps[index] + kinds := 0 + if step.Patch != nil { + kinds++ + } + if step.Expect != nil { + kinds++ + } + if kinds != 1 { + return fmt.Errorf("scenario step %d must set exactly one of patch/expect", index) + } + if step.Expect != nil { + timeout := 60 * time.Second + if step.Expect.TimeoutRaw != "" { + var err error + timeout, err = time.ParseDuration(step.Expect.TimeoutRaw) + if err != nil { + return fmt.Errorf("scenario step %d has invalid timeout %q: %w", index, step.Expect.TimeoutRaw, err) + } + } + step.Expect.Timeout = timeout + } + } + return nil +} diff --git a/tools/storm/aclagent/tests/logs.go b/tools/storm/aclagent/tests/logs.go new file mode 100644 index 0000000000..b61ef7ca0b --- /dev/null +++ b/tools/storm/aclagent/tests/logs.go @@ -0,0 +1,11 @@ +package tests + +import ( + stormaclconfig "tridenttools/storm/aclagent/utils/config" + stormvm "tridenttools/storm/utils/vm" + stormvmconfig "tridenttools/storm/utils/vm/config" +) + +func FetchLogs(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.AllVMConfig) error { + return stormvm.FetchLogs(vmConfig, testConfig.OutputPath) +} diff --git a/tools/storm/aclagent/tests/rollback.go b/tools/storm/aclagent/tests/rollback.go new file mode 100644 index 0000000000..f267d3ac96 --- /dev/null +++ b/tools/storm/aclagent/tests/rollback.go @@ -0,0 +1,135 @@ +package tests + +import ( + "context" + "fmt" + "os" + "time" + + stormproxies "tridenttools/storm/aclagent/proxies" + stormaclconfig "tridenttools/storm/aclagent/utils/config" + stormvm "tridenttools/storm/utils/vm" + stormvmconfig "tridenttools/storm/utils/vm/config" +) + +// RunRollback exercises trident-acl-agent's rollback annotation end-to-end +// against the real gRPC-backed RollbackService (rollback_stage + +// rollback_finalize) implemented by tridentd, followed by the real reboot +// and post-reboot commit. It assumes the VM is already staged/finalized to +// testConfig.TargetVersion (i.e. it runs after run-ab-update in the same +// scenario), so ManualRollbackAbStaged/Finalized has a prior version to roll +// back to. +func RunRollback(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.AllVMConfig) error { + vmIP, err := stormvm.GetVmIP(vmConfig) + if err != nil { + return fmt.Errorf("failed to get VM IP: %w", err) + } + if err := os.MkdirAll(testConfig.OutputPath, 0o755); err != nil { + return err + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // Rollback doesn't stage a new image from Nebraska - it re-activates the + // previously-finalized volume trident already has on disk - so only the + // fake apiserver is needed here, not the Nebraska/image-server mocks + // run-ab-update starts. trident-acl-agent never gets a config file at + // all (see prepareVmForAclAgent); rollback's PatchSteps leave + // `server`/`appId` unset too, since Nebraska is never queried during a + // rollback request. + nodeStore := stormproxies.NewNodeStore(stormproxies.NewSeedNode(testConfig.NodeName, map[string]string{})) + apiServer := stormproxies.NewAPIServer(testConfig.NodeName, nodeStore) + if _, err := apiServer.ListenAndServe(ctx, fmt.Sprintf("0.0.0.0:%d", testConfig.APIServerPort)); err != nil { + return fmt.Errorf("failed to start fake apiserver: %w", err) + } + + nodeStore.PatchLabels(map[string]string{stormproxies.NodeImageVersionLabel: testConfig.TargetVersion}) + nodeStore.SetReadyCondition(true) + + if err := prepareVmForAclAgent(vmConfig.VMConfig, vmIP, testConfig); err != nil { + return err + } + + rp := &stormproxies.RPClient{APIServerURL: fmt.Sprintf("http://%s:%d", testConfig.HostEndpointIP, testConfig.APIServerPort), NodeName: testConfig.NodeName} + scenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ + {Patch: &stormproxies.PatchStep{NodeUpdateID: "22222222-2222-2222-2222-222222222222", OperationID: "rollback-op", Operation: "rollback"}}, + {Expect: &stormproxies.ExpectStep{OperationID: "rollback-op", Operation: "rollback", Code: "Success", Timeout: 180 * time.Second}}, + }} + report, err := rp.RunScenario(ctx, scenario) + logScenarioTimeline("rollback stage/finalize", report) + if err != nil { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent rollback scenario failed (stage/finalize): %w", err) + } + if !report.Passed { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent rollback scenario failed (stage/finalize): %+v", report) + } + + // rollback_finalize triggers a real "systemctl reboot" from + // trident-acl-agent, same as update's finalize - wait it out the same + // way run-ab-update does. + nodeStore.SetReadyCondition(false) + if err := waitForVmRebootAndSshBack(vmConfig, vmIP, testConfig); err != nil { + return fmt.Errorf("failed waiting for VM to come back after rollback finalize reboot: %w", err) + } + nodeStore.SetReadyCondition(true) + + // Same rationale as run-ab-update: the rollback reboot lands on the + // previous root, which needs the fake kubeconfig re-delivered before it + // can talk to the fake apiserver again. + if err := prepareVmForAclAgent(vmConfig.VMConfig, vmIP, testConfig); err != nil { + return fmt.Errorf("failed to reconfigure ACL agent on post-rollback-reboot root: %w", err) + } + + finalScenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ + {Expect: &stormproxies.ExpectStep{OperationID: "rollback-op", Operation: "commit", Code: "Success", Timeout: 180 * time.Second}}, + }} + finalReport, err := rp.RunScenario(ctx, finalScenario) + logScenarioTimeline("post-rollback-reboot commit", finalReport) + if err != nil { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent rollback scenario failed (post-reboot commit): %w", err) + } + if !finalReport.Passed { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent rollback scenario failed (post-reboot commit): %+v", finalReport) + } + + snapshot := nodeStore.Snapshot() + if got := snapshot.Annotations[stormproxies.UpdateCommitStatusAnnotation]; got == "" { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("final rollback commit status annotation missing") + } + + // Regression coverage for the "rollback with nothing to roll back" + // bug: the only AB rollback available was just consumed above, so a + // second rollback request now must be detected as a no-op (via + // RollbackStage's servicing_kind, which tridentd now reports the same + // way update/install do) rather than reporting a false Success and + // rebooting the node again for no reason. This exercises that fix + // end-to-end against the real tridentd, not just the mock. + secondScenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ + {Patch: &stormproxies.PatchStep{NodeUpdateID: "33333333-3333-3333-3333-333333333333", OperationID: "rollback-op-2", Operation: "rollback"}}, + {Expect: &stormproxies.ExpectStep{OperationID: "rollback-op-2", Operation: "rollback", Code: "OperationFailed", Timeout: 60 * time.Second}}, + }} + secondReport, err := rp.RunScenario(ctx, secondScenario) + logScenarioTimeline("second rollback with empty chain", secondReport) + if err != nil { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent second-rollback (empty chain) scenario failed: %w", err) + } + if !secondReport.Passed { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent second-rollback (empty chain) scenario failed: %+v", secondReport) + } + // A no-op rollback must not trigger another reboot: the VM should + // still be reachable immediately, with no reboot wait needed. + if _, err := stormvm.GetVmIP(vmConfig); err != nil { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("VM appears to have rebooted (or become unreachable) after a no-op rollback, which should not trigger a reboot: %w", err) + } + + return collectAclArtifacts(vmConfig.VMConfig, vmIP, testConfig.OutputPath) +} diff --git a/tools/storm/aclagent/tests/update.go b/tools/storm/aclagent/tests/update.go new file mode 100644 index 0000000000..e3969baf84 --- /dev/null +++ b/tools/storm/aclagent/tests/update.go @@ -0,0 +1,548 @@ +package tests + +import ( + "archive/tar" + "context" + "crypto/sha512" + "encoding/hex" + "fmt" + "io" + "os" + "path/filepath" + "strings" + "time" + + stormproxies "tridenttools/storm/aclagent/proxies" + stormaclconfig "tridenttools/storm/aclagent/utils/config" + stormfile "tridenttools/storm/utils/file" + stormssh "tridenttools/storm/utils/ssh" + stormvm "tridenttools/storm/utils/vm" + stormvmconfig "tridenttools/storm/utils/vm/config" + + "github.com/sirupsen/logrus" +) + +// sha384File computes the lowercase hex-encoded SHA-384 digest that tridentd +// expects for a given image path. +// +// For a .cosi file, tridentd does NOT hash the whole archive: a COSI is a +// plain tar with an embedded "metadata.json" entry, and tridentd's Host +// Configuration "sha384" field must match the hash of just that entry's +// bytes (see crates/trident/src/osimage/cosi/mod.rs's read_cosi_metadata). +// For any other file, this hashes the whole file's contents directly. +// logScenarioTimeline prints a human-readable, step-by-step trace of a +// scenario's progress through the ACL agent's stage/finalize/commit state +// machine. It runs regardless of pass/fail so a test run's output always +// documents exactly how far the state machine got and why, instead of +// forcing readers to reconstruct the timeline from raw journal logs. +func logScenarioTimeline(label string, report *stormproxies.ScenarioReport) { + if report == nil { + return + } + logrus.Infof("=== %s state machine timeline ===", label) + for _, step := range report.Steps { + status := "PASS" + if !step.Passed { + status = "FAIL" + } + logrus.Infof(" [%d] %-6s kind=%-8s (%dms) %s", step.Index, status, step.Kind, step.ElapsedMS, step.Message) + if !step.Passed { + logrus.Infof(" expected: %+v", step.Expected) + logrus.Infof(" actual: %+v", step.Actual) + } + } + overall := "PASSED" + if !report.Passed { + overall = "FAILED" + } + logrus.Infof("=== %s state machine timeline: %s ===", label, overall) +} + +func sha384File(path string) (string, error) { + if strings.HasSuffix(path, ".cosi") { + return sha384CosiMetadata(path) + } + + f, err := os.Open(path) + if err != nil { + return "", err + } + defer f.Close() + + h := sha512.New384() + if _, err := io.Copy(h, f); err != nil { + return "", err + } + return hex.EncodeToString(h.Sum(nil)), nil +} + +// sha384CosiMetadata extracts the "metadata.json" entry from a COSI (a plain +// tar archive) and returns the lowercase hex-encoded SHA-384 digest of its +// raw bytes, matching what tridentd validates the Host Configuration's +// "sha384" field against. +func sha384CosiMetadata(path string) (string, error) { + f, err := os.Open(path) + if err != nil { + return "", err + } + defer f.Close() + + tr := tar.NewReader(f) + for { + header, err := tr.Next() + if err == io.EOF { + return "", fmt.Errorf("metadata.json entry not found in COSI file %s", path) + } + if err != nil { + return "", fmt.Errorf("failed to read COSI tar entries in %s: %w", path, err) + } + if header.Name != "metadata.json" { + continue + } + h := sha512.New384() + if _, err := io.Copy(h, tr); err != nil { + return "", fmt.Errorf("failed to hash metadata.json in COSI file %s: %w", path, err) + } + return hex.EncodeToString(h.Sum(nil)), nil + } +} + +// expectValidateConnection runs "trident-acl-agent --validate-connection +// " on the VM and asserts its exit status matches wantSuccess. All +// configuration is via TRIDENT_ACL_AGENT_* environment variables (see +// crates/trident-acl-agent/src/config.rs) - there is no config file - so +// envVars lets a caller inject one-off overrides for this single +// invocation only (nil/empty runs against whatever the agent's own +// environment already has, i.e. nothing, proving the compiled-in +// defaults). Used to prove both halves of the kubeconfig-server fix +// (https://github.com/microsoft/trident/pull/730): before the fake +// kubeconfig is delivered, only tridentd (socket-activated, no config +// needed) should succeed while kubernetes/nebraska fail on unreachable +// defaults; after prepareVmForAclAgent delivers it, kubernetes succeeds +// too. Nebraska has no static config at all, so proving it can succeed +// requires passing envVars explicitly (see the env-var-override checks in +// RunABUpdate). +func expectValidateConnection(cfg stormvmconfig.VMConfig, vmIP, mode string, wantSuccess bool, envVars map[string]string) error { + var prefix strings.Builder + prefix.WriteString("sudo") + if len(envVars) > 0 { + prefix.WriteString(" env") + for k, v := range envVars { + fmt.Fprintf(&prefix, " %s=%q", k, v) + } + } + command := fmt.Sprintf("%s trident-acl-agent --validate-connection %s", prefix.String(), mode) + out, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, command) + gotSuccess := err == nil + if gotSuccess != wantSuccess { + return fmt.Errorf("--validate-connection %s (envVars=%v): expected success=%v, got success=%v (err=%v, output=%s)", mode, envVars, wantSuccess, gotSuccess, err, out) + } + return nil +} + +func RunABUpdate(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.AllVMConfig) error { + vmIP, err := stormvm.GetVmIP(vmConfig) + if err != nil { + return fmt.Errorf("failed to get VM IP: %w", err) + } + if err := os.MkdirAll(testConfig.OutputPath, 0o755); err != nil { + return err + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + nodeStore := stormproxies.NewNodeStore(stormproxies.NewSeedNode(testConfig.NodeName, map[string]string{})) + apiServer := stormproxies.NewAPIServer(testConfig.NodeName, nodeStore) + // Bind on all interfaces (not 127.0.0.1) so the VM can reach the fake + // apiserver directly over the libvirt NAT network at testConfig.HostEndpointIP, + // instead of relying on reverse SSH tunnels. Tunnels don't survive a real + // VM reboot; a real host IP does. + if _, err := apiServer.ListenAndServe(ctx, fmt.Sprintf("0.0.0.0:%d", testConfig.APIServerPort)); err != nil { + return fmt.Errorf("failed to start fake apiserver: %w", err) + } + + nebraskaCodebase := testConfig.NebraskaCodebase + nebraskaPackageName := testConfig.NebraskaPackageName + nebraskaSHA384 := testConfig.NebraskaSHA384 + + // tridentd downloads and hashes the image itself as part of staging a + // runtime update; it is not enough for the acl-agent to merely reach the + // fake apiserver/Nebraska endpoints. When a real image is configured, serve + // it over plain HTTP from this same test runner and advertise its real + // SHA-384 hash, so tridentd's download+verify path is exercised faithfully + // instead of failing on an unreachable https://example.invalid URL or a + // hash that doesn't match any downloadable bytes. + imagePath := testConfig.ImagePath + if imagePath == "" { + found, err := stormfile.FindFile(testConfig.ArtifactsDir, ".*\\.cosi$") + if err != nil { + return fmt.Errorf("failed to find a .cosi update image under %s: %w", testConfig.ArtifactsDir, err) + } + imagePath = found + } + + { + hash, err := sha384File(imagePath) + if err != nil { + return fmt.Errorf("failed to hash image %s: %w", imagePath, err) + } + imageServer := &stormproxies.ImageServer{ImagePath: imagePath} + if _, err := imageServer.ListenAndServe(ctx, fmt.Sprintf("0.0.0.0:%d", testConfig.ImageServerPort)); err != nil { + return fmt.Errorf("failed to start fake image server: %w", err) + } + nebraskaCodebase = fmt.Sprintf("http://%s:%d/", testConfig.HostEndpointIP, testConfig.ImageServerPort) + nebraskaPackageName = imageServer.PackageBaseName() + nebraskaSHA384 = hash + } + + nebraska := &stormproxies.NebraskaProxy{Scenario: &stormproxies.NebraskaScenario{ + Available: true, + Version: testConfig.TargetVersion, + URL: nebraskaCodebase, + SHA384: nebraskaSHA384, + PackageName: nebraskaPackageName, + }} + if _, err := nebraska.ListenAndServe(ctx, fmt.Sprintf("0.0.0.0:%d", testConfig.NebraskaPort)); err != nil { + return fmt.Errorf("failed to start fake Nebraska endpoint: %w", err) + } + + nodeStore.PatchLabels(map[string]string{stormproxies.NodeImageVersionLabel: testConfig.ExpectedInitialVolume}) + nodeStore.SetReadyCondition(true) + + // Before the fake kubeconfig has been delivered to the VM, trident-acl-agent + // falls back to its compiled-in defaults for everything: tridentd's + // socket is reached via systemd socket activation regardless of any + // config, so it should succeed; kubernetes and nebraska both default to + // unreachable placeholders (no real kubeconfig at + // /var/lib/kubelet/kubeconfig yet, and https://nebraska.example.invalid + // / no app_id/server override in play), so both should fail. + if err := expectValidateConnection(vmConfig.VMConfig, vmIP, "tridentd", true, nil); err != nil { + return fmt.Errorf("pre-config validate-connection check failed: %w", err) + } + if err := expectValidateConnection(vmConfig.VMConfig, vmIP, "kubernetes", false, nil); err != nil { + return fmt.Errorf("pre-config validate-connection check failed: %w", err) + } + if err := expectValidateConnection(vmConfig.VMConfig, vmIP, "nebraska", false, nil); err != nil { + return fmt.Errorf("pre-config validate-connection check failed: %w", err) + } + + if err := prepareVmForAclAgent(vmConfig.VMConfig, vmIP, testConfig); err != nil { + return err + } + + // Once prepareVmForAclAgent has delivered the fake kubeconfig, kubernetes + // now succeeds (its own `server:` field points at the fake apiserver); + // tridentd is unaffected either way. trident-acl-agent never gets a + // config file at all (see prepareVmForAclAgent's doc comment) - Nebraska + // endpoint/app_id/track are only ever supplied per-request, either via + // the update-request annotation's `server`/`appId`/`track` fields or + // (for this one-off diagnostic check) a TRIDENT_ACL_AGENT_NEBRASKA_* + // env var - so a bare "--validate-connection nebraska" with neither in + // play still fails here exactly as it did before configuration, proving + // there is no static Nebraska configuration of any kind for the agent + // to fall back on. + for _, mode := range []string{"tridentd", "kubernetes"} { + if err := expectValidateConnection(vmConfig.VMConfig, vmIP, mode, true, nil); err != nil { + return fmt.Errorf("post-config validate-connection check failed: %w", err) + } + } + if err := expectValidateConnection(vmConfig.VMConfig, vmIP, "nebraska", false, nil); err != nil { + return fmt.Errorf("post-config validate-connection check failed (trident-acl-agent has no static Nebraska config): %w", err) + } + + rp := &stormproxies.RPClient{APIServerURL: fmt.Sprintf("http://%s:%d", testConfig.HostEndpointIP, testConfig.APIServerPort), NodeName: testConfig.NodeName} + nebraskaServer := fmt.Sprintf("http://%s:%d", testConfig.HostEndpointIP, testConfig.NebraskaPort) + nebraskaAppID := nebraska.AppID() + nebraskaTrack := nebraska.Track() + + // Proves the env-var override path itself: the same one-off diagnostic + // check that just failed with no override now succeeds when given the + // exact TRIDENT_ACL_AGENT_NEBRASKA_* values RunABUpdate is about to send + // via the update-request annotation, and fails again with a deliberately + // wrong endpoint - i.e. both the success and failure paths of the + // env-var config mechanism are exercised directly, not just inferred + // from the annotation-driven flow below. + nebraskaEnv := map[string]string{ + "TRIDENT_ACL_AGENT_NEBRASKA_ENDPOINT": nebraskaServer, + "TRIDENT_ACL_AGENT_NEBRASKA_APP_ID": nebraskaAppID, + "TRIDENT_ACL_AGENT_NEBRASKA_TRACK": nebraskaTrack, + } + if err := expectValidateConnection(vmConfig.VMConfig, vmIP, "nebraska", true, nebraskaEnv); err != nil { + return fmt.Errorf("env-var-override validate-connection check failed (expected success): %w", err) + } + wrongNebraskaEnv := map[string]string{ + "TRIDENT_ACL_AGENT_NEBRASKA_ENDPOINT": fmt.Sprintf("http://%s:%d", testConfig.HostEndpointIP, testConfig.NebraskaPort+1), + "TRIDENT_ACL_AGENT_NEBRASKA_APP_ID": nebraskaAppID, + "TRIDENT_ACL_AGENT_NEBRASKA_TRACK": nebraskaTrack, + } + if err := expectValidateConnection(vmConfig.VMConfig, vmIP, "nebraska", false, wrongNebraskaEnv); err != nil { + return fmt.Errorf("env-var-override validate-connection check failed (expected failure against wrong port): %w", err) + } + + scenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ + {Patch: &stormproxies.PatchStep{NodeUpdateID: "11111111-1111-1111-1111-111111111111", OperationID: "stage-op", Operation: "stage", TargetOSImageVersion: testConfig.TargetVersion, Server: nebraskaServer, AppId: nebraskaAppID, Track: nebraskaTrack}}, + {Expect: &stormproxies.ExpectStep{OperationID: "stage-op", Operation: "stage", Code: "Success", Timeout: 120 * time.Second}}, + {Patch: &stormproxies.PatchStep{NodeUpdateID: "11111111-1111-1111-1111-111111111111", OperationID: "finalize-op", Operation: "finalize", TargetOSImageVersion: testConfig.TargetVersion, Server: nebraskaServer, AppId: nebraskaAppID, Track: nebraskaTrack}}, + }} + report, err := rp.RunScenario(ctx, scenario) + logScenarioTimeline("stage/finalize", report) + if err != nil { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent scenario failed (stage/finalize): %w", err) + } + if !report.Passed { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent scenario failed (stage/finalize): %+v", report) + } + + // Finalize triggers a real "systemctl reboot" from trident-acl-agent + // itself. Reflect the VM actually going away/coming back in the fake + // Node's Ready condition, then wait for SSH to come back before + // checking the agent's post-reboot commit. + nodeStore.SetReadyCondition(false) + if err := waitForVmRebootAndSshBack(vmConfig, vmIP, testConfig); err != nil { + return fmt.Errorf("failed waiting for VM to come back after finalize reboot: %w", err) + } + nodeStore.SetReadyCondition(true) + + // /etc/trident and /var/lib/kubelet are their own dedicated ext4 + // partitions (not part of the A/B-swapped root), so the config and + // kubeconfig prepareVmForAclAgent delivered before staging carry over + // unchanged to the newly-activated root. trident-acl-agent.service is + // enabled by default there (baked into the update image) and starts + // with that same config at boot, so no re-delivery is needed here. + + finalScenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ + {Expect: &stormproxies.ExpectStep{OperationID: "finalize-op", Operation: "commit", Code: "Success", Timeout: 180 * time.Second}}, + }} + finalReport, err := rp.RunScenario(ctx, finalScenario) + logScenarioTimeline("post-reboot commit", finalReport) + if err != nil { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent scenario failed (post-reboot commit): %w", err) + } + if !finalReport.Passed { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent scenario failed (post-reboot commit): %+v", finalReport) + } + + snapshot := nodeStore.Snapshot() + if got := snapshot.Annotations[stormproxies.UpdateCommitStatusAnnotation]; got == "" { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("final commit status annotation missing") + } + + // The node annotation only proves the ACL-agent-facing rollout API + // reported success; it says nothing about whether trident-acl-agent + // actually drove Nebraska's own instance state machine correctly. Assert + // that too, against the real Nebraska instance_status_history this + // scenario's seeded application accumulated over stage/finalize/commit. + if err := nebraska.ValidateStatusHistory(stormproxies.ExpectedUpdateStatusSequence); err != nil { + collectAclArtifactsBestEffort(vmConfig.VMConfig, vmIP, testConfig.OutputPath) + return fmt.Errorf("ACL agent scenario failed Nebraska status validation: %w", err) + } + + return collectAclArtifacts(vmConfig.VMConfig, vmIP, testConfig.OutputPath) +} + +// collectAclArtifactsBestEffort collects the same diagnostic artifacts as +// collectAclArtifacts, but on a failure path where the harness is about to +// return an error anyway. run-ab-update's failure is otherwise a dead end +// for diagnostics: the storm-trident test runner marks collect-logs (and +// cleanup-vm) as NOTR ("dependency failure") whenever run-ab-update fails, +// so nothing ever calls collectAclArtifacts and the post-reboot journal +// (trident-acl-agent.log / tridentd.log) that would explain the failure is +// never captured or published as a pipeline artifact. Errors here are +// logged but swallowed so they never mask the original failure. +func collectAclArtifactsBestEffort(cfg stormvmconfig.VMConfig, vmIP string, outputPath string) { + if err := collectAclArtifacts(cfg, vmIP, outputPath); err != nil { + logrus.Warnf("best-effort artifact collection after test failure also failed: %v", err) + } +} + +func prepareVmForAclAgent(cfg stormvmconfig.VMConfig, vmIP string, testConfig stormaclconfig.TestConfig) error { + // trident-acl-agent needs no config file at all for this scenario: + // - nebraska.app_id and nebraska.endpoint are supplied per-request via + // the update-request annotation's `appId`/`server` fields instead + // (see PatchStep.AppId/Server usage in RunABUpdate). + // - kubernetes.api_server is left unset (its own default): the fake + // kubeconfig written below already has `server:` pointing at the + // fake apiserver, so there is nothing to override. + // - kubernetes.node_name is also left unset (its own default: the + // node's real hostname, lowercased). testConfig.NodeName is set to + // match the VM image's Image Customizer 'hostname' setting + // (baseimg-acl-agent.yaml), so the agent's own hostname-derived + // default already agrees with the fake apiserver's seeded Node - + // no override needed, exactly like a real deployment. + // - kubernetes.kubeconfig, trident.socket, and orchestration.goal_source + // are all already their compiled-in defaults. + // trident-acl-agent.conf is simply never written to the VM. + + // The fake apiserver has no real kubelet-managed kubeconfig backing it, + // and it takes plain HTTP with no auth/TLS, so provide a minimal + // insecure kubeconfig pointing at it instead of relying on a real + // kubelet bootstrap file that doesn't exist on this test image. + kubeconfig := fmt.Sprintf(`apiVersion: v1 +kind: Config +clusters: +- name: fake + cluster: + server: http://%s:%d + insecure-skip-tls-verify: true +contexts: +- name: fake + context: + cluster: fake + user: fake +current-context: fake +users: +- name: fake + user: {} +`, testConfig.HostEndpointIP, testConfig.APIServerPort) + + localKubeconfigFile, err := os.CreateTemp("", "trident-acl-agent-kubeconfig-*.yaml") + if err != nil { + return fmt.Errorf("failed to create local temp file for fake kubeconfig: %w", err) + } + defer os.Remove(localKubeconfigFile.Name()) + if _, err := localKubeconfigFile.WriteString(kubeconfig); err != nil { + localKubeconfigFile.Close() + return fmt.Errorf("failed to write local temp fake kubeconfig: %w", err) + } + if err := localKubeconfigFile.Close(); err != nil { + return fmt.Errorf("failed to close local temp fake kubeconfig: %w", err) + } + if _, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, "sudo mkdir -p /var/lib/kubelet"); err != nil { + return fmt.Errorf("failed to create /var/lib/kubelet on VM: %w", err) + } + if err := stormssh.ScpUploadFileWithSudo(cfg, vmIP, localKubeconfigFile.Name(), "/var/lib/kubelet/kubeconfig"); err != nil { + return fmt.Errorf("failed to upload fake kubeconfig to VM: %w", err) + } + + // Enable (without "--now"), then always issue a single "restart" of + // trident-acl-agent.service. Each RunX test case starts its own fresh + // fake-apiserver/Nebraska instances, so a plain "enable --now" isn't + // enough: it's a no-op restart-wise if the service is already active + // (e.g. run-rollback runs right after run-ab-update, no reboot in + // between), leaving the agent's watch connected to the prior test + // case's now-torn-down apiserver, silently missing the new one and + // timing out. A single unconditional "restart" fixes that by both + // starting the unit if needed and cleanly restarting it if already + // running. Do NOT combine "enable --now" with a separate "restart" + // call: on the post-reboot path the unit auto-starts at boot and can + // already be mid-commit (calling tridentd) by the time these SSH + // commands run, so a second restart right after "--now" started it can + // kill and restart the agent mid-call, and the new instance's retry + // then fails with tridentd's "Servicing is active" error. + command := strings.Join([]string{ + "sudo systemctl restart tridentd.service", + "sudo systemctl enable trident-acl-agent.service", + "sudo systemctl restart trident-acl-agent.service", + // api_server lives only in the fake kubeconfig now (agent config + // never sets it - see this function's doc comment). + fmt.Sprintf("sudo grep -qF '%s:%d' /var/lib/kubelet/kubeconfig", testConfig.HostEndpointIP, testConfig.APIServerPort), + // Lock in the design invariant: trident-acl-agent.conf is never + // written to the VM at all - app_id/endpoint are only ever supplied + // per-request via the update-request annotation's `appId`/`server` + // fields. + "! sudo test -e /etc/trident/trident-acl-agent.conf", + }, " && ") + if _, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, command); err != nil { + return fmt.Errorf("failed to prepare VM for ACL agent: %w", err) + } + + // Both services can briefly report "activating" right after + // "enable --now" before settling into "active" -- poll rather than + // checking is-active exactly once. + for _, svc := range []string{"tridentd.service", "trident-acl-agent.service"} { + if err := waitForServiceActive(cfg, vmIP, svc, 30*time.Second); err != nil { + return fmt.Errorf("failed waiting for %s to become active: %w", svc, err) + } + } + return nil +} + +func waitForServiceActive(cfg stormvmconfig.VMConfig, vmIP, service string, timeout time.Duration) error { + deadline := time.Now().Add(timeout) + var lastErr error + for time.Now().Before(deadline) { + out, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, fmt.Sprintf("sudo systemctl is-active %s", service)) + if err == nil && strings.TrimSpace(out) == "active" { + return nil + } + lastErr = err + time.Sleep(2 * time.Second) + } + + // Pull the service's journal so a timeout is self-diagnosing even when + // the scenario fails before the dedicated collect-logs test case runs. + journal, journalErr := stormssh.SshCommandCombinedOutput(cfg, vmIP, fmt.Sprintf("sudo journalctl -u %s --no-pager -n 200", service)) + if journalErr != nil { + journal = fmt.Sprintf("", journalErr) + } + return fmt.Errorf("service %s did not become active within %s (last error: %v)\njournal for %s:\n%s", service, timeout, lastErr, service, journal) +} + +// waitForVmRebootAndSshBack polls SSH until it is unreachable (confirming +// the agent's real "systemctl reboot" actually took the VM down) and then +// reachable again (confirming it came back up), mirroring the real-reboot +// wait pattern already used by the storm servicing scenario. +func waitForVmRebootAndSshBack(vmConfig stormvmconfig.AllVMConfig, vmIP string, testConfig stormaclconfig.TestConfig) error { + wentDown := false + downTimeout := time.Now().Add(60 * time.Second) + for time.Now().Before(downTimeout) { + if _, err := stormssh.SshCommandCombinedOutput(vmConfig.VMConfig, vmIP, "true"); err != nil { + wentDown = true + break + } + time.Sleep(2 * time.Second) + } + if !wentDown { + return fmt.Errorf("VM never became unreachable over SSH within %s; reboot did not appear to happen", 60*time.Second) + } + + // A single successful SSH command right after boot is not proof the VM + // is stably back up: sshd (or the network stack) can accept one + // connection and then bounce again moments later while later boot + // units are still settling (observed in practice as one successful + // "true" immediately followed by "connection refused" on the very + // next SSH dial). Require a few consecutive successes, spaced out, + // before declaring the VM ready, so callers that immediately issue + // real SSH commands (e.g. prepareVmForAclAgent) don't race a + // still-settling boot. + const requiredConsecutiveSuccesses = 3 + consecutiveSuccesses := 0 + upTimeout := time.Now().Add(5 * time.Minute) + for time.Now().Before(upTimeout) { + if _, err := stormssh.SshCommandCombinedOutput(vmConfig.VMConfig, vmIP, "true"); err == nil { + consecutiveSuccesses++ + if consecutiveSuccesses >= requiredConsecutiveSuccesses { + return nil + } + } else { + consecutiveSuccesses = 0 + } + time.Sleep(2 * time.Second) + } + return fmt.Errorf("VM did not come back up over SSH within timeout after finalize reboot") +} + +func collectAclArtifacts(cfg stormvmconfig.VMConfig, vmIP string, outputPath string) error { + if outputPath == "" { + return nil + } + cmds := []string{ + "sudo journalctl --no-pager -u trident-acl-agent.service > /tmp/trident-acl-agent.log && sudo chmod 644 /tmp/trident-acl-agent.log", + "sudo journalctl --no-pager -u tridentd.service > /tmp/tridentd.log && sudo chmod 644 /tmp/tridentd.log", + } + for _, cmd := range cmds { + _, _ = stormssh.SshCommandCombinedOutput(cfg, vmIP, cmd) + } + for _, remote := range []string{"/tmp/trident-acl-agent.log", "/tmp/tridentd.log"} { + if err := stormssh.ScpDownloadFile(cfg, vmIP, remote, filepath.Join(outputPath, filepath.Base(remote))); err != nil { + logrus.Warnf("failed to download %s: %v", remote, err) + } + } + return nil +} diff --git a/tools/storm/aclagent/tests/vm.go b/tools/storm/aclagent/tests/vm.go new file mode 100644 index 0000000000..a80ef66eb4 --- /dev/null +++ b/tools/storm/aclagent/tests/vm.go @@ -0,0 +1,43 @@ +package tests + +import ( + "fmt" + + stormaclconfig "tridenttools/storm/aclagent/utils/config" + stormvm "tridenttools/storm/utils/vm" + stormvmconfig "tridenttools/storm/utils/vm/config" + + "github.com/sirupsen/logrus" +) + +func CheckDeployment(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.AllVMConfig) error { + return stormvm.CheckDeployment(vmConfig, testConfig.ExpectedInitialVolume) +} + +func DeployVM(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.AllVMConfig) error { + if vmConfig.VMConfig.Platform == stormvmconfig.PlatformQEMU { + logrus.Tracef("Deploying VM on QEMU platform with name '%s'", vmConfig.VMConfig.Name) + if err := vmConfig.QemuConfig.DeployQemuVM(vmConfig.VMConfig.Name, testConfig.ArtifactsDir, testConfig.OutputPath, testConfig.Verbose); err != nil { + return fmt.Errorf("failed to deploy qemu vm: %w", err) + } + } else if vmConfig.VMConfig.Platform == stormvmconfig.PlatformAzure { + logrus.Tracef("Deploying VM on Azure platform with name '%s'", vmConfig.VMConfig.Name) + if err := vmConfig.AzureConfig.DeployAzureVM(vmConfig.VMConfig.Name, vmConfig.VMConfig.User); err != nil { + return fmt.Errorf("failed to deploy azure vm: %w", err) + } + } + return nil +} + +func CleanupVM(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.AllVMConfig) error { + if vmConfig.VMConfig.Platform == stormvmconfig.PlatformAzure { + if err := vmConfig.AzureConfig.CleanupAzureVM(); err != nil { + return fmt.Errorf("failed to cleanup Azure VM: %w", err) + } + } else if vmConfig.VMConfig.Platform == stormvmconfig.PlatformQEMU { + if err := vmConfig.QemuConfig.CleanupQemuVM(vmConfig.VMConfig.Name); err != nil { + return fmt.Errorf("failed to cleanup QEMU VM: %w", err) + } + } + return nil +} diff --git a/tools/storm/aclagent/trident.go b/tools/storm/aclagent/trident.go new file mode 100644 index 0000000000..2f5ab036f5 --- /dev/null +++ b/tools/storm/aclagent/trident.go @@ -0,0 +1,92 @@ +package aclagent + +import ( + "fmt" + "os" + "path/filepath" + + stormtests "tridenttools/storm/aclagent/tests" + stormaclconfig "tridenttools/storm/aclagent/utils/config" + stormvmazure "tridenttools/storm/utils/vm/azure" + stormvmconfig "tridenttools/storm/utils/vm/config" + stormvmqemu "tridenttools/storm/utils/vm/qemu" + + "github.com/microsoft/storm" + "github.com/sirupsen/logrus" +) + +type TridentAclAgentScenario struct { + args TridentAclAgentScenarioArgs +} + +type TridentAclAgentScenarioArgs struct { + stormaclconfig.TestConfig `embed:""` + stormvmconfig.VMConfig `embed:""` + stormvmqemu.QemuConfig `embed:""` + stormvmazure.AzureConfig `embed:""` + TestCaseToRun string `help:"Name of the test case to run. If not specified, all test cases will be run." default:"all"` +} + +func (s *TridentAclAgentScenario) Name() string { return "aclagent" } +func (s *TridentAclAgentScenario) Args() any { return &s.args } +func (s *TridentAclAgentScenario) Tags() []string { return []string{} } +func (s *TridentAclAgentScenario) StagePaths() []string { return []string{} } +func (s *TridentAclAgentScenario) RequiredFiles() []string { return nil } +func (s TridentAclAgentScenario) Setup(ctx storm.SetupCleanupContext) error { return nil } + +func (s *TridentAclAgentScenario) Cleanup(ctx storm.SetupCleanupContext) error { + if s.args.TestConfig.ForceCleanup { + _ = stormtests.CleanupVM(s.args.TestConfig, stormvmconfig.AllVMConfig{VMConfig: s.args.VMConfig, QemuConfig: s.args.QemuConfig, AzureConfig: s.args.AzureConfig}) + } + return nil +} + +func (s *TridentAclAgentScenario) RegisterTestCases(r storm.TestRegistrar) error { + r.RegisterTestCase("deploy-vm", s.deployVm) + r.RegisterTestCase("check-deployment", s.checkDeployment) + r.RegisterTestCase("run-ab-update", s.runABUpdate) + r.RegisterTestCase("run-rollback", s.runRollback) + r.RegisterTestCase("collect-logs", s.collectLogs) + r.RegisterTestCase("cleanup-vm", s.cleanupVm) + return nil +} + +func (s *TridentAclAgentScenario) runTestCase(tc storm.TestCase, testFunc func(stormaclconfig.TestConfig, stormvmconfig.AllVMConfig) error) error { + if tc.Name() != s.args.TestCaseToRun && s.args.TestCaseToRun != "all" { + tc.Skip(fmt.Sprintf("Test case '%s' does not align to TestCaseToRun '%s'", tc.Name(), s.args.TestCaseToRun)) + return nil + } + logrus.Infof("Running test case '%s'", tc.Name()) + testCaseSpecificConfig := s.args.TestConfig + if testCaseSpecificConfig.OutputPath != "" { + testCaseSpecificConfig.OutputPath = filepath.Join(testCaseSpecificConfig.OutputPath, tc.Name()) + if err := os.MkdirAll(testCaseSpecificConfig.OutputPath, 0o755); err != nil { + tc.FailFromError(err) + } + } + if err := testFunc(testCaseSpecificConfig, stormvmconfig.AllVMConfig{VMConfig: s.args.VMConfig, QemuConfig: s.args.QemuConfig, AzureConfig: s.args.AzureConfig}); err != nil { + logrus.Infof("test case '%s' failed", tc.Name()) + tc.FailFromError(err) + } + logrus.Infof("test case '%s' passed", tc.Name()) + return nil +} + +func (s *TridentAclAgentScenario) deployVm(tc storm.TestCase) error { + return s.runTestCase(tc, stormtests.DeployVM) +} +func (s *TridentAclAgentScenario) checkDeployment(tc storm.TestCase) error { + return s.runTestCase(tc, stormtests.CheckDeployment) +} +func (s *TridentAclAgentScenario) runABUpdate(tc storm.TestCase) error { + return s.runTestCase(tc, stormtests.RunABUpdate) +} +func (s *TridentAclAgentScenario) runRollback(tc storm.TestCase) error { + return s.runTestCase(tc, stormtests.RunRollback) +} +func (s *TridentAclAgentScenario) collectLogs(tc storm.TestCase) error { + return s.runTestCase(tc, stormtests.FetchLogs) +} +func (s *TridentAclAgentScenario) cleanupVm(tc storm.TestCase) error { + return s.runTestCase(tc, stormtests.CleanupVM) +} diff --git a/tools/storm/aclagent/utils/config/config.go b/tools/storm/aclagent/utils/config/config.go new file mode 100644 index 0000000000..fa5ce93f55 --- /dev/null +++ b/tools/storm/aclagent/utils/config/config.go @@ -0,0 +1,19 @@ +package config + +type TestConfig struct { + ArtifactsDir string `help:"Directory containing artifacts for the VM" default:"."` + OutputPath string `help:"Path to the output directory for logs and artifacts" default:"./output"` + Verbose bool `help:"Enable verbose logging" default:"false"` + ForceCleanup bool `help:"Force cleanup of VM when test finishes" default:"false"` + APIServerPort int `help:"Runner port exposed into VM for fake apiserver" default:"18080"` + NebraskaPort int `help:"Runner port exposed into VM for fake Nebraska endpoint" default:"18081"` + TargetVersion string `help:"Target OS image version to request" default:"202507.28.0"` + NebraskaPackageName string `help:"Package name returned by the fake Nebraska endpoint (overridden by the image file name when ImagePath is set)" default:"acl.cosi"` + NebraskaCodebase string `help:"Base URL returned by the fake Nebraska endpoint (overridden to point at the fake image server when ImagePath is set)" default:"https://example.invalid/images/"` + NebraskaSHA384 string `help:"SHA384 returned by the fake Nebraska endpoint (overridden by the real hash of ImagePath when set)" default:"111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"` + ImagePath string `help:"Path to a real OS update image (e.g. a .cosi file) to serve to tridentd during staging; when set, this takes precedence over NebraskaCodebase/NebraskaPackageName/NebraskaSHA384. When empty, the first *.cosi file found under ArtifactsDir is used."` + ImageServerPort int `help:"Runner port exposed into VM for the fake image server" default:"18082"` + NodeName string `help:"Node name served by the fake apiserver; must match the VM image's hostname (Image Customizer 'hostname' setting in baseimg-acl-agent.yaml), since trident-acl-agent's [kubernetes].node_name defaults to the node's own real hostname" default:"trident-acl-agent-testimg"` + HostEndpointIP string `help:"Host IP the VM can reach the fake apiserver/Nebraska endpoints at" default:"192.168.122.1"` + ExpectedInitialVolume string `help:"Expected active volume immediately after deployment" default:"volume-a"` +} diff --git a/tools/storm/servicing/tests/update.go b/tools/storm/servicing/tests/update.go index 5cf6a1f37f..66d628d217 100644 --- a/tools/storm/servicing/tests/update.go +++ b/tools/storm/servicing/tests/update.go @@ -249,7 +249,7 @@ func innerUpdateLoop(testConfig stormsvcconfig.TestConfig, vmConfig stormvmconfi return fmt.Errorf("failed to copy staged trident log to output path: %w", err) } if err := os.Chmod(stageLogPath, 0644); err != nil { - logrus.Errorf("failed to change permissions for staged trident log: %w", err) + logrus.Errorf("failed to change permissions for staged trident log: %v", err) } if lsOut, err := exec.Command("ls", "-lh", stageLogPath).Output(); err == nil { logrus.Tracef("Staged trident log details for iteration %d:\n%s", i, lsOut) diff --git a/tools/storm/utils/vm/qemu/qemu.go b/tools/storm/utils/vm/qemu/qemu.go index fcec985b9e..3f757d1ddb 100644 --- a/tools/storm/utils/vm/qemu/qemu.go +++ b/tools/storm/utils/vm/qemu/qemu.go @@ -19,8 +19,9 @@ import ( ) type QemuConfig struct { - SecureBoot bool `help:"Enable secure boot for the VM" default:"false"` - SerialLog string `help:"Path to the serial log file" default:"/tmp/trident-vm-verity-test.log"` + SecureBoot bool `help:"Enable secure boot for the VM" default:"false"` + SerialLog string `help:"Path to the serial log file" default:"/tmp/trident-vm-verity-test.log"` + ImagePattern string `help:"Regex pattern used to find the base VM image (.qcow2) in the artifacts directory" default:"^trident-vm-.*-testimage.qcow2$"` } func (cfg QemuConfig) DeployQemuVM(vmName string, artifactsDir string, outputPath string, verbose bool) error { @@ -32,7 +33,7 @@ func (cfg QemuConfig) DeployQemuVM(vmName string, artifactsDir string, outputPat } // Find image file - imageFile, err := stormfile.FindFile(artifactsDir, "^trident-vm-.*-testimage.qcow2$") + imageFile, err := stormfile.FindFile(artifactsDir, cfg.ImagePattern) if err != nil { return fmt.Errorf("failed to find image file: %w", err) } From 4a5b130bd9e9b50106b38a42f6c5b410d10e1f7f Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Thu, 20 Aug 2026 21:32:37 +0000 Subject: [PATCH 02/11] storm/aclagent: pin annotation prefix and version key via systemd drop-in trident-acl-agent ' s compiled-in defaults changed to the generic acl.microsoft.com/VERSION_ID, so this scenario now explicitly overrides them to acl.azure.com/IMAGE_VERSION (the values it has always exercised) via a drop-in written before the service is enabled/restarted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tools/storm/aclagent/README.md | 29 ++++++++++-------- tools/storm/aclagent/tests/update.go | 44 ++++++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 14 deletions(-) diff --git a/tools/storm/aclagent/README.md b/tools/storm/aclagent/README.md index d872b51938..50292f7011 100644 --- a/tools/storm/aclagent/README.md +++ b/tools/storm/aclagent/README.md @@ -40,18 +40,23 @@ safe for the scenario to deliver the kubeconfig and enable the service once, after `deploy-vm`, rather than baking enablement into the image: the state persists across `run-ab-update`'s finalize the same way the kubeconfig does. -`prepareVmForAclAgent` never writes a `trident-acl-agent.conf` at all - the -agent's compiled-in defaults already cover everything it needs (see the -function's own doc comment): `nebraska.app_id`/`nebraska.endpoint` are -supplied per-request via the update-request annotation's `appId`/`server` -fields instead (see `RunABUpdate`'s `PatchStep.AppId`/`PatchStep.Server`), -`kubernetes.node_name` defaults to the node's real hostname (which the VM -image's Image Customizer config sets to match `TestConfig.NodeName`), and -`kubernetes.api_server` is left unset so the fake kubeconfig's own `server:` -field is used as-is. `prepareVmForAclAgent` writes only that fake -kubeconfig, then runs `systemctl enable --now trident-acl-agent.service`. -Before that runs, the service simply isn't started -- no crash-looping, no -log noise. +`prepareVmForAclAgent` never writes a `trident-acl-agent.conf` at all - +`nebraska.app_id`/`nebraska.endpoint` are supplied per-request via the +update-request annotation's `appId`/`server` fields instead (see +`RunABUpdate`'s `PatchStep.AppId`/`PatchStep.Server`), `kubernetes.node_name` +defaults to the node's real hostname (which the VM image's Image Customizer +config sets to match `TestConfig.NodeName`), and `kubernetes.api_server` is +left unset so the fake kubeconfig's own `server:` field is used as-is. +`kubernetes.annotation_prefix` and `current_version.key`, however, ARE +overridden, via a systemd drop-in +(`/etc/systemd/system/trident-acl-agent.service.d/override.conf`) that +pins them to this scenario's expected `acl.azure.com`/`IMAGE_VERSION` +values (see `proxies/constants.go`) - these used to be the agent's own +compiled-in defaults and needed no override, but no longer are now that +the defaults have moved to the generic `acl.microsoft.com`/`VERSION_ID`. +`prepareVmForAclAgent` writes that drop-in and the fake kubeconfig, then +enables and restarts `trident-acl-agent.service`. Before that runs, the +service simply isn't started -- no crash-looping, no log noise. ## Local usage diff --git a/tools/storm/aclagent/tests/update.go b/tools/storm/aclagent/tests/update.go index e3969baf84..a57f71df28 100644 --- a/tools/storm/aclagent/tests/update.go +++ b/tools/storm/aclagent/tests/update.go @@ -375,9 +375,19 @@ func prepareVmForAclAgent(cfg stormvmconfig.VMConfig, vmIP string, testConfig st // (baseimg-acl-agent.yaml), so the agent's own hostname-derived // default already agrees with the fake apiserver's seeded Node - // no override needed, exactly like a real deployment. - // - kubernetes.kubeconfig, trident.socket, and orchestration.goal_source - // are all already their compiled-in defaults. + // - kubernetes.kubeconfig and trident.socket are also already their + // compiled-in defaults. // trident-acl-agent.conf is simply never written to the VM. + // + // kubernetes.annotation_prefix and current_version.key DO need an + // explicit override, via a systemd drop-in, below: this scenario + // pins its expectations to the AKS-era values (annotation prefix + // "acl.azure.com" - see proxies/constants.go - and current-version key + // "IMAGE_VERSION"), which used to be trident-acl-agent's own compiled-in + // defaults and so needed no override at all. Now that the defaults have + // moved to the generic "acl.microsoft.com"/"VERSION_ID", this scenario + // must set them explicitly to keep exercising the same values it always + // has. // The fake apiserver has no real kubelet-managed kubeconfig backing it, // and it takes plain HTTP with no auth/TLS, so provide a minimal @@ -420,6 +430,36 @@ users: return fmt.Errorf("failed to upload fake kubeconfig to VM: %w", err) } + // Pin this scenario's annotation prefix and current-version key to the + // AKS-era values it has always exercised (see doc comment above), via a + // systemd drop-in rather than trident-acl-agent.conf, matching how a + // real deployment overrides these settings. + overrideConf := "[Service]\n" + + "Environment=TRIDENT_ACL_AGENT_KUBERNETES_ANNOTATION_PREFIX=acl.azure.com\n" + + "Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY=IMAGE_VERSION\n" + + localOverrideFile, err := os.CreateTemp("", "trident-acl-agent-override-*.conf") + if err != nil { + return fmt.Errorf("failed to create local temp file for systemd drop-in: %w", err) + } + defer os.Remove(localOverrideFile.Name()) + if _, err := localOverrideFile.WriteString(overrideConf); err != nil { + localOverrideFile.Close() + return fmt.Errorf("failed to write local temp systemd drop-in: %w", err) + } + if err := localOverrideFile.Close(); err != nil { + return fmt.Errorf("failed to close local temp systemd drop-in: %w", err) + } + if _, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, "sudo mkdir -p /etc/systemd/system/trident-acl-agent.service.d"); err != nil { + return fmt.Errorf("failed to create trident-acl-agent.service.d on VM: %w", err) + } + if err := stormssh.ScpUploadFileWithSudo(cfg, vmIP, localOverrideFile.Name(), "/etc/systemd/system/trident-acl-agent.service.d/override.conf"); err != nil { + return fmt.Errorf("failed to upload systemd drop-in to VM: %w", err) + } + if _, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, "sudo systemctl daemon-reload"); err != nil { + return fmt.Errorf("failed to reload systemd on VM after writing drop-in: %w", err) + } + // Enable (without "--now"), then always issue a single "restart" of // trident-acl-agent.service. Each RunX test case starts its own fresh // fake-apiserver/Nebraska instances, so a plain "enable --now" isn't From 614c9649b2f01a556e46205137c313d9da5f8c79 Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Thu, 20 Aug 2026 21:53:46 +0000 Subject: [PATCH 03/11] storm/aclagent: bake annotation-prefix/version-key drop-in into both images A drop-in written under /etc/systemd/system at runtime lives only on the currently-active root and does not survive this usr-verity image ' s A/B swap - the previous commit ' s runtime SSH write only ever reached the base image ' s root, so the update image booted back to the agent ' s generic acl.microsoft.com/VERSION_ID defaults and never emitted the acl.azure.com/IMAGE_VERSION status this scenario expects, failing post-reboot commit. Bake the same override.conf into both baseimg-acl-agent.yaml and updateimg-acl-agent.yaml via additionalFiles instead, so each image ' s own root already has it. prepareVmForAclAgent no longer writes it at runtime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../base/baseimg-acl-agent.yaml | 9 +++ .../files/trident-acl-agent-override.conf | 3 + .../base/updateimg-acl-agent.yaml | 4 ++ tools/storm/aclagent/README.md | 59 ++++++++++++------- tools/storm/aclagent/tests/update.go | 45 +++----------- 5 files changed, 64 insertions(+), 56 deletions(-) create mode 100644 tests/images/trident-vm-testimage/base/files/trident-acl-agent-override.conf diff --git a/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml b/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml index 4215e1cbdd..87270e8c91 100644 --- a/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml +++ b/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml @@ -176,6 +176,15 @@ os: destination: /etc/systemd/network/99-dhcp-eth0.network - source: files/sudoers-wheel destination: /etc/sudoers.d/wheel + # Pins this scenario's expected annotation prefix/current-version key + # (see proxies/constants.go) to the AKS-era values it has always + # exercised, now that trident-acl-agent's compiled-in defaults are the + # generic acl.microsoft.com/VERSION_ID. Baked into both the base and + # update image builds directly, since a drop-in written at runtime + # under /etc/systemd/system does not survive this usr-verity image's + # A/B root swap. + - source: files/trident-acl-agent-override.conf + destination: /etc/systemd/system/trident-acl-agent.service.d/override.conf services: enable: diff --git a/tests/images/trident-vm-testimage/base/files/trident-acl-agent-override.conf b/tests/images/trident-vm-testimage/base/files/trident-acl-agent-override.conf new file mode 100644 index 0000000000..63b3326a4b --- /dev/null +++ b/tests/images/trident-vm-testimage/base/files/trident-acl-agent-override.conf @@ -0,0 +1,3 @@ +[Service] +Environment=TRIDENT_ACL_AGENT_KUBERNETES_ANNOTATION_PREFIX=acl.azure.com +Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY=IMAGE_VERSION diff --git a/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml b/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml index fbf15e0374..3dcd0e250d 100644 --- a/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml +++ b/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml @@ -184,6 +184,10 @@ os: destination: /etc/systemd/network/99-dhcp-eth0.network - source: files/sudoers-wheel destination: /etc/sudoers.d/wheel + # See baseimg-acl-agent.yaml for why this is baked in rather than + # written at runtime. + - source: files/trident-acl-agent-override.conf + destination: /etc/systemd/system/trident-acl-agent.service.d/override.conf services: enable: diff --git a/tools/storm/aclagent/README.md b/tools/storm/aclagent/README.md index 50292f7011..b56366236d 100644 --- a/tools/storm/aclagent/README.md +++ b/tools/storm/aclagent/README.md @@ -29,34 +29,53 @@ The VM image used by this scenario must already contain: - `tridentd.socket` installed and enabled (starts `tridentd.service` on demand) - `trident-acl-agent` package installed, but **`trident-acl-agent.service` - left disabled** -- it must not start before the fake kubeconfig exists + left disabled in the base image** -- it must not start before the fake + kubeconfig exists. `updateimg-acl-agent.yaml` enables it directly in the + image build instead (see below for why). +- a `/etc/systemd/system/trident-acl-agent.service.d/override.conf` drop-in + baked into **both** `baseimg-acl-agent.yaml` and `updateimg-acl-agent.yaml` + (see below) - the same SSH user/key setup expected by the existing storm servicing scenario -Both the enabled/disabled state of `trident-acl-agent.service` -(`/etc/systemd/system/multi-user.target.wants/...`) and the fake kubeconfig -at `/var/lib/kubelet/kubeconfig` live under paths that are not part of the -A/B-swapped `/usr`/root volume pair in this usr-verity layout. That makes it -safe for the scenario to deliver the kubeconfig and enable the service once, -after `deploy-vm`, rather than baking enablement into the image: the state -persists across `run-ab-update`'s finalize the same way the kubeconfig does. - -`prepareVmForAclAgent` never writes a `trident-acl-agent.conf` at all - +The fake kubeconfig at `/var/lib/kubelet/kubeconfig` lives on a partition +that is not part of the A/B-swapped `/usr`/root volume pair in this +usr-verity layout, so it's safe for `prepareVmForAclAgent` to deliver it +once, at runtime, after `deploy-vm`: it persists across `run-ab-update`'s +finalize/reboot the same way Trident's own persistent partitions do. + +Anything under `/etc/systemd/system` (service enablement state, drop-ins), +by contrast, lives on the swapped root itself and does NOT carry over a +reboot onto the other A/B volume - it has to be baked into whichever image +(base or update) builds that volume: + +- **service enablement**: the base image ships it disabled; + `prepareVmForAclAgent` enables it at runtime for the base image's own + root, but that runtime "enable" doesn't need to (and can't) carry over to + the update image's root - `updateimg-acl-agent.yaml` bakes enablement in + directly instead. +- **the annotation-prefix/current-version-key drop-in**: for the same + reason, this is baked into `additionalFiles` in both + `baseimg-acl-agent.yaml` and `updateimg-acl-agent.yaml`, rather than + written at runtime by `prepareVmForAclAgent`. + +`prepareVmForAclAgent` never writes a `trident-acl-agent.conf`, nor the +systemd drop-in, at all - `nebraska.app_id`/`nebraska.endpoint` are supplied per-request via the update-request annotation's `appId`/`server` fields instead (see `RunABUpdate`'s `PatchStep.AppId`/`PatchStep.Server`), `kubernetes.node_name` defaults to the node's real hostname (which the VM image's Image Customizer config sets to match `TestConfig.NodeName`), and `kubernetes.api_server` is left unset so the fake kubeconfig's own `server:` field is used as-is. -`kubernetes.annotation_prefix` and `current_version.key`, however, ARE -overridden, via a systemd drop-in -(`/etc/systemd/system/trident-acl-agent.service.d/override.conf`) that -pins them to this scenario's expected `acl.azure.com`/`IMAGE_VERSION` -values (see `proxies/constants.go`) - these used to be the agent's own -compiled-in defaults and needed no override, but no longer are now that -the defaults have moved to the generic `acl.microsoft.com`/`VERSION_ID`. -`prepareVmForAclAgent` writes that drop-in and the fake kubeconfig, then -enables and restarts `trident-acl-agent.service`. Before that runs, the -service simply isn't started -- no crash-looping, no log noise. +`kubernetes.annotation_prefix` and `current_version.key` differ from their +compiled-in defaults for this scenario - it pins its expectations to the +AKS-era values `acl.azure.com`/`IMAGE_VERSION` (see `proxies/constants.go`), +which used to be trident-acl-agent's own compiled-in defaults and so needed +no override at all, but no longer are now that the defaults have moved to +the generic `acl.microsoft.com`/`VERSION_ID` - hence the baked-in drop-in +above. `prepareVmForAclAgent` writes only the fake kubeconfig, then enables +and restarts `trident-acl-agent.service`. Before that runs, the service +simply isn't started -- no crash-looping, no log noise. + ## Local usage diff --git a/tools/storm/aclagent/tests/update.go b/tools/storm/aclagent/tests/update.go index a57f71df28..b67018601b 100644 --- a/tools/storm/aclagent/tests/update.go +++ b/tools/storm/aclagent/tests/update.go @@ -379,15 +379,18 @@ func prepareVmForAclAgent(cfg stormvmconfig.VMConfig, vmIP string, testConfig st // compiled-in defaults. // trident-acl-agent.conf is simply never written to the VM. // - // kubernetes.annotation_prefix and current_version.key DO need an - // explicit override, via a systemd drop-in, below: this scenario - // pins its expectations to the AKS-era values (annotation prefix + // kubernetes.annotation_prefix and current_version.key DO differ from + // their compiled-in defaults for this scenario: it pins its + // expectations to the AKS-era values (annotation prefix // "acl.azure.com" - see proxies/constants.go - and current-version key // "IMAGE_VERSION"), which used to be trident-acl-agent's own compiled-in // defaults and so needed no override at all. Now that the defaults have - // moved to the generic "acl.microsoft.com"/"VERSION_ID", this scenario - // must set them explicitly to keep exercising the same values it always - // has. + // moved to the generic "acl.microsoft.com"/"VERSION_ID", both VM images + // (baseimg-acl-agent.yaml and updateimg-acl-agent.yaml) bake in a + // systemd drop-in setting them explicitly - not done here at runtime, + // because a drop-in written under /etc/systemd/system after boot would + // live only on the currently-active root and not survive this + // usr-verity image's A/B root swap. // The fake apiserver has no real kubelet-managed kubeconfig backing it, // and it takes plain HTTP with no auth/TLS, so provide a minimal @@ -430,36 +433,6 @@ users: return fmt.Errorf("failed to upload fake kubeconfig to VM: %w", err) } - // Pin this scenario's annotation prefix and current-version key to the - // AKS-era values it has always exercised (see doc comment above), via a - // systemd drop-in rather than trident-acl-agent.conf, matching how a - // real deployment overrides these settings. - overrideConf := "[Service]\n" + - "Environment=TRIDENT_ACL_AGENT_KUBERNETES_ANNOTATION_PREFIX=acl.azure.com\n" + - "Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY=IMAGE_VERSION\n" - - localOverrideFile, err := os.CreateTemp("", "trident-acl-agent-override-*.conf") - if err != nil { - return fmt.Errorf("failed to create local temp file for systemd drop-in: %w", err) - } - defer os.Remove(localOverrideFile.Name()) - if _, err := localOverrideFile.WriteString(overrideConf); err != nil { - localOverrideFile.Close() - return fmt.Errorf("failed to write local temp systemd drop-in: %w", err) - } - if err := localOverrideFile.Close(); err != nil { - return fmt.Errorf("failed to close local temp systemd drop-in: %w", err) - } - if _, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, "sudo mkdir -p /etc/systemd/system/trident-acl-agent.service.d"); err != nil { - return fmt.Errorf("failed to create trident-acl-agent.service.d on VM: %w", err) - } - if err := stormssh.ScpUploadFileWithSudo(cfg, vmIP, localOverrideFile.Name(), "/etc/systemd/system/trident-acl-agent.service.d/override.conf"); err != nil { - return fmt.Errorf("failed to upload systemd drop-in to VM: %w", err) - } - if _, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, "sudo systemctl daemon-reload"); err != nil { - return fmt.Errorf("failed to reload systemd on VM after writing drop-in: %w", err) - } - // Enable (without "--now"), then always issue a single "restart" of // trident-acl-agent.service. Each RunX test case starts its own fresh // fake-apiserver/Nebraska instances, so a plain "enable --now" isn't From 41dbee4c74779502b18b8c0c3b07efb37a006926 Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Fri, 21 Aug 2026 20:01:08 +0000 Subject: [PATCH 04/11] storm/aclagent: use UUID-format operation IDs in test scenarios trident-acl-agent now rejects non-UUID operation_id values in UpdateRequest per the formal schema (validate() added in PR 730). The storm aclagent test scenarios used human-readable placeholder operation IDs (stage-op, finalize-op, rollback-op, rollback-op-2), which are no longer accepted and caused run-ab-update/run-rollback to fail with InvalidRequest instead of Success. Replace them with fixed UUID-format placeholders, matching what real callers (AKS ACL Update Service) actually send. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tools/storm/aclagent/tests/rollback.go | 10 +++++----- tools/storm/aclagent/tests/update.go | 8 ++++---- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/tools/storm/aclagent/tests/rollback.go b/tools/storm/aclagent/tests/rollback.go index f267d3ac96..b638e62121 100644 --- a/tools/storm/aclagent/tests/rollback.go +++ b/tools/storm/aclagent/tests/rollback.go @@ -53,8 +53,8 @@ func RunRollback(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al rp := &stormproxies.RPClient{APIServerURL: fmt.Sprintf("http://%s:%d", testConfig.HostEndpointIP, testConfig.APIServerPort), NodeName: testConfig.NodeName} scenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ - {Patch: &stormproxies.PatchStep{NodeUpdateID: "22222222-2222-2222-2222-222222222222", OperationID: "rollback-op", Operation: "rollback"}}, - {Expect: &stormproxies.ExpectStep{OperationID: "rollback-op", Operation: "rollback", Code: "Success", Timeout: 180 * time.Second}}, + {Patch: &stormproxies.PatchStep{NodeUpdateID: "22222222-2222-2222-2222-222222222222", OperationID: "cccccccc-3333-3333-3333-333333333333", Operation: "rollback"}}, + {Expect: &stormproxies.ExpectStep{OperationID: "cccccccc-3333-3333-3333-333333333333", Operation: "rollback", Code: "Success", Timeout: 180 * time.Second}}, }} report, err := rp.RunScenario(ctx, scenario) logScenarioTimeline("rollback stage/finalize", report) @@ -84,7 +84,7 @@ func RunRollback(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al } finalScenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ - {Expect: &stormproxies.ExpectStep{OperationID: "rollback-op", Operation: "commit", Code: "Success", Timeout: 180 * time.Second}}, + {Expect: &stormproxies.ExpectStep{OperationID: "cccccccc-3333-3333-3333-333333333333", Operation: "commit", Code: "Success", Timeout: 180 * time.Second}}, }} finalReport, err := rp.RunScenario(ctx, finalScenario) logScenarioTimeline("post-rollback-reboot commit", finalReport) @@ -111,8 +111,8 @@ func RunRollback(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al // rebooting the node again for no reason. This exercises that fix // end-to-end against the real tridentd, not just the mock. secondScenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ - {Patch: &stormproxies.PatchStep{NodeUpdateID: "33333333-3333-3333-3333-333333333333", OperationID: "rollback-op-2", Operation: "rollback"}}, - {Expect: &stormproxies.ExpectStep{OperationID: "rollback-op-2", Operation: "rollback", Code: "OperationFailed", Timeout: 60 * time.Second}}, + {Patch: &stormproxies.PatchStep{NodeUpdateID: "33333333-3333-3333-3333-333333333333", OperationID: "dddddddd-4444-4444-4444-444444444444", Operation: "rollback"}}, + {Expect: &stormproxies.ExpectStep{OperationID: "dddddddd-4444-4444-4444-444444444444", Operation: "rollback", Code: "OperationFailed", Timeout: 60 * time.Second}}, }} secondReport, err := rp.RunScenario(ctx, secondScenario) logScenarioTimeline("second rollback with empty chain", secondReport) diff --git a/tools/storm/aclagent/tests/update.go b/tools/storm/aclagent/tests/update.go index b67018601b..7c497eb76d 100644 --- a/tools/storm/aclagent/tests/update.go +++ b/tools/storm/aclagent/tests/update.go @@ -281,9 +281,9 @@ func RunABUpdate(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al } scenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ - {Patch: &stormproxies.PatchStep{NodeUpdateID: "11111111-1111-1111-1111-111111111111", OperationID: "stage-op", Operation: "stage", TargetOSImageVersion: testConfig.TargetVersion, Server: nebraskaServer, AppId: nebraskaAppID, Track: nebraskaTrack}}, - {Expect: &stormproxies.ExpectStep{OperationID: "stage-op", Operation: "stage", Code: "Success", Timeout: 120 * time.Second}}, - {Patch: &stormproxies.PatchStep{NodeUpdateID: "11111111-1111-1111-1111-111111111111", OperationID: "finalize-op", Operation: "finalize", TargetOSImageVersion: testConfig.TargetVersion, Server: nebraskaServer, AppId: nebraskaAppID, Track: nebraskaTrack}}, + {Patch: &stormproxies.PatchStep{NodeUpdateID: "11111111-1111-1111-1111-111111111111", OperationID: "aaaaaaaa-1111-1111-1111-111111111111", Operation: "stage", TargetOSImageVersion: testConfig.TargetVersion, Server: nebraskaServer, AppId: nebraskaAppID, Track: nebraskaTrack}}, + {Expect: &stormproxies.ExpectStep{OperationID: "aaaaaaaa-1111-1111-1111-111111111111", Operation: "stage", Code: "Success", Timeout: 120 * time.Second}}, + {Patch: &stormproxies.PatchStep{NodeUpdateID: "11111111-1111-1111-1111-111111111111", OperationID: "bbbbbbbb-2222-2222-2222-222222222222", Operation: "finalize", TargetOSImageVersion: testConfig.TargetVersion, Server: nebraskaServer, AppId: nebraskaAppID, Track: nebraskaTrack}}, }} report, err := rp.RunScenario(ctx, scenario) logScenarioTimeline("stage/finalize", report) @@ -314,7 +314,7 @@ func RunABUpdate(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al // with that same config at boot, so no re-delivery is needed here. finalScenario := &stormproxies.Scenario{Steps: []stormproxies.ScenarioStep{ - {Expect: &stormproxies.ExpectStep{OperationID: "finalize-op", Operation: "commit", Code: "Success", Timeout: 180 * time.Second}}, + {Expect: &stormproxies.ExpectStep{OperationID: "bbbbbbbb-2222-2222-2222-222222222222", Operation: "commit", Code: "Success", Timeout: 180 * time.Second}}, }} finalReport, err := rp.RunScenario(ctx, finalScenario) logScenarioTimeline("post-reboot commit", finalReport) From 5c2a065065061996593e98346ef01799ce1b0ff6 Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Fri, 21 Aug 2026 20:17:16 +0000 Subject: [PATCH 05/11] storm/aclagent: fix Copilot review findings (PR 731) Go fixes: - apiserver.go: decode handlePatch into typed struct so DisallowUnknownFields is effective; fix handleWatch double-header write after WriteHeader(200) has already been sent - rp.go: propagate expectStatus JSON decode errors instead of swallowing them - trident.go: return immediately after FailFromError so a failed test case is not also logged as passed - vm.go: DeployVM/CleanupVM now error on an unrecognized VM platform instead of silently returning nil - nebraska.go: unset NEBRASKA_DB_URL on shutdown so it does not leak into the next test case in the same process - config.go: correct ImagePath help text to match FindFile actual behavior (errors on multiple matches, not first-match) - update.go/rollback.go: guard MkdirAll(OutputPath) against an empty path; bind fake servers to HostEndpointIP instead of 0.0.0.0; fix shell != portability; correct misleading comment about why the agent needs reconnecting after reboot Docs: - Rewrite README.md, trident-vm-testimage/README.md, and TridentAclAgent-Tests.md to describe the current annotation-driven design (fake kubeconfig + per-request Nebraska fields) instead of the earlier label-driven/config-file/reverse-SSH design, and fix the --artifacts-dir default and missing run-rollback test case. Verified: full storm aclagent E2E suite (6/6 PASS) after these changes, including the HostEndpointIP bind change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 98412869-e5c8-4f3f-a97c-fe870db70701 --- .../Testing/TridentAclAgent-Tests.md | 82 ++++++++++++------- tests/images/trident-vm-testimage/README.md | 15 ++-- tools/storm/aclagent/README.md | 39 ++++++--- tools/storm/aclagent/proxies/apiserver.go | 17 +++- tools/storm/aclagent/proxies/nebraska.go | 3 + tools/storm/aclagent/proxies/rp.go | 5 +- tools/storm/aclagent/tests/rollback.go | 14 ++-- tools/storm/aclagent/tests/update.go | 24 +++--- tools/storm/aclagent/tests/vm.go | 4 + tools/storm/aclagent/trident.go | 2 + tools/storm/aclagent/utils/config/config.go | 2 +- 11 files changed, 135 insertions(+), 72 deletions(-) diff --git a/docs/Development/Testing/TridentAclAgent-Tests.md b/docs/Development/Testing/TridentAclAgent-Tests.md index a3004c6a59..8341e46b94 100644 --- a/docs/Development/Testing/TridentAclAgent-Tests.md +++ b/docs/Development/Testing/TridentAclAgent-Tests.md @@ -5,13 +5,14 @@ sidebar_position: 9 # Trident ACL Agent Tests `storm-trident run aclagent` is the single supported validation entrypoint for -the label-driven `trident-acl-agent` protocol described in the ACL AKS -node-label design. Unlike [Servicing Tests](Servicing-Tests.md), which drive -Trident's own `stage`/`finalize` gRPC calls directly, this scenario validates -`trident-acl-agent` itself: it deploys a VM, starts fake in-process test -doubles for the Kubernetes API server and the Nebraska/Omaha update server, -seeds bootstrap node labels, and lets the real `trident-acl-agent` binary -running inside the VM drive a full A/B update against those fakes. +the annotation-driven `trident-acl-agent` protocol. Unlike [Servicing +Tests](Servicing-Tests.md), which drive Trident's own `stage`/`finalize` gRPC +calls directly, this scenario validates `trident-acl-agent` itself: it +deploys a VM, starts fake in-process test doubles for the Kubernetes API +server and the Nebraska/Omaha update server, patches update-request +annotations onto the fake Node, and lets the real `trident-acl-agent` binary +running inside the VM drive a full A/B update (and, separately, a rollback) +against those fakes. There is intentionally no fake `tridentd` โ€” the scenario talks to the real `tridentd` and real `trident-acl-agent` running inside the VM. @@ -19,10 +20,13 @@ There is intentionally no fake `tridentd` โ€” the scenario talks to the real ## What It Validates - `trident-acl-agent` watching its own Kubernetes Node object for - RP-authored label changes (via `kube::runtime::watcher()`) -- Reading the update image URL/hash from labels and triggering a real - Trident `stage` + `finalize` A/B update through the normal gRPC path -- Patching back observed-state labels/annotations as the update progresses + RP-authored `acl.azure.com/update-request` annotation changes (via + `kube::runtime::watcher()`) +- Reading the target OS image version, Nebraska server/appId/track from the + request annotation and triggering a real Trident `stage` + `finalize` A/B + update (or `rollback` + rollback `finalize`) through the normal gRPC path +- Writing back progress/result via the `acl.azure.com/update-status` and + `acl.azure.com/update-commit-status` annotations as the update progresses - Resuming correctly after a real reboot (see [Reboot Choice](#reboot-choice)) ## VM Image Contents @@ -32,17 +36,29 @@ The VM image used by this scenario must already contain: - `tridentd.socket` installed and enabled (starts `tridentd.service` on demand) - `trident-acl-agent` package installed, but **`trident-acl-agent.service` - left disabled** โ€” it must not start before a config file exists + left disabled** in the base image โ€” it must not start before the fake + kubeconfig exists (the harness delivers it at runtime and enables/starts + the service itself; the update image, by contrast, bakes enablement in + directly, since there is no test harness left to run `systemctl enable + --now` after a real A/B update boots into it) - the same SSH user/key setup expected by the [servicing](Servicing-Tests.md) scenario -Both the enabled/disabled state of `trident-acl-agent.service` and -`/etc/trident/trident-acl-agent.conf` live under `/etc`, which is not part of -the A/B-swapped `/usr`/root volume pair in this usr-verity image layout. That -makes it safe for the scenario to write the config and enable the service -once, after `deploy-vm`, rather than baking enablement into the image โ€” the -state persists across `run-ab-update`'s finalize the same way the config file -does. +The harness never writes `/etc/trident/trident-acl-agent.conf` at all โ€” +`nebraska.endpoint`/`nebraska.app_id`/`nebraska.track` are supplied per-request +via the update-request annotation's `server`/`appId`/`track` fields instead, +and `kubernetes.node_name` defaults to the node's own real hostname (which +the image's hostname is set to match). What the harness does deliver, once +at runtime after `deploy-vm`, is a fake kubeconfig at +`/var/lib/kubelet/kubeconfig` pointing at the fake apiserver. That path lives +on its own dedicated ext4 partition (not part of the A/B-swapped `/usr`/root +volume pair in this usr-verity image layout), so it persists across +`run-ab-update`'s finalize reboot without needing to be re-delivered. +`trident-acl-agent.service`'s enablement state, by contrast, lives on the +swapped root itself and does not carry over a reboot onto the other A/B +volume โ€” the harness re-runs its short prepare/restart step after each +reboot to reconnect the agent to that test case's fresh fake-apiserver +instance, not to re-create the kubeconfig. ## Prerequisites @@ -136,30 +152,34 @@ sudo bin/storm-trident run aclagent \ The scenario runs these test cases in order: 1. **deploy-vm** โ€” Copies the base qcow2 image and creates a QEMU VM -2. **check-deployment** โ€” Verifies the VM booted and is accessible via SSH; - writes `/etc/trident/trident-acl-agent.conf` pointing at the - `localhost:` endpoints storm reverse-SSH-forwards into the VM, then - runs `systemctl enable --now trident-acl-agent.service` +2. **check-deployment** โ€” Verifies the VM booted and is accessible via SSH 3. **run-ab-update** โ€” Starts the fake apiserver and fake Nebraska/Omaha - endpoints in-process, seeds bootstrap node labels, patches the desired - update-image label, and waits for `trident-acl-agent` to drive a real - Trident A/B update to completion (including a real reboot) -4. **collect-logs** โ€” Fetches `trident-acl-agent` and Trident logs from the + endpoints in-process, delivers a fake kubeconfig and restarts + `trident-acl-agent.service`, patches the `acl.azure.com/update-request` + annotation, and waits for `trident-acl-agent` to drive a real Trident A/B + update to completion (including a real reboot) +4. **run-rollback** โ€” Exercises the `rollback` annotation end-to-end against + tridentd's `RollbackService` gRPC API, followed by a real reboot and + post-reboot commit; must run after `run-ab-update` in the same VM + lifetime, since it rolls back to the volume active before that update +5. **collect-logs** โ€” Fetches `trident-acl-agent` and Trident logs from the VM via SSH; also runs automatically (with a `journalctl` dump for - `trident-acl-agent.service`) if `run-ab-update` times out waiting for the - service to become active, to make crash-loops self-diagnosing -5. **cleanup-vm** โ€” Destroys the QEMU VM + `trident-acl-agent.service`) if an update/rollback step times out waiting + for the service to become active, to make crash-loops self-diagnosing +6. **cleanup-vm** โ€” Destroys the QEMU VM ### Flags | Flag | Description | Default | |------|-------------|---------| -| `--artifacts-dir` | Directory containing VM images | `/tmp` | +| `--artifacts-dir` | Directory containing VM images | `.` | | `--output-path` | Output directory for logs | `./output` | | `--platform` | `qemu` or `azure` | `qemu` | | `--ssh-private-key-path` | Path to SSH private key | `~/.ssh/id_rsa` | | `--api-server-port` | Port for the fake Kubernetes API server | `18080` | | `--nebraska-port` | Port for the fake Nebraska/Omaha server | `18081` | +| `--host-endpoint-ip` | Host IP the VM reaches the fake endpoints at | `192.168.122.1` | +| `--image-path` | Real `.cosi` update image to serve during staging | first `*.cosi` found under `--artifacts-dir` | | `--verbose` | Enable verbose logging | `false` | | `--test-case-to-run` | Run a specific test case only | `all` | diff --git a/tests/images/trident-vm-testimage/README.md b/tests/images/trident-vm-testimage/README.md index 51d1d3c3b3..d7f37b5e38 100644 --- a/tests/images/trident-vm-testimage/README.md +++ b/tests/images/trident-vm-testimage/README.md @@ -16,13 +16,14 @@ For both, a set of corresponding update images is available. The ACL-agent variant reuses the servicing-style VM image layout but additionally installs `trident-acl-agent` so storm ACL-agent scenarios can drive a real in-guest agent talking to `tridentd`. The base (qcow2) image installs the package but -leaves `trident-acl-agent.service` disabled -- the storm scenario enables and -starts it itself after seeding a real config. Only the update image enables -`trident-acl-agent.service` by default, since after a real A/B update boots -into it there is no test harness left to `systemctl enable --now` it. Neither -image preseeds /etc/trident/trident-acl-agent.conf with runner-specific -tunnel ports; the test scenario should SSH in after boot and write the real -localhost proxy endpoints for Nebraska and the Kubernetes API server. +leaves `trident-acl-agent.service` disabled -- the storm scenario delivers a +fake kubeconfig at runtime and enables/starts the service itself. Only the +update image enables `trident-acl-agent.service` by default, since after a +real A/B update boots into it there is no test harness left to `systemctl +enable --now` it. Neither image preseeds `/etc/trident/trident-acl-agent.conf` +-- the harness never writes that file at all; instead it delivers a fake +kubeconfig to `/var/lib/kubelet/kubeconfig` and supplies the Nebraska +endpoint/appId/track per-request via the update-request annotation. ## Additional Prerequisites diff --git a/tools/storm/aclagent/README.md b/tools/storm/aclagent/README.md index b56366236d..6d0418de51 100644 --- a/tools/storm/aclagent/README.md +++ b/tools/storm/aclagent/README.md @@ -1,14 +1,15 @@ # Trident ACL agent storm scenario `storm-trident aclagent` is the single supported validation entrypoint for the -label-driven trident ACL agent protocol. +annotation-driven trident ACL agent protocol. ## What it does - deploys a QEMU or Azure VM using the existing storm VM helpers - starts the fake single-node Kubernetes apiserver in-process inside the storm binary - starts the fake Nebraska/Omaha endpoint in-process inside the storm binary -- seeds bootstrap node annotations and simulated Ready flips with an in-process kubelet helper +- seeds bootstrap node annotations and simulated Ready flips by mutating the + fake Node directly via `NodeStore` - talks to the real `tridentd` and real `trident-acl-agent` running inside the VM - lets `trident-acl-agent` issue a real `systemctl reboot` on finalize, then polls SSH until it drops and comes back up to confirm the reboot actually happened @@ -87,24 +88,32 @@ wrong key. ```bash make bin/storm-trident -./bin/storm-trident run aclagent deploy-vm \ +./bin/storm-trident run aclagent -- --test-case-to-run=deploy-vm \ --artifacts-dir --ssh-private-key-path /id_rsa -./bin/storm-trident run aclagent check-deployment \ +./bin/storm-trident run aclagent -- --test-case-to-run=check-deployment \ --artifacts-dir --ssh-private-key-path /id_rsa -./bin/storm-trident run aclagent run-ab-update \ +./bin/storm-trident run aclagent -- --test-case-to-run=run-ab-update \ --artifacts-dir --ssh-private-key-path /id_rsa -./bin/storm-trident run aclagent run-rollback \ +./bin/storm-trident run aclagent -- --test-case-to-run=run-rollback \ --artifacts-dir --ssh-private-key-path /id_rsa -./bin/storm-trident run aclagent collect-logs \ +./bin/storm-trident run aclagent -- --test-case-to-run=collect-logs \ --artifacts-dir --ssh-private-key-path /id_rsa -./bin/storm-trident run aclagent cleanup-vm \ +./bin/storm-trident run aclagent -- --test-case-to-run=cleanup-vm \ + --artifacts-dir --ssh-private-key-path /id_rsa +``` + +Or, to run every test case in the same VM lifetime (the default, +`--test-case-to-run=all`, and how the pipeline runs it): + +```bash +./bin/storm-trident run aclagent -- \ --artifacts-dir --ssh-private-key-path /id_rsa ``` Common overrides mirror other storm VM scenarios, for example: ```bash -./bin/storm-trident run aclagent run-ab-update \ +./bin/storm-trident run aclagent -- \ --platform qemu \ --artifacts-dir ./artifacts \ --output-path ./output/aclagent \ @@ -115,10 +124,14 @@ Common overrides mirror other storm VM scenarios, for example: ## Reboot choice -This scenario keeps the shim-based reboot interception from the old tester. -That is less realistic than a full VM reboot, but it keeps the test deterministic -and lets the storm runner hold the reverse SSH tunnels and in-process fake services -steady while the agent drives the finalize path. +This scenario waits out a real VM reboot rather than using a shim: +`trident-acl-agent` issues a genuine `systemctl reboot` on finalize, and +`waitForVmRebootAndSshBack` polls SSH until it goes unreachable (confirming +the reboot actually happened) and then reachable again (confirming the VM +came back up). The fake apiserver/Nebraska/image-server endpoints are bound +to `HostEndpointIP` (the libvirt NAT gateway address), which the VM can +reach directly โ€” this avoids relying on reverse SSH tunnels, which don't +survive the VM actually going down for a real reboot. ## `run-rollback` diff --git a/tools/storm/aclagent/proxies/apiserver.go b/tools/storm/aclagent/proxies/apiserver.go index 73edf1ba96..ed67ea6eb6 100644 --- a/tools/storm/aclagent/proxies/apiserver.go +++ b/tools/storm/aclagent/proxies/apiserver.go @@ -9,6 +9,7 @@ import ( "strings" "sync" + "github.com/sirupsen/logrus" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -272,12 +273,15 @@ func (s *APIServer) handlePatch(w http.ResponseWriter, r *http.Request) { defer r.Body.Close() body := json.NewDecoder(r.Body) body.DisallowUnknownFields() - var raw map[string]any - if err := body.Decode(&raw); err != nil { + // Decode into the typed metadataPatch (not map[string]any) so + // DisallowUnknownFields actually rejects unexpected fields, instead of + // silently accepting arbitrary keys. + var patch metadataPatch + if err := body.Decode(&patch); err != nil { http.Error(w, fmt.Sprintf("invalid patch body: %v", err), http.StatusBadRequest) return } - bytes, err := json.Marshal(raw) + bytes, err := json.Marshal(patch) if err != nil { http.Error(w, fmt.Sprintf("failed to re-marshal patch: %v", err), http.StatusInternalServerError) return @@ -301,7 +305,11 @@ func (s *APIServer) handleWatch(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) if err := writeWatchEvent(w, "ADDED", current); err != nil { - http.Error(w, err.Error(), http.StatusInternalServerError) + // Headers (and a 200 status) have already been written, so calling + // http.Error here would just append a second, malformed + // header/body onto the response. Log and close the connection + // instead. + logrus.Errorf("failed to write initial watch event: %v", err) return } flusher.Flush() @@ -314,6 +322,7 @@ func (s *APIServer) handleWatch(w http.ResponseWriter, r *http.Request) { return } if err := writeWatchEvent(w, "MODIFIED", node); err != nil { + logrus.Errorf("failed to write watch event: %v", err) return } flusher.Flush() diff --git a/tools/storm/aclagent/proxies/nebraska.go b/tools/storm/aclagent/proxies/nebraska.go index a9f024320c..dbb0e38ed4 100644 --- a/tools/storm/aclagent/proxies/nebraska.go +++ b/tools/storm/aclagent/proxies/nebraska.go @@ -176,6 +176,9 @@ func (p *NebraskaProxy) ListenAndServe(ctx context.Context, listenAddr string) ( <-ctx.Done() _ = server.Shutdown(context.Background()) stopEphemeralPostgres(containerID) + // Unset so it doesn't leak into other tests/scenarios running in + // this same process after this proxy has shut down. + _ = os.Unsetenv("NEBRASKA_DB_URL") }() go func() { _ = server.Serve(listener) }() return listener, nil diff --git a/tools/storm/aclagent/proxies/rp.go b/tools/storm/aclagent/proxies/rp.go index 2d61ed0df4..8df7ed2b79 100644 --- a/tools/storm/aclagent/proxies/rp.go +++ b/tools/storm/aclagent/proxies/rp.go @@ -80,7 +80,10 @@ func (c *RPClient) expectStatus(ctx context.Context, index int, step *ExpectStep if err != nil { return nil, err } - status, _ := decodeStatus(node, annotationKey) + status, err := decodeStatus(node, annotationKey) + if err != nil { + return nil, fmt.Errorf("failed to decode %s annotation: %w", annotationKey, err) + } if status != nil { lastObserved = map[string]string{"operation-id": status.OperationID, "operation": status.Operation, "code": status.Code} if status.Code == step.Code && (step.OperationID == "" || status.OperationID == step.OperationID) && (step.Operation == "" || status.Operation == step.Operation) { diff --git a/tools/storm/aclagent/tests/rollback.go b/tools/storm/aclagent/tests/rollback.go index b638e62121..7552ecbbb8 100644 --- a/tools/storm/aclagent/tests/rollback.go +++ b/tools/storm/aclagent/tests/rollback.go @@ -24,8 +24,10 @@ func RunRollback(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al if err != nil { return fmt.Errorf("failed to get VM IP: %w", err) } - if err := os.MkdirAll(testConfig.OutputPath, 0o755); err != nil { - return err + if testConfig.OutputPath != "" { + if err := os.MkdirAll(testConfig.OutputPath, 0o755); err != nil { + return err + } } ctx, cancel := context.WithCancel(context.Background()) @@ -76,9 +78,11 @@ func RunRollback(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al } nodeStore.SetReadyCondition(true) - // Same rationale as run-ab-update: the rollback reboot lands on the - // previous root, which needs the fake kubeconfig re-delivered before it - // can talk to the fake apiserver again. + // Restart the agent so it reconnects to this test case's fresh fake + // apiserver instance (see prepareVmForAclAgent's doc comment in + // update.go). /var/lib/kubelet is its own dedicated ext4 partition, so + // the fake kubeconfig itself already persists across the rollback + // reboot's root swap and doesn't need to be re-delivered. if err := prepareVmForAclAgent(vmConfig.VMConfig, vmIP, testConfig); err != nil { return fmt.Errorf("failed to reconfigure ACL agent on post-rollback-reboot root: %w", err) } diff --git a/tools/storm/aclagent/tests/update.go b/tools/storm/aclagent/tests/update.go index 7c497eb76d..0c3d716c20 100644 --- a/tools/storm/aclagent/tests/update.go +++ b/tools/storm/aclagent/tests/update.go @@ -145,8 +145,10 @@ func RunABUpdate(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al if err != nil { return fmt.Errorf("failed to get VM IP: %w", err) } - if err := os.MkdirAll(testConfig.OutputPath, 0o755); err != nil { - return err + if testConfig.OutputPath != "" { + if err := os.MkdirAll(testConfig.OutputPath, 0o755); err != nil { + return err + } } ctx, cancel := context.WithCancel(context.Background()) @@ -154,11 +156,13 @@ func RunABUpdate(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al nodeStore := stormproxies.NewNodeStore(stormproxies.NewSeedNode(testConfig.NodeName, map[string]string{})) apiServer := stormproxies.NewAPIServer(testConfig.NodeName, nodeStore) - // Bind on all interfaces (not 127.0.0.1) so the VM can reach the fake - // apiserver directly over the libvirt NAT network at testConfig.HostEndpointIP, - // instead of relying on reverse SSH tunnels. Tunnels don't survive a real - // VM reboot; a real host IP does. - if _, err := apiServer.ListenAndServe(ctx, fmt.Sprintf("0.0.0.0:%d", testConfig.APIServerPort)); err != nil { + // Bind on HostEndpointIP (not 127.0.0.1) so the VM can reach the fake + // apiserver directly over the libvirt NAT network, instead of relying + // on reverse SSH tunnels (which don't survive a real VM reboot; a real + // host IP does). Binding to that specific address rather than 0.0.0.0 + // avoids unintentionally exposing the fake server on every other host + // interface too. + if _, err := apiServer.ListenAndServe(ctx, fmt.Sprintf("%s:%d", testConfig.HostEndpointIP, testConfig.APIServerPort)); err != nil { return fmt.Errorf("failed to start fake apiserver: %w", err) } @@ -188,7 +192,7 @@ func RunABUpdate(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al return fmt.Errorf("failed to hash image %s: %w", imagePath, err) } imageServer := &stormproxies.ImageServer{ImagePath: imagePath} - if _, err := imageServer.ListenAndServe(ctx, fmt.Sprintf("0.0.0.0:%d", testConfig.ImageServerPort)); err != nil { + if _, err := imageServer.ListenAndServe(ctx, fmt.Sprintf("%s:%d", testConfig.HostEndpointIP, testConfig.ImageServerPort)); err != nil { return fmt.Errorf("failed to start fake image server: %w", err) } nebraskaCodebase = fmt.Sprintf("http://%s:%d/", testConfig.HostEndpointIP, testConfig.ImageServerPort) @@ -203,7 +207,7 @@ func RunABUpdate(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al SHA384: nebraskaSHA384, PackageName: nebraskaPackageName, }} - if _, err := nebraska.ListenAndServe(ctx, fmt.Sprintf("0.0.0.0:%d", testConfig.NebraskaPort)); err != nil { + if _, err := nebraska.ListenAndServe(ctx, fmt.Sprintf("%s:%d", testConfig.HostEndpointIP, testConfig.NebraskaPort)); err != nil { return fmt.Errorf("failed to start fake Nebraska endpoint: %w", err) } @@ -459,7 +463,7 @@ users: // written to the VM at all - app_id/endpoint are only ever supplied // per-request via the update-request annotation's `appId`/`server` // fields. - "! sudo test -e /etc/trident/trident-acl-agent.conf", + "sudo test ! -e /etc/trident/trident-acl-agent.conf", }, " && ") if _, err := stormssh.SshCommandCombinedOutput(cfg, vmIP, command); err != nil { return fmt.Errorf("failed to prepare VM for ACL agent: %w", err) diff --git a/tools/storm/aclagent/tests/vm.go b/tools/storm/aclagent/tests/vm.go index a80ef66eb4..de8a911feb 100644 --- a/tools/storm/aclagent/tests/vm.go +++ b/tools/storm/aclagent/tests/vm.go @@ -25,6 +25,8 @@ func DeployVM(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.AllVM if err := vmConfig.AzureConfig.DeployAzureVM(vmConfig.VMConfig.Name, vmConfig.VMConfig.User); err != nil { return fmt.Errorf("failed to deploy azure vm: %w", err) } + } else { + return fmt.Errorf("unsupported VM platform '%s'", vmConfig.VMConfig.Platform) } return nil } @@ -38,6 +40,8 @@ func CleanupVM(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.AllV if err := vmConfig.QemuConfig.CleanupQemuVM(vmConfig.VMConfig.Name); err != nil { return fmt.Errorf("failed to cleanup QEMU VM: %w", err) } + } else { + return fmt.Errorf("unsupported VM platform '%s'", vmConfig.VMConfig.Platform) } return nil } diff --git a/tools/storm/aclagent/trident.go b/tools/storm/aclagent/trident.go index 2f5ab036f5..016a34fa20 100644 --- a/tools/storm/aclagent/trident.go +++ b/tools/storm/aclagent/trident.go @@ -62,11 +62,13 @@ func (s *TridentAclAgentScenario) runTestCase(tc storm.TestCase, testFunc func(s testCaseSpecificConfig.OutputPath = filepath.Join(testCaseSpecificConfig.OutputPath, tc.Name()) if err := os.MkdirAll(testCaseSpecificConfig.OutputPath, 0o755); err != nil { tc.FailFromError(err) + return nil } } if err := testFunc(testCaseSpecificConfig, stormvmconfig.AllVMConfig{VMConfig: s.args.VMConfig, QemuConfig: s.args.QemuConfig, AzureConfig: s.args.AzureConfig}); err != nil { logrus.Infof("test case '%s' failed", tc.Name()) tc.FailFromError(err) + return nil } logrus.Infof("test case '%s' passed", tc.Name()) return nil diff --git a/tools/storm/aclagent/utils/config/config.go b/tools/storm/aclagent/utils/config/config.go index fa5ce93f55..eaa52b4d7a 100644 --- a/tools/storm/aclagent/utils/config/config.go +++ b/tools/storm/aclagent/utils/config/config.go @@ -11,7 +11,7 @@ type TestConfig struct { NebraskaPackageName string `help:"Package name returned by the fake Nebraska endpoint (overridden by the image file name when ImagePath is set)" default:"acl.cosi"` NebraskaCodebase string `help:"Base URL returned by the fake Nebraska endpoint (overridden to point at the fake image server when ImagePath is set)" default:"https://example.invalid/images/"` NebraskaSHA384 string `help:"SHA384 returned by the fake Nebraska endpoint (overridden by the real hash of ImagePath when set)" default:"111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"` - ImagePath string `help:"Path to a real OS update image (e.g. a .cosi file) to serve to tridentd during staging; when set, this takes precedence over NebraskaCodebase/NebraskaPackageName/NebraskaSHA384. When empty, the first *.cosi file found under ArtifactsDir is used."` + ImagePath string `help:"Path to a real OS update image (e.g. a .cosi file) to serve to tridentd during staging; when set, this takes precedence over NebraskaCodebase/NebraskaPackageName/NebraskaSHA384. When empty, ArtifactsDir must contain exactly one *.cosi file, which is used."` ImageServerPort int `help:"Runner port exposed into VM for the fake image server" default:"18082"` NodeName string `help:"Node name served by the fake apiserver; must match the VM image's hostname (Image Customizer 'hostname' setting in baseimg-acl-agent.yaml), since trident-acl-agent's [kubernetes].node_name defaults to the node's own real hostname" default:"trident-acl-agent-testimg"` HostEndpointIP string `help:"Host IP the VM can reach the fake apiserver/Nebraska endpoints at" default:"192.168.122.1"` From 5f6b5159ea24424205e1c2e271095c1ba87d0ed8 Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Fri, 21 Aug 2026 20:38:36 +0000 Subject: [PATCH 06/11] storm/aclagent: fix latest Copilot review findings (PR 731) - rollback.go: bind the fake apiserver to HostEndpointIP instead of 0.0.0.0, matching run-ab-update and avoiding unintended exposure - qemu/qemu.go: escape the literal dot in the default ImagePattern regex (.qcow2 -> \.qcow2) so it does not match arbitrary characters - docs/Development/Testing/Testing.md: fix stale "label-driven" wording for the ACL agent test link (protocol is annotation-driven) Verified: full storm aclagent E2E suite (6/6 PASS) after these changes, including the rollback HostEndpointIP bind. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 98412869-e5c8-4f3f-a97c-fe870db70701 --- docs/Development/Testing/Testing.md | 2 +- tools/storm/aclagent/tests/rollback.go | 2 +- tools/storm/utils/vm/qemu/qemu.go | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Development/Testing/Testing.md b/docs/Development/Testing/Testing.md index 22d318d9d3..af053c47fc 100644 --- a/docs/Development/Testing/Testing.md +++ b/docs/Development/Testing/Testing.md @@ -55,7 +55,7 @@ manual rollback chains without using `netlaunch` or an installer ISO. - [Rollback Tests](Rollback-Tests.md) โ€” full rollback chain (A/B + runtime updates) via `storm-trident run rollback` - [Trident ACL Agent Tests](TridentAclAgent-Tests.md) โ€” validates - `trident-acl-agent`'s label-driven update protocol against fake + `trident-acl-agent`'s annotation-driven update protocol against fake Kubernetes API server and Nebraska/Omaha endpoints via `storm-trident run aclagent` diff --git a/tools/storm/aclagent/tests/rollback.go b/tools/storm/aclagent/tests/rollback.go index 7552ecbbb8..730e53e69f 100644 --- a/tools/storm/aclagent/tests/rollback.go +++ b/tools/storm/aclagent/tests/rollback.go @@ -42,7 +42,7 @@ func RunRollback(testConfig stormaclconfig.TestConfig, vmConfig stormvmconfig.Al // rollback request. nodeStore := stormproxies.NewNodeStore(stormproxies.NewSeedNode(testConfig.NodeName, map[string]string{})) apiServer := stormproxies.NewAPIServer(testConfig.NodeName, nodeStore) - if _, err := apiServer.ListenAndServe(ctx, fmt.Sprintf("0.0.0.0:%d", testConfig.APIServerPort)); err != nil { + if _, err := apiServer.ListenAndServe(ctx, fmt.Sprintf("%s:%d", testConfig.HostEndpointIP, testConfig.APIServerPort)); err != nil { return fmt.Errorf("failed to start fake apiserver: %w", err) } diff --git a/tools/storm/utils/vm/qemu/qemu.go b/tools/storm/utils/vm/qemu/qemu.go index 3f757d1ddb..96cdfad8d4 100644 --- a/tools/storm/utils/vm/qemu/qemu.go +++ b/tools/storm/utils/vm/qemu/qemu.go @@ -21,7 +21,7 @@ import ( type QemuConfig struct { SecureBoot bool `help:"Enable secure boot for the VM" default:"false"` SerialLog string `help:"Path to the serial log file" default:"/tmp/trident-vm-verity-test.log"` - ImagePattern string `help:"Regex pattern used to find the base VM image (.qcow2) in the artifacts directory" default:"^trident-vm-.*-testimage.qcow2$"` + ImagePattern string `help:"Regex pattern used to find the base VM image (.qcow2) in the artifacts directory" default:"^trident-vm-.*-testimage\\.qcow2$"` } func (cfg QemuConfig) DeployQemuVM(vmName string, artifactsDir string, outputPath string, verbose bool) error { From 0dbe5a82c6985d86edf6da6986656e93346e0140 Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Fri, 21 Aug 2026 21:37:35 +0000 Subject: [PATCH 07/11] storm/aclagent: fix misattributed doc comment for sha384File sha384File's doc comment ran directly into logScenarioTimeline's with no blank line between them, so godoc associated the whole merged block with logScenarioTimeline and left sha384File undocumented. Split them into separate comment blocks, each immediately above its own function. Verified: gofmt -l (clean), go vet ./storm/... (clean), storm-trident rebuilds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 98412869-e5c8-4f3f-a97c-fe870db70701 --- tools/storm/aclagent/tests/update.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/tools/storm/aclagent/tests/update.go b/tools/storm/aclagent/tests/update.go index 0c3d716c20..cdd1469291 100644 --- a/tools/storm/aclagent/tests/update.go +++ b/tools/storm/aclagent/tests/update.go @@ -22,14 +22,6 @@ import ( "github.com/sirupsen/logrus" ) -// sha384File computes the lowercase hex-encoded SHA-384 digest that tridentd -// expects for a given image path. -// -// For a .cosi file, tridentd does NOT hash the whole archive: a COSI is a -// plain tar with an embedded "metadata.json" entry, and tridentd's Host -// Configuration "sha384" field must match the hash of just that entry's -// bytes (see crates/trident/src/osimage/cosi/mod.rs's read_cosi_metadata). -// For any other file, this hashes the whole file's contents directly. // logScenarioTimeline prints a human-readable, step-by-step trace of a // scenario's progress through the ACL agent's stage/finalize/commit state // machine. It runs regardless of pass/fail so a test run's output always @@ -58,6 +50,14 @@ func logScenarioTimeline(label string, report *stormproxies.ScenarioReport) { logrus.Infof("=== %s state machine timeline: %s ===", label, overall) } +// sha384File computes the lowercase hex-encoded SHA-384 digest that tridentd +// expects for a given image path. +// +// For a .cosi file, tridentd does NOT hash the whole archive: a COSI is a +// plain tar with an embedded "metadata.json" entry, and tridentd's Host +// Configuration "sha384" field must match the hash of just that entry's +// bytes (see crates/trident/src/osimage/cosi/mod.rs's read_cosi_metadata). +// For any other file, this hashes the whole file's contents directly. func sha384File(path string) (string, error) { if strings.HasSuffix(path, ".cosi") { return sha384CosiMetadata(path) From 8a18732d1e5e598be3fde2977891fce35fbb8ae8 Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Tue, 25 Aug 2026 23:41:39 +0000 Subject: [PATCH 08/11] test: add explicit mode/version-path/fallback env vars to acl-agent dropin --- .../base/files/trident-acl-agent-override.conf | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/images/trident-vm-testimage/base/files/trident-acl-agent-override.conf b/tests/images/trident-vm-testimage/base/files/trident-acl-agent-override.conf index 63b3326a4b..8b54668123 100644 --- a/tests/images/trident-vm-testimage/base/files/trident-acl-agent-override.conf +++ b/tests/images/trident-vm-testimage/base/files/trident-acl-agent-override.conf @@ -1,3 +1,6 @@ [Service] Environment=TRIDENT_ACL_AGENT_KUBERNETES_ANNOTATION_PREFIX=acl.azure.com Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY=IMAGE_VERSION +Environment=TRIDENT_ACL_AGENT_ORCHESTRATION_MODE=annotations +Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH=/etc/os-release +Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_FALLBACK=always From 15a2c56a58bbcf94df2548c85dd540ccf6a60d2a Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Wed, 26 Aug 2026 01:32:45 +0000 Subject: [PATCH 09/11] storm/aclagent: fix 2 open Copilot review comments - tests/update.go: use \ systemctl start\ instead of \restart\ for tridentd.service in prepareVmForAclAgent. tridentd is socket-activated and holds no per-test-case state to refresh, so restarting it is unnecessary and, on the post-reboot path, risks killing an in-progress gRPC call (e.g. a commit already underway). - docs/.../TridentAclAgent-Tests.md: fix stale \Go 1.24+\ prerequisite to match tools/go.mod's actual \go 1.25.0\ requirement. --- docs/Development/Testing/TridentAclAgent-Tests.md | 2 +- tools/storm/aclagent/tests/update.go | 11 ++++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/Development/Testing/TridentAclAgent-Tests.md b/docs/Development/Testing/TridentAclAgent-Tests.md index 8341e46b94..9c478d9c08 100644 --- a/docs/Development/Testing/TridentAclAgent-Tests.md +++ b/docs/Development/Testing/TridentAclAgent-Tests.md @@ -65,7 +65,7 @@ instance, not to re-create the kubeconfig. - **Linux host** with root access - **libvirt and QEMU** installed and configured - **Docker** (for building images with Image Customizer) -- **Go 1.24+** (for building Go tools) +- **Go 1.25+** (for building Go tools; matches `tools/go.mod`'s `go 1.25.0`) - **Rust** (latest stable, for building Trident and `trident-acl-agent`) See [Dependencies](../Building/Dependencies.md) for full build dependency diff --git a/tools/storm/aclagent/tests/update.go b/tools/storm/aclagent/tests/update.go index cdd1469291..8248e797b3 100644 --- a/tools/storm/aclagent/tests/update.go +++ b/tools/storm/aclagent/tests/update.go @@ -452,8 +452,17 @@ users: // commands run, so a second restart right after "--now" started it can // kill and restart the agent mid-call, and the new instance's retry // then fails with tridentd's "Servicing is active" error. + // + // tridentd.service is socket-activated (tridentd.socket): it doesn't + // hold any per-test-case state that needs refreshing, so a "restart" + // is unnecessary and, on the post-reboot path, risks killing an + // in-progress gRPC call (e.g. a commit already underway) that a + // trident-acl-agent instance auto-started at boot may be mid-flight + // on. Use "start" instead: it's a no-op if tridentd is already + // running (whether via socket activation or a prior start here), and + // otherwise brings it up without disturbing anything already using it. command := strings.Join([]string{ - "sudo systemctl restart tridentd.service", + "sudo systemctl start tridentd.service", "sudo systemctl enable trident-acl-agent.service", "sudo systemctl restart trident-acl-agent.service", // api_server lives only in the fake kubeconfig now (agent config From 548e0f3bd82806e8918d2366b6905e00c4374a68 Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Thu, 27 Aug 2026 23:52:27 +0000 Subject: [PATCH 10/11] docs: fix broken anchor link in Trident-ACL-Agent.md Docusaurus slugifies headings by dropping '/' rather than converting it to a hyphen, so "## Pre/post-reboot state and the watchdog" generates the anchor #prepost-reboot-state-and-the-watchdog, not #pre-post-reboot-state-and-the-watchdog. Update the manual link to match. --- docs/Explanation/Trident-ACL-Agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Explanation/Trident-ACL-Agent.md b/docs/Explanation/Trident-ACL-Agent.md index b48c2bee72..21e08a60cc 100644 --- a/docs/Explanation/Trident-ACL-Agent.md +++ b/docs/Explanation/Trident-ACL-Agent.md @@ -181,7 +181,7 @@ A status annotation (`/update-status` or (`toVersion` is absent for `rollback`, whose target is implicit). - `startedUtc`/`lastUpdatedUtc`/`finishedUtc` bound the operation: `lastUpdatedUtc` refreshes on a heartbeat cadence while `code` is - `InProgress` (see [below](#pre-post-reboot-state-and-the-watchdog)); + `InProgress` (see [below](#prepost-reboot-state-and-the-watchdog)); `finishedUtc` is absent until `code` reaches a terminal value. `code` is one of: From f74d19dafd0f60ecd7603eef562bdfd80e11a77b Mon Sep 17 00:00:00 2001 From: Brian Fjeldstad Date: Fri, 28 Aug 2026 18:07:52 +0000 Subject: [PATCH 11/11] storm/aclagent: install trident-acl RPM instead of trident-acl-agent Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml | 2 +- tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml b/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml index 87270e8c91..383e7e9857 100644 --- a/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml +++ b/tests/images/trident-vm-testimage/base/baseimg-acl-agent.yaml @@ -164,7 +164,7 @@ os: - openssh-server - systemd-boot - systemd-udev - - trident-acl-agent + - trident-acl - veritysetup - vim - netplan diff --git a/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml b/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml index 3dcd0e250d..6b5aa62266 100644 --- a/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml +++ b/tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml @@ -172,7 +172,7 @@ os: - openssh-server - systemd-boot - systemd-udev - - trident-acl-agent + - trident-acl - veritysetup - vim - netplan