smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26) by james-nesbitt · Pull Request #642 · Mirantis/launchpad

james-nesbitt · 2026-06-08T09:08:00Z

What

Adds TestCuttingEdgeCluster: rhel10, rocky10 and ubuntu26 as both managers and workers, on MCR stable-29.8.0 and MKE 3.9.7, replacing the single-node version of that test currently on main.

Why

No smoke job covers RHEL 10, Rocky 10 or Ubuntu 26.04 in worker roles. MCR packages for them now exist on the public repo, so the matrix can run without an internal-repo detour.

MCR package availability on repos.mirantis.com, checked 2026-11-03:

Platform	Channels
`rhel/10` (also serves rocky10)	`stable-29.2` … `stable-29.8.0`
`ubuntu` `resolute`	`stable-29.8.0`, `test-29.8.0` only
`sles/16`	absent — `sles/` has `12`, `12.3`, `15`

How

WrkSles16 dropped, and the Sles16 entry removed from test/platforms.go (its only consumer). A sles16 worker cannot install MCR, so it could only ever fail the run
MCRChannel stable-29.6.1 → stable-29.8.0: 29.6.1 does not exist for resolute
MKEVersion 3.9.6 → 3.9.7, the pairing with MCR 29.8.0 exercised live in PRODENG-4062
main's TestCuttingEdgeCluster (from [PRODENG-3593] Add Ubuntu 26.04 (Resolute) host configurer #661) is removed. It ran one ubuntu_26.04 manager against repos-internal.mirantis.com test-29 because resolute packages existed nowhere else on 2026-09-29; the public repo now carries them. Cost: no smoke test exercises the repos-internal path any more. MCRRepoURL stays in smokeConfig for the next test that needs it
Module constraint is the exact 0.1.8 pin from main, not this branch's >= 0.1.7: 0.2.0–0.2.2 fail at plan time ([PRODENG-4062] Pin provision-aws terraform module to 0.1.8 #679)

Testing

go vet ./test/... and gofmt -l test/: clean
Not run: the smoke job itself

Known risk

The RHEL10/Rocky10 MCR-daemon-start gap — missing xt_* kernel modules break iptables, and MCR's nftables backend is incompatible with swarm mode — is unresolved and may still fail this matrix.

Links

Checklist

Tests added or updated
Docs updated if user-visible behaviour changed (N/A)
No debug output or dead code left in

Written by AI: claude-opus-5

Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version == "26.04", matching the existing per-codename pattern (bionic, focal, jammy, noble, xenial). No version-specific behavior needed: the shared ubuntu.Configurer.InstallMCR already builds the APT repo URL generically from VERSION_CODENAME. MCR has not published Docker EE packages for Ubuntu 26.04 on either repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail on this platform until MCR ships support - this configurer only makes launchpad recognize the OS, it does not unblock installs. Same gap already tracked by TestCuttingEdgeCluster in PR #642. Verified: go build ./..., go vet, go test ./pkg/product/mke/config/... ./pkg/configurer/..., golangci-lint run (no new issues).

EnableMCR's error on a failed "systemctl start docker" was always a bare "exit status 1" with no way to tell why: rig's StartService issues "systemctl start docker 2> /dev/null", discarding systemd's own diagnostic output. Reproduced live on RHEL10/Rocky10 in TestCuttingEdgeCluster (PR #642) with the latest available MCR (stable-29.6.1) and MKE (3.9.6): package install succeeds, but docker.service still fails to start, with no way from launchpad's own logs to see why -- exactly the gap this PR's description flags as "root cause not yet isolated (stderr suppressed by the install command)". Best-effort fetch journalctl output for the docker unit on failure and include it in the returned error, so the actual systemd failure reason is visible without needing rig changes or manual SSH access. Ignore a failure to fetch the diagnostic itself, since that must not mask the original error. Written by AI: claude-sonnet-5 Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

james-nesbitt · 2026-09-17T15:41:03Z

Current status: re-verified today with the latest available MCR (stable-29.6.1) and MKE (3.9.6). Still not passing, but root causes are now fully isolated for every platform in this test.

RHEL 10 / Rocky 10: package install succeeds; docker.service still fails to start. Added a diagnostic capture (journalctl -xeu docker) to pkg/configurer/linux.go, since rig's StartService was discarding systemd's own stderr. Isolated cause: RHEL 10/Rocky 10's iptables-nft backend does not support the addrtype match Docker's bridge driver requires ("Extension addrtype revision 0 not supported"). Matches MCR 29.6.1's own release notes, which list Swarm/nftables parity as still "planned for upcoming releases." Not a launchpad bug.
Ubuntu 26.04: was aborting the whole test at OS detection (launchpad didn't recognize the OS at all). Fixed by cherry-picking the configurer from [PRODENG-3593] Add Ubuntu 26.04 (Resolute) host configurer #661. Now correctly reaches Install MCR, where it fails because MCR has not published Ubuntu 26.04 packages yet (unchanged since July).
SLES 16: still no packages published at all (unchanged since July).

Confirmed identically across 2 local runs and 1 official CI run (35238411403) with no discrepancies. No orphaned AWS resources left behind.

Written by AI: claude-sonnet-5

Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version == "26.04", matching the existing per-codename pattern (bionic, focal, jammy, noble, xenial). No version-specific behavior needed: the shared ubuntu.Configurer.InstallMCR already builds the APT repo URL generically from VERSION_CODENAME. MCR has not published Docker EE packages for Ubuntu 26.04 on either repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail on this platform until MCR ships support - this configurer only makes launchpad recognize the OS, it does not unblock installs. Same gap already tracked by TestCuttingEdgeCluster in PR #642. Verified: go build ./..., go vet, go test ./pkg/product/mke/config/... ./pkg/configurer/..., golangci-lint run (no new issues).

EnableMCR's error on a failed "systemctl start docker" was always a bare "exit status 1" with no way to tell why: rig's StartService issues "systemctl start docker 2> /dev/null", discarding systemd's own diagnostic output. Reproduced live on RHEL10/Rocky10 in TestCuttingEdgeCluster (PR #642) with the latest available MCR (stable-29.6.1) and MKE (3.9.6): package install succeeds, but docker.service still fails to start, with no way from launchpad's own logs to see why -- exactly the gap this PR's description flags as "root cause not yet isolated (stderr suppressed by the install command)". Best-effort fetch journalctl output for the docker unit on failure and include it in the returned error, so the actual systemd failure reason is visible without needing rig changes or manual SSH access. Ignore a failure to fetch the diagnostic itself, since that must not mask the original error. Written by AI: claude-sonnet-5 Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version == "26.04", matching the existing per-codename pattern (bionic, focal, jammy, noble, xenial). No version-specific behavior needed: the shared ubuntu.Configurer.InstallMCR already builds the APT repo URL generically from VERSION_CODENAME. MCR has not published Docker EE packages for Ubuntu 26.04 on either repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail on this platform until MCR ships support - this configurer only makes launchpad recognize the OS, it does not unblock installs. Same gap already tracked by TestCuttingEdgeCluster in PR #642. Verified: go build ./..., go vet, go test ./pkg/product/mke/config/... ./pkg/configurer/..., golangci-lint run (no new issues).

* [PRODENG-3593] Add Ubuntu 26.04 (Resolute) host configurer Registers ResoluteConfigurer for os.ID == "ubuntu" && os.Version == "26.04", matching the existing per-codename pattern (bionic, focal, jammy, noble, xenial). No version-specific behavior needed: the shared ubuntu.Configurer.InstallMCR already builds the APT repo URL generically from VERSION_CODENAME. MCR has not published Docker EE packages for Ubuntu 26.04 on either repos.mirantis.com or repos-internal.mirantis.com as of 2026-09-02 (both list only trusty/xenial/bionic/focal/jammy/noble). InstallMCR will fail on this platform until MCR ships support - this configurer only makes launchpad recognize the OS, it does not unblock installs. Same gap already tracked by TestCuttingEdgeCluster in PR #642. Verified: go build ./..., go vet, go test ./pkg/product/mke/config/... ./pkg/configurer/..., golangci-lint run (no new issues). * Fix resolute.go registration for rig v2 migration PRODENG-3594 (#651) migrated OS module registration from k0sproject/rig + rig/os/registry (func(rig.OSVersion) bool) to pkg/configurer.RegisterOSModule + rigos.Release (func(*rigos.Release) bool), landed on main after this branch was created. Update ResoluteConfigurer to match the current pattern used by NobleConfigurer; without this the branch fails to compile against main. Written by AI: claude-sonnet-5 Signed-off-by: James Nesbitt <jnesbitt@mirantis.com> * Add TestCuttingEdgeCluster smoke test for Ubuntu 26.04 - test/platforms.go: add Ubuntu26 platform entry (ubuntu_26.04) - test/smoke/smoke_test.go: add MCRRepoURL to smokeConfig, wired into the launchpad tfvars map as mcr_repo_url when non-empty (defaults to the public repo otherwise, via PR #669's optional tfvar); add TestCuttingEdgeCluster exercising a single ubuntu_26.04 manager against repos-internal.mirantis.com channel test-29, the only place Ubuntu 26.04 MCR packages exist as of 2026-09-29 Verified: TestCuttingEdgeCluster passed against real AWS infra (docker-testing-533267045383, us-east-1), MCR engine 29.8.0m1-rc1 installed and detected by MKE 3.9.2, cluster reported healthy, Reset succeeded, all 28 AWS resources destroyed cleanly. Written by AI: claude-sonnet-5 Signed-off-by: James Nesbitt <jnesbitt@mirantis.com> --------- Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

Adds a new smoke test targeting the latest gen-10 distributions: - RHEL 10, Rocky 10, Ubuntu 24.04 as managers and workers - SLES 15 as an additional worker Also adds: - Rhel10 and Rocky10 platform definitions in test/platforms.go - smoke-cutting-edge Makefile target (50m timeout) - smoke-cutting-edge CI job triggered by the smoke-cutting-edge label NOTE: This test depends on terraform-mirantis-modules/terraform-mirantis-provision-aws PR #22 being merged (adds rhel_10 and rocky_10 platform keys to the provision-aws module). The test will fail until the module is updated and the version bumped in examples/terraform/aws-simple/. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

- shorten Name "cutting-edge" -> "cuttingedge" (11 chars): the 12-char value overflowed the 32-char AWS LB/target-group name limit and was the actual CI failure ("name" cannot be longer than 32 characters) - pin examples/terraform/aws-simple to provision-aws >= 0.1.7, which ships the rhel_10/rocky_10 platform keys (provision-aws#22, released) - refresh test doc comment now the upstream dependency is merged/released gofmt also normalized two pre-existing violations in smoke_test.go.

rhel10/rocky10 AMIs and provision-aws platform keys (v0.1.7) are already available, but launchpad's own smoke coverage previously failed with 'package manager could not install docker-ee' because repos.mirantis.com had not yet published docker-ee for rhel/10 or rocky/10. Checked repos.mirantis.com directly: docker-ee is now present in both rhel/10/x86_64/stable-29.4 and rocky/10/x86_64/stable-29.4 -- that gap looks closed, matching the AMI-availability timing pattern seen elsewhere this cycle (Oracle Linux, Rocky 8). Strategy going forward: keep TestCuttingEdgeCluster merged and label-gated rather than blocking on it going green immediately, so it picks up MCR support automatically as soon as it lands, with no further code changes. Documented that intent on the Makefile target and the GitHub Actions job. Written by AI: claude-sonnet-5

Adds a Sles16 platform definition (sles_16) and switches TestCuttingEdgeCluster's additional worker from sles15 to sles16. NOTE: MCR has not published Docker EE packages for SLES 16 as of 2026-07-22 (repos.mirantis.com/sles/ only has 12/12.3/15). This node is expected to fail at Install MCR until MCR ships SLES16 support - same category of gap as RHEL10/Rocky10 (docker-ee installs but the daemon won't start), tracked separately for that pair. Flagged in a doc comment on the test so CI failures here are understood, not mysterious. gofmt also cleaned two pre-existing, unrelated violations in these files (trailing blank line in platforms.go; WrkWin2025 alignment in TestFIPSCluster).

Adds a Ubuntu26 platform definition (ubuntu_26.04, codename resolute) and switches TestCuttingEdgeCluster's Ubuntu manager+worker from 24.04 to 26.04 - the current LTS as of 2026-07 (24.04 remains fully supported, just no longer the newest release). The ubuntu_26.04 key is already present in the pinned provision-aws module (>= v0.1.7), AMI pattern confirmed against a live published AMI (ubuntu-resolute-26.04-amd64-server-*, published as recently as 2026-07-22). NOTE: MCR has not published Docker EE packages for Ubuntu 26.04 yet (repos.mirantis.com/ubuntu/dists/ only has trusty/xenial/bionic/focal/ jammy/noble - no resolute). This node is expected to fail at Install MCR alongside the existing SLES16 and RHEL10/Rocky10 gaps, until MCR ships support. Flagged in the test's doc comment.

Re-checking whether the RHEL10/Rocky10 MCR-daemon-start gap and the Ubuntu 26.04/SLES 16 missing-package gaps (documented in this PR's description) have closed with the newest available releases. Was stable-29.4/MKE 3.9.2 (docker-ee-29.4.1m1); repos.mirantis.com now also publishes stable-29.6.1 (docker-ee-29.6.1m2) for rhel/10. MSRVersion unchanged (3.1.18), already current elsewhere. Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

EnableMCR's error on a failed "systemctl start docker" was always a bare "exit status 1" with no way to tell why: rig's StartService issues "systemctl start docker 2> /dev/null", discarding systemd's own diagnostic output. Reproduced live on RHEL10/Rocky10 in TestCuttingEdgeCluster (PR #642) with the latest available MCR (stable-29.6.1) and MKE (3.9.6): package install succeeds, but docker.service still fails to start, with no way from launchpad's own logs to see why -- exactly the gap this PR's description flags as "root cause not yet isolated (stderr suppressed by the install command)". Best-effort fetch journalctl output for the docker unit on failure and include it in the returned error, so the actual systemd failure reason is visible without needing rig changes or manual SSH access. Ignore a failure to fetch the diagnostic itself, since that must not mask the original error. Written by AI: claude-sonnet-5 Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

…gle-node test MCR package availability on repos.mirantis.com, checked 2026-11-03: rhel/10 (also serves rocky10) stable-29.2 ... stable-29.8.0 ubuntu resolute stable-29.8.0 and test-29.8.0 only sles/16 no directory; sles/ has 12, 12.3, 15 Changes following from that: - WrkSles16 removed from TestCuttingEdgeCluster and the Sles16 entry removed from test/platforms.go, its only consumer. A sles16 worker cannot install MCR at all, so it can only ever fail the run. - MCRChannel stable-29.6.1 -> stable-29.8.0. 29.6.1 does not exist for resolute, so the previous value could not install on ubuntu26. - MKEVersion 3.9.6 -> 3.9.7, the pairing with MCR 29.8.0 exercised live in PRODENG-4062. main's TestCuttingEdgeCluster (added by #661) is removed and replaced by this matrix. That test provisioned one ubuntu_26.04 manager against repos-internal.mirantis.com test-29 because resolute packages existed nowhere else on 2026-09-29. repos.mirantis.com now carries them, so the public-repo matrix covers ubuntu26 in both manager and worker roles. The cost is that no smoke test exercises the repos-internal path any more; MCRRepoURL in smokeConfig is retained for the next test that needs it. The provision-aws module constraint is the exact pin 0.1.8 from main rather than this branch's >= 0.1.7: 0.2.0-0.2.2 fail at plan time. Not run: the smoke job itself. The RHEL10/Rocky10 MCR-daemon-start gap (missing xt_* kernel modules, nftables vs swarm) is unresolved and may still fail this matrix. Written by AI: claude-opus-5 Signed-off-by: James Nesbitt <jnesbitt@mirantis.com>

james-nesbitt added the smoke-cutting-edge Trigger cutting-edge smoke test label Jun 8, 2026

james-nesbitt force-pushed the smoke/cutting-edge-test branch from 093c76b to d1a9f2d Compare July 22, 2026 07:24

james-nesbitt mentioned this pull request Sep 2, 2026

[PRODENG-3593] Add Ubuntu 26.04 (Resolute) host configurer #661

Merged

2 of 3 tasks

james-nesbitt force-pushed the smoke/cutting-edge-test branch from 33a8927 to eac251c Compare September 17, 2026 15:01

james-nesbitt changed the title ~~smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu24/sles15)~~ smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26/sles16) Sep 17, 2026

james-nesbitt force-pushed the smoke/cutting-edge-test branch from 310d91c to a9592df Compare September 20, 2026 06:59

james-nesbitt marked this pull request as draft September 21, 2026 07:25

james-nesbitt and others added 8 commits November 3, 2026 08:29

james-nesbitt force-pushed the smoke/cutting-edge-test branch from a9592df to 514d679 Compare November 3, 2026 06:44

james-nesbitt changed the title ~~smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26/sles16)~~ smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26) Nov 3, 2026

james-nesbitt marked this pull request as ready for review November 5, 2026 12:44

james-nesbitt added 2 commits November 6, 2026 08:52

Merge remote-tracking branch 'origin/main' into smoke/cutting-edge-test

e55ef89

Merge remote-tracking branch 'origin/main' into smoke/cutting-edge-test

7ae3bf9

Navigation Menu

Repository navigation

smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26) - #642

smoke: add TestCuttingEdgeCluster (rhel10/rocky10/ubuntu26)#642
james-nesbitt wants to merge 10 commits into
mainfrom
smoke/cutting-edge-test

james-nesbitt commented Jun 8, 2026 •

edited

Loading

Uh oh!

james-nesbitt commented Sep 17, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

james-nesbitt commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Testing

Known risk

Links

Checklist

Uh oh!

james-nesbitt commented Sep 17, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

james-nesbitt commented Jun 8, 2026 •

edited

Loading