Integration test framework: Talos/QEMU cluster + nvmet fabric - #439
Draft
noctarius wants to merge 25 commits into
Draft
Integration test framework: Talos/QEMU cluster + nvmet fabric#439noctarius wants to merge 25 commits into
noctarius wants to merge 25 commits into
Conversation
Generalises three separate stale-controller fixes — f99b5b5, 498c2c5 and the device-discovery rework in #399 — into one diagnosis/repair layer, and rewires the CSI node plugin onto it. All three were chasing the same shape of bug: a connect succeeds at one layer of the NVMe object tree while the layer below it is unusable, and the check that gates the retry sits at the higher layer, so nothing looks missing and the retry spins. A subsystem attached with live controllers that exports no namespace at all, so no block device ever appears and kubelet retries NodeStageVolume forever. A controller that is live and contributes no path to the namespace, so the volume runs a path short while every connect answers "already connected" — in one 42-hour run that left volumes routinely below their configured redundancy with no operator-visible signal. Inspect diagnoses, read-only, and names each defect positively from state the kernel already publishes rather than inferring it from a wait that ran out or from the text of an nvme-cli error. A timeout cannot tell "slow" from "broken"; "live controller, zero namespaces" is decidable on the first look. Each Defect carries the teardown scope, the controllers to release in teardown order, and the co-tenant namespaces a repair would leave with no usable path. Repair acts on one defect; Repairer.Attach adds the policy — narrowest scope first, and never at the cost of another volume's block device, the caller's own device, or a repair that just ran. The cooldown key is what makes the last of those work: it must be narrow enough that two outstanding repairs do not collide and stable enough to survive a repair that did not stick, and neither the controller id nor the kernel subsystem id qualifies in general because a repair re-creates both. The CSI driver diagnoses through Inspect and tears down through Repair over the nvme-cli connector, so detection, teardown ordering and the disconnect itself are shared; what stays local is policy alone. Reading sysfs instead of shelling out also removes a process per monitor tick and the reverse-engineering of udev's link naming. testdata/sysfs replays kernel state captured from a live cluster: a healthy node with four subsystems and eight namespaces, and the three defects that can be forced with nvmet. The healthy one is the false-positive guard, and it is the assertion that matters most — a repair tears down live data paths, so a diagnosis that fires on a healthy fabric is worse than no diagnosis, and hand-built fixtures cannot rule that out because they encode the belief under test. Two things are deliberately not covered. The auto-repair path has no automated end-to-end test: forcing a non-contributing controller at a *published* endpoint needs a storage node to stop exporting a namespace on one of its published listeners, which the real control plane will not do on request. And defects are logged but not otherwise surfaced; counters would fix that and would also be the natural assertion surface for an e2e suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
noctarius
force-pushed
the
integration-test-framework
branch
from
August 18, 2026 05:41
95492b9 to
37b577b
Compare
First piece of an integration tier that runs the CSI driver against a real kernel, so the fabric states that only exist below Kubernetes can be produced deliberately rather than waited for. Nodes have to host an NVMe-oF target, not just attach to one. That rules out containers — kind and k3d share the host kernel, so two "nodes" cannot host two targets for one NQN — and it makes the choice of guest OS the whole design. Talos ships nvmet and nvmet_tcp as modules on both amd64 and arm64, declared in the machine config and loaded at boot, where Ubuntu and Debian cloud images keep them in linux-modules-extra and need an apt install per node. Talos also builds its kernel with CONFIG_NVME_TARGET_AUTH, which is what would make target-side DHCHAP reachable. talosctl's QEMU provisioner is used rather than driving QEMU directly. It gives nodes real addresses on a cluster network — QEMU's user-mode networking puts every guest on one address with no route between them, which is exactly what a same-NQN target pair cannot live with — and it runs on macOS/arm64 as well as Linux, so the developer loop and CI take the same path. A single node is the default, at 2 GiB with scheduling allowed on the control plane. Only a same-NQN target pair needs a second kernel, so those specs ask for a worker and everything else does not pay for one. The CI workflow is separate from the unit-test ones because it needs hardware virtualisation. /dev/kvm exists on ubuntu-latest but the runner user is not in the kvm group, and a group added in a step does not apply to that step's shell, so anything that spawns QEMU runs under `sg kvm -c`. Without that the fallback is emulation, where a boot takes minutes and the suite times out looking like a hang. talosctl is pinned: a cluster failure should not turn out to be version skew. Not yet written: the spec-generated control-plane emulation, the nvmet fabric setup (a privileged pod, since Talos has no shell), and the suites themselves. No cluster has been booted with this yet, so two assumptions are still unverified — that the machine-config patch loads the modules, and that a privileged pod can mount and write nvmet configfs on Talos. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cluster harness rested on two unverified assumptions: that the machine-config patch actually loads the NVMe-oF target modules, and that a privileged pod can reach the node's configfs. Everything downstream depends on both, and a failure in either would have surfaced somewhere far less legible. This adds the access needed to check them and the test that does. Talos has no shell and no SSH, so node access is a privileged pod pinned with nodeName, carrying hostPID and the host's /sys, /dev and /lib/modules. Run executes in the pod, which is enough to read /sys; RunOnHost goes through nsenter into PID 1 so that modprobe and mount affect the node rather than the container. That is the same access the CSI driver has, which makes it a fair way to reach a node rather than a way around Talos. Kubernetes access is kubectl, not client-go: the harness needs apply, exec and a few waits, and client-go becomes the right answer once it needs watches. The choice is confined to one file. The smoke test involves no CSI driver and no control plane, so a first failure points at one thing. It reads /proc/modules rather than modprobing, which proves the patch took effect instead of merely proving the modules exist; it mounts configfs, since Talos does not; and it checks attr_cntlid_min, which is the gate on everything else — without it the host rejects a second controller for one NQN as a duplicate cntlid, and a controller that contributes no path cannot be produced at all. The suite skips unless SB_INTEGRATION is set, so `go test ./...` in a repository check does not boot virtual machines. The workflow sets it; without that the job would have reported green having booted nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The QEMU provisioner needs root — it builds a CNI bridge and opens the accelerator — and says so only after downloading the image and generating PKI, which is why a locally aborted probe never reached the check and why the claim that it ran unprivileged was wrong. Only talosctl is elevated. Running the test itself as root would leave the Go build cache and every artifact root-owned for whatever runs next, so the harness prefixes sudo -E on its own when it is not already root, and hands the work directory back afterwards: talosctl writes the kubeconfig as root and both kubectl and the test read it unprivileged. -E matters more than it looks: talosctl finds its image cache through HOME, and losing that re-downloads the disk image on every run. The workflow drops the sg kvm wrapper, which is moot once the QEMU processes are root's children, and keeps only the check that /dev/kvm exists — the fast signal that the runner cannot accelerate at all. It also reclaims ~/.talos before the cache action tries to save a tree root has been writing into. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cluster creation now succeeds, and the next step failed: talosctl kubeconfig is a separate command that needs --nodes, and cluster create leaves endpoints in the talosconfig without setting nodes. The address has to be read back out. `config info --output json` supplies it. Both nodes and endpoints are tried, because which of the two a freshly created cluster populates is exactly what the failure showed is not obvious, and an unrecognised shape reports the raw output rather than failing as an empty list — a format change should say what it changed to. The addresses are kept on the cluster: they are also what an nvmet target on a node advertises as its traddr, so the fabric work will want them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems, one found by CI and one found by looking again at the code it was about to reach. Talos enforces the baseline pod-security profile by default, which allows none of what this pod needs: host namespaces, hostPath volumes, a privileged container. The namespace now carries the privileged labels, which is what the CSI e2e suite already does for the same reason. The second problem would have been the next failure. RunOnHost used nsenter --mount to enter the node's mount namespace and then ran /bin/sh there — and Talos has no /bin/sh, so nothing on the other side of that nsenter could have executed. It was also unnecessary: loading a module is not namespaced, so modprobe from a privileged container carrying the node's /lib/modules affects the node's kernel, and nvmet's configfs objects are kernel-global, so configfs mounted inside the pod exposes the node's nvmet tree rather than a private copy. So the pod stays in its own namespaces and mounts configfs at /configfs — a path of its own rather than under the /sys bind mount, which avoids propagation questions this does not need to answer. EnsureConfigFS reports the module being absent as such, since that is the likely reason for nvmet not appearing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cluster create merges the kubeconfig it generates through the standard loading rules, and unlike the talosconfig there is no flag to redirect it. Left alone it rewrites ~/.kube/config, makes the throwaway cluster the current context, and — since destroy removes the VMs but not the kubeconfig entries — leaves that context behind pointing at nothing. A test suite must not do that to the machine it runs on. KUBECONFIG now points at the cluster's own file. It is set through `env` rather than only inherited because sudo resets the environment: -E asks for it back, but whether that is honoured depends on the local sudoers policy, and `env` does not. -E is still needed for HOME, which is how talosctl finds its image cache. Found by running the provisioner by hand on macOS, where it reported the merge in its output. The suite had never run locally, so nothing had exercised this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`go test` alone has two sharp edges here. The harness elevates talosctl rather than the test process, so the sudo prompt appears mid-run, where go test has captured stdout and it is invisible — the run just appears to hang. And the prerequisites are not obvious: talosctl, qemu-img, kubectl, plus /dev/kvm on Linux or an arm mac, since Talos' QEMU provisioner supports only arm on macOS. preflight checks all of that and refreshes the sudo credential cache before the test starts, so the prompt is asked for where it can be seen. It runs as a prerequisite of test, so there is nothing to remember. clean-clusters exists because an interrupted run leaves the cluster behind holding memory and a network, and the name is not something to have to look up. The root Makefile delegates as it does for the other components, with one difference: the aggregate `test` target does not include integration-test, which boots virtual machines and takes minutes. `lint` and `vet` do include it, since those are cheap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the suite twice on one machine produced a failure that read as a timeout on a missing bridge100, and was neither. The first run was still going, holding the vmnet network; the second built its own on top and failed describing an interface it did not own. Three separate mistakes made that possible. Cluster state was going to a per-run temp directory, which is why the failure was unrecoverable: `talosctl cluster destroy --name` cannot find a cluster whose state is somewhere it does not look, so `make integration-clean` was useless exactly when it was needed. State now stays in talosctl's default location. Polluting ~/.talos/clusters is harmless and destroy cleans it; root-owned processes nobody can reach are not harmless. Cluster names are now derived per process, so two runs cannot pick the same name and fight over the same network. SB_CLUSTER_NAME still pins it when a run has to be identifiable from outside. Create now clears a same-named cluster first. Building over the top of a leftover fails confusingly, because the network exists but belongs to nobody. clean-clusters no longer needs to be told a name: it destroys every leftover sb-integration* cluster and kills the helper processes an interrupted run reparents to init — sudo's children outlive the sudo the test killed, which is how a loadbalancer-launch survives holding port 6443. The CI teardown does the same, since there is no single name to destroy any more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The machine-config patch listed nvme_tcp and nvme_fabrics alongside the target modules. Talos builds NVME_TARGET and NVME_TARGET_TCP as modules but compiles NVME_TCP, NVME_FABRICS and NVME_CORE in, so two of the four entries told Talos to find a .ko that was never built. The patch now names only the two real modules, with a note not to add the others back — they are already there, and their absence from /proc/modules is expected rather than a problem. CI passed with the wrong patch, which is worth noting: the same four entries on amd64 did not stop a cluster coming up, so this was latent there and only bit on the first local run. Whether it is the whole reason bootstrap timed out is not yet proven — the patch is wrong either way. Create now collects `cluster show` and the nodes' dmesg tail before destroying, because destroy removes the only evidence of why create failed, and a bootstrap timeout says nothing about its own cause. Best effort throughout: it runs where something has already gone wrong and must not turn one failure into two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Go caches a passing package result and replays its whole transcript on a later run: the RUN lines, the PASS lines, the timings, everything but the word "(cached)". For a suite whose entire purpose is to boot a real kernel, that means an invocation which started nothing can print exactly the output of one that booted a cluster, ran three assertions against it and tore it down. That is worse than no result, and it showed up immediately — a local run reported a full green transcript for a cluster it had not created, which is not a thing anyone should have to notice. -count=1 in both the Makefile and the workflow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
talosctl cluster create writes controlplane.yaml and worker.yaml into its working directory, which for `go test` is the package source directory. Those files carry the cluster's PKI, bootstrap token and secretbox key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
…gnosis The detection this exercises was verified once, by hand, on a live cluster with two nvmet targets — and the snapshots that verification produced are what nvmeof's unit tests replay. Nothing rebuilt the fabric itself, so a change to how the host merges controllers, or to nvmet's cntlid handling, would go unnoticed until the fixtures were stale. The spec stands up the topology that produces the defect: two targets sharing an NQN, model and serial across two nodes, one exporting namespace 1 and one exporting nothing. The host merges them into a single subsystem with two live controllers, only one of which serves a path — a controller that looks connected from every angle a connect checks while contributing neither redundancy nor I/O. Two nodes rather than two targets on one: nvmet names a subsystem's configfs directory after its NQN, so one kernel hosts one subsystem per NQN. Disjoint cntlid ranges are the other half — two targets on nvmet's default range both hand out cntlid 1 and the host rejects the second controller, collapsing the topology before the test starts. The assertion that matters is not that a defect appears but which controller it names. Naming the serving one would be an actively harmful diagnosis: the repair tears down a controller, and that one carries the only path. atlas's own sysfs resolver runs against the state, which the test process cannot read directly — the kernel is the node's and the host may not even be Linux. So the node's NVMe sysfs is copied out as text and rebuilt locally, in the format hack/nvmet/capture-sysfs.sh already produces, which makes this suite the source of the fixtures the unit tests replay rather than a second way of writing them. That bridge is checked without a cluster: fabric's own test replays the committed controller-not-contributing snapshot through ReconstructSysfs and Inspect, so a failure in the suite says whether the fabric was wrong or the harness was. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
noctarius
force-pushed
the
integration-test-framework
branch
from
August 18, 2026 06:51
37b577b to
ffad47b
Compare
… leaves A two-node spec failed to create on macOS with "interface bridge100 not found" and a one-minute timeout, while the single-node smoke test in the same run came up fine. The difference between them was four characters of cluster name. talosctl builds the QEMU monitor socket path as <state>/<cluster>/<cluster>-controlplane-N.monitor — the name appears twice — and sockaddr_un.sun_path holds 104 bytes on macOS. The longer name produced 105, so QEMU refused to start, so the node that was supposed to bring up the bridge never ran, so the dhcpd step waited out its full minute for a bridge nothing would create. Every layer reported the symptom it saw and none of them mentioned a path: the QEMU error was in a per-node log file inside the state directory. So the name is checked before talosctl runs, against the state path it will actually use, and the names the suite generates are short. The check is a one-line arithmetic guard whose value is entirely in its error message. The same failure also exposed a state directory that nothing collects. talosctl removes it on destroy, but only when it could read state.yaml, and a create that dies this early never writes one — so destroy fails on the missing file, leaves 2.3 GB of sparse disk images behind, and fails identically forever after. Destroy now treats that as "nothing to destroy" and removes the directory itself, and clean-clusters does the same for those already stranded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
The shell pod installed nvme-cli and util-linux at start. That made every command in the package depend on a package mirror, and the failure mode was silent: when the download had not finished, BusyBox's losetup answered instead of util-linux's and rejected -j and --show, on one node but not the other. Nothing is installed now. The target uses BusyBox's losetup syntax — list, next free device, two-argument associate — and the initiator writes a connect options line to /dev/nvme-fabrics and drives delete_controller and rescan_controller through sysfs. That is also the mechanism atlas's FabricsConnector uses, so the fabric these tests build now comes up the way the code under test brings one up, and the diagnostic dump reads sysfs directly rather than asking the resolver whose output is the thing under suspicion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
The two-node spec passed on arm64 and failed on amd64: Talos checks usable memory against 874 MiB, and the 1 GiB allocated to the worker left 853 MiB of it. How much firmware and the kernel take is architecture-dependent, so the margin has to cover the worst case rather than the host in front of you. Both node types now get the 2 GiB talosctl gives them by default. The check does not fail the create, it retries — so what the test saw was the 20-minute create timeout, reported as "signal: killed" because CommandContext kills on deadline. A run that times out now says so. Two things that should have made the cause obvious were also wrong. `cluster show` was called without --provisioner, so it described a docker cluster and printed empty fields, and the workflow's diagnostic and cleanup steps still globbed the old sb-integration* names. The per-node QEMU logs — the only place a VM that never started explains itself — are now collected on failure, and a state directory that survives destroy is removed rather than cached. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
The integration suite can build a real NVMe-oF fabric but has nothing to tell a
client where that fabric is. This answers the endpoints the volume-attach path
reads, so a volume's connect entries can be made to point at the nvmet targets
the harness stood up.
Generated from shared/openapi.json — models and the server interface both. The
interface is the part that matters: routing through it means the spec decides
which paths exist, so an endpoint added upstream shows up as a new stub in a
committed file, and one that is renamed or removed makes generation fail rather
than leaving a handler serving no route. A hand-written mux would have answered
the endpoints it knew about and 404'd the rest, which is indistinguishable from a
missing volume. Unimplemented endpoints answer 501 for the same reason: "the
simulator does not do this" and "the volume is not there" are different answers.
Two types have to be overridden. FastAPI declares no response model for /connect,
so the spec says `{}` and generates nothing for the one response the simulator
exists to produce; the overlay declares it as the NvmeConnectEntry array it is.
The other is the volume-scoped migrations-list operation, whose declared path
parameters do not match its URL — the same fixup atlas-lib's client carries, and
one that belongs upstream.
The entries themselves follow sbcli: one per data NIC per node in node order,
tunables from simplyblock_core/constants.py, and DHCHAP secrets only inside the
prebuilt connect command, which is where the real control plane puts them and
where a client has to parse them back out.
The tests drive it through atlas-lib's own client, which strict-decodes every
response against its validation rules — so a field named wrong here fails rather
than decoding as a zero value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
shared/openapi.json is the contract atlas-lib's client, the integration suite's control-plane simulator and the operator's spec-backed mock are generated or checked against, and it is exported from the control plane — so it goes stale silently. Nothing here fails when sbcli grows an endpoint, and the committed generated code keeps compiling against endpoints that no longer exist. The export is the one simplyblock-documentation already uses: import the FastAPI app, ask it for its spec, keep the /api/v2 paths. It needs sbcli's requirements installed but no database and no FoundationDB client library, since nothing connects at import time. It lives in a script rather than in the workflow so the same command reproduces the file locally. The job regenerates what the spec feeds in the same change, because a pull request carrying only the spec would leave the committed generated code describing the previous revision. That step is allowed to fail: the interesting spec changes are exactly the ones that break generation, and oapi-codegen applies its overlays in strict mode, so an endpoint that moved makes an overlay patching its old path a hard error. When that happens the pull request still opens with the spec alone and says what to fix, because a visible broken change beats an invisible stale one. The body leads with what a reviewer needs from a large mechanical JSON diff: endpoints added, endpoints removed, and endpoints changed in place — that last group being the ones a generated client compiles against without noticing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
Reusing one branch means the action updates the open pull request rather than opening another, but it rewrites the title and body on every run where the branch differs from the base — unconditionally, not only when the branch was pushed. The body names the sbcli commit the spec was exported from, so that alone would have rewritten the pull request and bumped its timestamp every night while nothing about the spec had changed. So the export is now compared against what is already on the sync branch, not only against the base, and a match stops the job there: the change has been proposed and is waiting for a review, which is not news. An export that differs from the branch still goes through, and a branch closed without merging keeps this quiet about that exact change while still reporting the next one. The pull-request step itself is no longer gated on the spec having changed, which lets the action do its own cleanup: when an open pull request no longer differs from the base — the spec having landed some other way, or been reverted upstream — it closes it and deletes the branch instead of leaving a proposal open against a diff that does not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
noctarius
force-pushed
the
integration-test-framework
branch
from
August 18, 2026 19:30
dc40909 to
e7ee4fe
Compare
The body tells a reviewer which workflow produced the change, and the rename to repo_openapi_sync.yaml left it naming a file that no longer exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
The comment claimed the kernel rejects a second controller whose model or serial disagrees with the first. It does not. nvme_init_subsystem looks a subsystem up by subsysnqn alone, and copies the model and serial off whichever controller arrived first without comparing them again. What nvme_validate_cntlid rejects is a duplicate cntlid, or a controller that does not advertise multi-controller support. Model and Serial are still set, because the detection reads them and a real cluster's pair always agrees, but the disjoint cntlid ranges are what the merge depends on. This matters beyond the comment: mismatching the identity is not a way to force two subsystem instances onto one NQN, so it is not a way to inject an ambiguous-head defect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
A host outage cannot be injected from inside the guest: it takes down the pod a test would drive it from, and the initiator observing it has to be somewhere else anyway. talosctl's provisioner opens a QEMU monitor per node, and the node names Kubernetes reports are the names the provisioner used, so a node name is enough to find its socket. What this makes expressible is the difference between a host that is gone and one that has stopped answering. Deleting an nvmet subsystem is a deliberate teardown — the kernel's configfs paths call nvmet_port_del_ctrls and nvmet_subsys_del_ctrls — so the initiator sees its connections close. A frozen virtual machine closes nothing and answers nothing, and the initiator finds out when keep-alive expires. Only the second is what a failing host does, and no amount of configfs on a live host reproduces it. Freeze, thaw, link down and up, ACPI shutdown and power off. Power off has no counterpart: the provisioner owns the QEMU processes and cannot restart one, so a node cut that way is gone for the cluster's life. The sockets are root-owned because QEMU created them as root, and connecting to a unix socket needs write permission on it, so Create hands them over the way it already hands over the work directory. The line protocol is tested against a stand-in that greets and prompts the way HMP does, which covers the framing — that a reply is read to its end, that output is not confused with the banner, that a monitor dying mid-command is reported, and that `quit` losing the connection counts as success. What a stand-in cannot cover is whether QEMU accepts the commands, so the smoke test asks a real one and freezes the node to prove the fault takes effect rather than merely being accepted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
wasmos drives QEMU monitors for the same purpose and pins its NIC's id explicitly so set_link can target it — "it must keep id=net0". That option belongs to whoever writes the command line, and here that is talosctl, not us. Reading net0 off a log and hardcoding it would turn a change there into a link fault that reports success and disconnects nothing, which is the worst way for a fault injector to fail. So the backend is read from `info network` and cached per connection, and a monitor that reports no backend is an error rather than a guess. The dial retries while the socket refuses connections, which is what wasmos does for the same reason: QEMU binds the socket at startup but a node that has just been created may not have got there yet. Its monitor is QMP, with an escape hatch to HMP for what the typed calls do not cover. That choice is not available here — talosctl launches with -monitor, so the socket speaks HMP and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
The monitor runs its input through readline even over a socket, so a reply arrives
behind the command redrawn one character at a time, interleaved with erase and
cursor-left sequences. The stand-in these tests were written against answered
cleanly, which hid it.
Left in place the echo is indistinguishable from output, and the field that breaks
first is the one parsed positionally: the first line of `info network` would be the
echo, so set_link would be handed that as the network backend's name and would
fault nothing while reporting success.
The echo is the first line — readline redraws the command as it is typed and the
newline is the Enter that submitted it — so that is where the reply starts.
Cutting at the command's text instead does not work: HMP quotes what it is
complaining about ("Error: Bus 'pcie.0' does not support hotplugging"), so the
search lands inside the message worth reading. The first attempt did exactly that
and the rejected-command test caught it.
The stand-in now echoes the way the monitor does, so the tests exercise the case
that was broken.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TXYkVSGj2zuTRg3SQq2sBK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration test framework: Talos/QEMU cluster + nvmet fabric
Draft. First two pieces of an integration tier that runs against a real kernel, so the
fabric states that only exist below Kubernetes can be produced deliberately rather than
waited for.
Nothing here tests the CSI driver yet. This PR exists to prove the harness itself works in
CI before anything is built on it.
Why a virtual machine at all
Nodes have to host an NVMe-oF target, not just attach to one — that is the only way to
produce a subsystem exporting no namespace, or a controller serving no path, without a
storage backend. Which rules out the cheap options:
kindandk3dshare the host kernel, so two "nodes" cannot hosttwo targets for one NQN. macOS has no
nvmetat all.nvmetinlinux-modules-extra, so every node needsan apt install before it is useful — which means building and cache-invalidating a
provisioned base image.
Why Talos
CONFIG_NVME_TARGET=mandCONFIG_NVME_TARGET_TCP=mon both amd64 and arm64, declared inthe machine config and loaded at boot. Nothing to install. It also builds with
CONFIG_NVME_TARGET_AUTH=y, which is what would make target-side DHCHAP reachable — relevantto #417.
talosctl's QEMU provisioner is used rather than driving QEMU directly, for two reasons thatturned out to matter more than convenience:
guest on one address with no route between them, which is exactly what a same-NQN target
pair cannot live with. I wrote ~600 lines of QEMU/cloud-init/SSH harness before noticing
this; it was discarded.
path. I had assumed the QEMU provisioner was Linux-only. It is not.
Resource cost
Single node by default: 2 GiB, scheduling allowed on the control plane. Only a same-NQN
target pair needs a second kernel, so those specs ask for a worker and nothing else pays for
one (3 GiB when they do).
ubuntu-latestis 4 vCPU / 16 GB on a public repo, so RAM is not the constraint. Forcomparison, k3s on Ubuntu would be roughly 1 GiB less for the same two nodes — against which
Talos removes the apt provisioning, the base-image cache, and the addressing problem.
The CI wrinkle worth knowing
/dev/kvmexists onubuntu-latest, but the runner user is not in thekvmgroup, and agroup added in a step does not apply to that step's shell. Anything that spawns QEMU
therefore runs under
sg kvm -c. Without it QEMU falls back to emulation, where a Talos boottakes minutes and the suite times out looking like a hang rather than a misconfiguration.
The workflow checks
/dev/kvmexists and that the accelerator is reachable as the runneruser, and fails fast with an annotation if not.
talosctlis version-pinned so a clusterfailure is never version skew.
What is in it
cluster/talos.gonvmetmodule patch, single-node scheduling patch, split control-plane/worker memory, kubeconfig fetchcluster/kubectl.gofabric/nodeshell.gosuites/smoke_test.go.github/workflows/integration_test.yamlNode access. A privileged pod pinned with
nodeName, carryinghostPIDand the host's/sys,/devand/lib/modules.Runexecutes in the pod, which is enough to read/sys;RunOnHostgoes throughnsenter --target 1somodprobeandmountaffect the node ratherthan the container. This is the same access the CSI driver itself has.
The smoke test involves no driver and no control plane, so a first failure points at one
thing. It:
/proc/modulesrather than modprobing — which proves the machine-config patch tookeffect, not merely that the modules exist;
attr_cntlid_min, the gate on everything else: without it the host rejects a secondcontroller for one NQN as a duplicate cntlid, so a controller that contributes no path
cannot be produced.
Honest status
No cluster has booted with this yet. This PR is the first run. The three smoke assertions
are precisely the things I could not verify locally, and CI failing them is a useful result.
Kubernetes access is
kubectl, notclient-go— the harness needs apply, exec and a fewwaits, and
client-gobecomes the right answer once it needs watches. Confined to one file.The suite skips unless
SB_INTEGRATION=1, sogo test ./...in a repository check does notboot virtual machines.
Next, in order
fabric/two-target pair, asserting the kernel merged them into one subsystem with onecontroller carrying an empty leg set. That is
controller-not-contributing, and it needsno CSI driver and no control plane — the configfs sequence is already proven by hand
against a live cluster.
shared/openapi.jsonwith the pinnedoapi-codegen, serving the 11 routes the driver calls and handing out nvmet endpoints as the
published paths. Generating the server from the same spec is what keeps the fake from
drifting: a spec change breaks it at compile time.
needs, since the driver has to be the thing that tears the controller down.
Review notes
The privileged pod manifest in
fabric/nodeshell.gois worth a look —hostPIDplus host/sysand/devis a lot of access, and it lands in CI. It is confined to thesb-integrationnamespace and torn down per test.