volumes: attach-only claims leave mounting to the workload

docs/deploy.md

@@ -219,6 +219,18 @@ with concurrent readers.
   proceed; a crashed writer leaves it, and `ro` claims are refused until one
   `rw` claim replays and releases cleanly.
 
+**Attach-only claims opt out of the marker, not out of admission.** A claim
+sent with `volumes_attach_only` gets the device attached and nothing else, so
+sandboxd neither writes nor clears `<path>.dirty` for it — it cannot verify a
+mount it did not perform. Everything in this section still applies to the
+default, mounting claims, and the exclusion rules above apply to attach-only
+claims exactly the same way, which is what keeps other tenants safe. The
+operator-visible difference: an image whose attach-only writer released
+without unmounting cleanly carries no marker, so the next `ro` claim is
+admitted and fails at mount time (500) instead of being refused early (409).
+Whoever hands out attach-only `rw` access owns that trade; see
+[sandboxd-api](sandboxd-api.md#attach-only-volumes).
+
 ### A fuller config
 
 The block above is the minimum. A production node with tenants, guarded

docs/sandboxd-api.md

@@ -47,6 +47,12 @@ Auth: `Authorization: Bearer <api_token>` (when configured).
   `mode` is `"ro"` (default, omitted) or `"rw"`; `"rw"` requires the catalog
   entry's `writable: true` (see [deploy](deploy.md#dataset-volumes)). Volumes
   require Cloud Hypervisor
+- `volumes_attach_only` (default `false`) attaches every requested volume
+  without mounting it, handing the whole mount contract to the workload. It
+  requires at least one volume, and rejects any entry carrying a `mount` —
+  the path is meaningless when the caller mounts the device. Everything below
+  describes the default, eager behaviour, which is completely unchanged; the
+  attach-only contract is in [its own section](#attach-only-volumes)
 
 Success:
 
@@ -75,6 +81,51 @@ filesystem — read-only, unless the entry requested and was granted `rw` —
 before returning. A custom mount may shadow an existing populated guest
 directory for the claim's life.
 
+### Attach-only volumes
+
+`"volumes_attach_only": true` stops after the attach. The device appears in
+the guest, nothing is mounted, and the echoed entries carry no `mount` key:
+
+```json
+{"id": "sb_…", "token": "…",
+ "volumes": [{"name": "imagenet"}, {"name": "scratch-db", "mode": "rw"}]}
+```
+
+Find each device by its virtio serial, which is the catalog name:
+
+```sh
+for dev in /sys/block/*/serial; do
+  [ "$(cat "$dev")" = scratch-db ] && echo "/dev/$(basename "$(dirname "$dev")")"
+done
+```
+
+Then mount it however the workload needs. A `ro` entry is attached
+`--readonly` and stays read-only at the guest block layer no matter who
+mounts it, so the guarantee does not depend on your mount flags.
+
+The whole consistency contract moves to the caller with the mount:
+
+- sandboxd writes no dirty marker for an attach-only `rw` claim and clears
+  none, because it cannot verify your unmount and therefore promises nothing.
+  Releasing without a clean unmount of your own leaves the image exactly as a
+  filesystem crash would. The next *eager* `ro` claim then fails at mount time
+  with a 500 — loud and attributable to the claim that skipped its unmount —
+  instead of the marker's 409. Recovery is any `rw` cycle that replays the
+  journal.
+- a marker left by an earlier *eager* `rw` crash is not cleared by attach-only
+  cycles, and still refuses eager `ro` claims with 409 until a `rw` claim
+  releases cleanly.
+- writes straight to the block device bypass the filesystem journal entirely
+  and are outside marker protection in either mode.
+
+What still protects other claims is unchanged: admission excludes an
+attach-only `rw` claim against every other claim of the name (and readers
+against a writer), an attach-only `ro` claim of a marker-bearing image is
+refused with 409, and a claim with volumes still refuses checkpoint, fork and
+hibernate. An attach-only claim costs one disk attach per volume — no device
+settle, no mount round-trip — and release costs nothing at all: removing the
+VM closes the devices.
+
 Redirects (mutually exclusive with the fields above) name peers to retry
 at — sent on a warm miss with warm peers, when the node lacks a golden for
 the key but gossip names a template owner, and when the node is at
@@ -102,7 +153,9 @@ name (attributed in the usage journal and counted against the tenant's
 `max_claims`). A catalog access list may restrict an entry to named tenants;
 an unknown and a forbidden volume return the same error text.
 
-Errors: 400 unknown template axis, invalid/duplicate volumes, `mode: "rw"`
+Errors: 400 unknown template axis, invalid/duplicate volumes,
+`volumes_attach_only` with no volumes or with an entry carrying a `mount`,
+`mode: "rw"`
 against a non-writable entry, or a volume that is unknown or forbidden (the
 latter two are deliberately indistinguishable), Firecracker with volumes, or
 bad body; 401 bad api token; 409 egress requested on a node without an egress

docs/sdk-python.md

@@ -96,6 +96,7 @@ sb = client.new("ghcr.io/cocoonstack/sandbox/rt:24.04",
 | `net` | `"none"`, `"egress"` | `"none"` | Cloud Hypervisor network shape: `none` disables the NIC and uses vsock-only I/O; `egress` attaches a bridge/CNI NIC |
 | `size` | `"small"`, `"medium"`, `"large"`, `"xlarge"` | `"small"` | resource tier: 1cpu/512M, 2cpu/1G, 4cpu/4G, 4cpu/8G |
 | `volumes` | bare names or `{name, mount?, mode?}` mappings | `None` | attach and mount up to eight unique catalog dataset disks; an omitted mount defaults to `/volumes/<name>`; `mode` is `"ro"` (default) or `"rw"` — `"rw"` requires the catalog entry's `writable: true`; accepted by `Client.new` and `Template.new` |
+| `mount` | bool | `True` | mount every requested volume. `False` attaches the devices and leaves the mounting to the workload; a mapping carrying `mount` is then a `TypeError` |
 | `ttl_seconds` | int | server default 5m | sandbox TTL, server-capped at 24h. The node reaps the sandbox after the TTL even if the client vanishes |
 
 `new` returns when the sandbox's silkd answers: a warm hit is milliseconds,
@@ -113,6 +114,16 @@ an error — double-release and reap races stay silent).
 Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Checkpoint
 branches do not accept volumes of either mode in this version.
 
+`mount=False` on `client.new` or `template.new` claims the same volumes
+without mounting them: the dictionaries in `sb.volumes` carry no `mount` key,
+and the workload finds each device by polling `/sys/block/*/serial` for the
+catalog name. Everything above describes the default (`mount=True`) and is
+unchanged by it. What changes is that the mount and its consistency are
+entirely yours: sandboxd writes and clears no dirty marker for an attach-only
+`rw` claim, because it cannot verify your unmount, so releasing without
+unmounting cleanly leaves the image as a crash would — see
+[sandboxd-api](sandboxd-api.md#attach-only-volumes) for the full contract.
+
 The caller-visible constraints are deliberate: volume claims may consume a
 warm VM, remain non-capturable, mount read-only by default, and require Cloud
 Hypervisor.

docs/sdk.md

@@ -144,6 +144,7 @@ defer sb.Close()
 | `WithNetwork(n)` | `NetNone`, `NetEgress` | `NetNone` | Cloud Hypervisor network shape: `NetNone` disables the NIC and uses vsock-only I/O; `NetEgress` attaches a bridge/CNI NIC |
 | `WithSize(s)` | `Small`, `Medium`, `Large`, `XLarge` | `Small` | resource tier: 1cpu/512M, 2cpu/1G, 4cpu/4G, 4cpu/8G |
 | `WithVolumes(volumes...)` | `Volume{Name, Mount?, Mode?}` entries | none | attach and mount up to eight unique catalog dataset disks; `Mount` defaults to `/volumes/<name>`; `Mode` is `"ro"` (default) or `"rw"` — `"rw"` requires the catalog entry's `writable: true`; supported by `Client.New` and `Template.New` |
+| `WithVolumesAttachOnly()` | — | mount | attach the requested volumes without mounting them; the workload finds each device and owns the mount. Rejects a `Volume.Mount` locally |
 | `WithTimeout(d)` | duration | server default 5m | sandbox TTL, rounded up to seconds, server-capped at 24h. The node reaps the sandbox after the TTL even if the client vanishes |
 
 `New` returns when the sandbox's silkd answers: a warm hit is milliseconds,
@@ -163,6 +164,16 @@ Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Passing
 `WithVolumes` to `Checkpoint.New` returns a local error because checkpoint
 branches do not support volumes of either mode in this version.
 
+`WithVolumesAttachOnly()` claims the same volumes without mounting them: the
+entries in `Sandbox.Volumes` carry an empty `Mount`, and the workload finds
+each device by polling `/sys/block/*/serial` for the catalog name. Everything
+above describes the default and is unchanged by this option. What changes is
+that the mount and its consistency are entirely yours: sandboxd writes and
+clears no dirty marker for an attach-only `rw` claim, because it cannot verify
+your unmount, so releasing without unmounting cleanly leaves the image as a
+crash would — see
+[sandboxd-api](sandboxd-api.md#attach-only-volumes) for the full contract.
+
 The caller-visible constraints are deliberate: volume claims may consume a
 warm VM, remain non-capturable, mount read-only by default, and require Cloud
 Hypervisor.

e2e/e2e_test.go

@@ -6,6 +6,7 @@ package e2e
 
 import (
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"maps"
@@ -426,6 +427,99 @@ func TestVolumeModeWireShape(t *testing.T) {
 	}
 }
 
+// TestAttachOnlyVolumeEndToEnd drives one attach-only writable claim through
+// the whole stack: the device is attached writable and nothing else happens —
+// no mount, no marker, no unmount at release — while admission still excludes
+// every other claim on the name, which is what protects third parties.
+func TestAttachOnlyVolumeEndToEnd(t *testing.T) {
+	scratch := writeVolumeImage(t, "scratch.img", "scratch-bytes")
+	stack := startTenantStack(t, "node-token", nil,
+		[]config.VolumeSpec{{Name: "scratch", Path: scratch, Writable: true}})
+
+	sb, err := stack.client.New(t.Context(), "rt:24.04",
+		sandbox.WithVolumes(sandbox.Volume{Name: "scratch", Mode: "rw"}),
+		sandbox.WithVolumesAttachOnly())
+	if err != nil {
+		t.Fatalf("attach-only claim: %v", err)
+	}
+	if want := []sandbox.Volume{{Name: "scratch", Mode: "rw"}}; !slices.Equal(sb.Volumes, want) {
+		t.Errorf("claim volumes %+v, want %+v", sb.Volumes, want)
+	}
+	applied := []string{"attach:scratch:rw"}
+	if got := stack.eng.volumeOpsLog(); !slices.Equal(got, applied) {
+		t.Errorf("engine ops %v, want %v", got, applied)
+	}
+	assertNoDirtyMarker(t, scratch, "apply")
+	for _, requested := range []string{`{"name":"scratch"}`, `{"name":"scratch","mode":"rw"}`} {
+		if status, _ := rawClaim(t, stack, requested); status != http.StatusConflict {
+			t.Errorf("claim %s under a live attach-only writer: %d, want 409", requested, status)
+		}
+	}
+
+	if err := sb.Close(); err != nil {
+		t.Fatalf("Close: %v", err)
+	}
+	want := slices.Concat(applied, []string{"remove"})
+	if got := stack.eng.volumeOpsLog(); !slices.Equal(got, want) {
+		t.Errorf("engine ops after release %v, want %v", got, want)
+	}
+	assertNoDirtyMarker(t, scratch, "release")
+}
+
+// TestAttachOnlyVolumeWireShape pins both claim replies' volume bytes: an
+// attach-only entry echoes without a mount, the eager entry is unchanged, and
+// the request flag never rides back in either.
+func TestAttachOnlyVolumeWireShape(t *testing.T) {
+	scratch := writeVolumeImage(t, "scratch.img", "scratch-bytes")
+	stack := startTenantStack(t, "node-token", nil,
+		[]config.VolumeSpec{{Name: "scratch", Path: scratch, Writable: true}})
+	for _, tt := range []struct {
+		name    string
+		request string
+		want    map[string]any
+	}{
+		{
+			"attach-only omits the mount",
+			`{"template":"rt:24.04","volumes_attach_only":true,"volumes":[{"name":"scratch","mode":"rw"}]}`,
+			map[string]any{"name": "scratch", "mode": "rw"},
+		},
+		{
+			"eager claim is unchanged",
+			`{"template":"rt:24.04","volumes":[{"name":"scratch","mode":"rw"}]}`,
+			map[string]any{"name": "scratch", "mount": "/volumes/scratch", "mode": "rw"},
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			status, body := rawJSON(t, stack, http.MethodPost, "/v1/claim", tt.request)
+			if status != http.StatusOK {
+				t.Fatalf("claim: %d %s, want 200", status, body)
+			}
+			var reply map[string]any
+			if err := json.Unmarshal(body, &reply); err != nil {
+				t.Fatalf("decode claim %s: %v", body, err)
+			}
+			if _, leaked := reply["volumes_attach_only"]; leaked {
+				t.Errorf("reply %s carries the request flag", body)
+			}
+			entries, _ := reply["volumes"].([]any)
+			if len(entries) != 1 {
+				t.Fatalf("reply volumes %v, want one entry", reply["volumes"])
+			}
+			entry, _ := entries[0].(map[string]any)
+			if !maps.Equal(entry, tt.want) {
+				t.Errorf("reply volume %v, want %v", entry, tt.want)
+			}
+			var claimed rawClaimResponse
+			if err := json.Unmarshal(body, &claimed); err != nil {
+				t.Fatalf("decode claim %s: %v", body, err)
+			}
+			if err := stack.client.Attach(stack.addr, claimed.ID, claimed.Token).Close(); err != nil {
+				t.Fatalf("release: %v", err)
+			}
+		})
+	}
+}
+
 // TestDirtyVolumeRefusesReader: the marker a crashed writer leaves behind
 // (pre-created here) turns read-only claims into 409s over the wire.
 func TestDirtyVolumeRefusesReader(t *testing.T) {
@@ -504,6 +598,15 @@ func writeVolumeImage(t *testing.T, name, content string) string {
 	return image
 }
 
+// assertNoDirtyMarker fails if the image carries the write-ahead marker: an
+// attach-only claim makes no consistency promise, so it must never write one.
+func assertNoDirtyMarker(t *testing.T, image, when string) {
+	t.Helper()
+	if _, err := os.Stat(image + ".dirty"); !errors.Is(err, os.ErrNotExist) {
+		t.Errorf("dirty marker at %s: stat=%v, want no marker", when, err)
+	}
+}
+
 // rawClaimResponse decodes the volume entries generically, so the assertion
 // is the server's own JSON rather than the SDK's mirror of it.
 type rawClaimResponse struct {

sandboxd/pool/claim.go

@@ -56,7 +56,7 @@ func (m *Manager) ClaimWarm(ctx context.Context, key types.PoolKey, ttl time.Dur
 	}
 	m.kickRefill()
 	if volumeErr := m.applyVolumes(ctx, sb, volumeSpecs, applied); volumeErr != nil {
-		m.destroy(ctx, sb.VMName)
+		m.abortVolumeClaim(ctx, sb.VMName, &reserved)
 		return nil, volumeErr
 	}
 	sb.Tenant = tenant
@@ -284,10 +284,9 @@ func (m *Manager) finalizeBatch(ctx context.Context, sbs []*types.Sandbox, ttl t
 	if quotaErr := m.quotaErr(len(sbs), sbs[0].Tenant); quotaErr != nil {
 		m.mu.Unlock()
 		for _, sb := range sbs {
-			// No quiesce: the claim was never handed out, so nothing wrote to
-			// the guest and the marker converges on the next writable claim.
-			m.unreserveVolumes(sb.Volumes)
-			m.destroy(ctx, sb.VMName)
+			// No quiesce: never handed out, so nothing wrote to the guest and the
+			// marker waits for the next writable claim; the holds ride the removal.
+			m.abortVolumeClaim(ctx, sb.VMName, &sb.Volumes)
 		}
 		return quotaErr
 	}
@@ -338,6 +337,13 @@ func (m *Manager) rollbackClaim(ctx context.Context, sbs []*types.Sandbox) {
 	}
 }
 
+// abortVolumeClaim removes a failed claim's VM, holds riding the teardown;
+// clearing reserved stops the deferred unreserve from double-releasing them.
+func (m *Manager) abortVolumeClaim(ctx context.Context, vmName string, reserved *[]types.Volume) {
+	m.removeOrRetry(ctx, vmName, "", "", volumeTeardown{holds: *reserved})
+	*reserved = nil
+}
+
 func (m *Manager) reapOnce(ctx context.Context) {
 	now := time.Now()
 	type victim struct {
@@ -529,7 +535,7 @@ func (m *Manager) claimProvision(ctx context.Context, key types.PoolKey, ttl tim
 		return nil, err
 	}
 	if volumeErr := m.applyVolumes(ctx, sb, volumeSpecs, applied); volumeErr != nil {
-		m.destroy(ctx, sb.VMName)
+		m.abortVolumeClaim(ctx, sb.VMName, &reserved)
 		return nil, volumeErr
 	}
 	sb.TemplateDigest = golden.templateDigest

sandboxd/pool/volume.go

@@ -139,7 +139,9 @@ func (m *Manager) resolveVolumes(key types.PoolKey, tenant string, requested []t
 	if len(requested) == 0 {
 		return nil, nil
 	}
-	applied, err := types.ValidateVolumes(requested)
+	// An attach-only claim arrives with no mounts to default: re-defaulting
+	// them here would mount what the caller asked to mount itself.
+	applied, err := types.ValidateVolumes(requested, types.VolumesAttachOnly(requested))
 	if err != nil {
 		return nil, fmt.Errorf("%w: %v", ErrBadVolume, err)
 	}
@@ -160,6 +162,7 @@ func (m *Manager) resolveVolumes(key types.PoolKey, tenant string, requested []t
 		}
 		disk := entry.disk
 		disk.RW = volume.RW()
+		volume.AttachOnly = false // past validation the empty mount carries it
 		resolved = append(resolved, resolvedVolume{disk: disk, applied: volume})
 	}
 	return resolved, nil
@@ -204,15 +207,20 @@ func (m *Manager) applyVolumes(ctx context.Context, sb *types.Sandbox, volumes [
 
 // applyVolume keeps one volume's steps strictly ordered; siblings overlap freely.
 func (m *Manager) applyVolume(ctx context.Context, sb *types.Sandbox, volume resolvedVolume) error {
+	// Attach-only mounts nothing, so there is no umount to verify and no marker.
+	attachOnly := volume.applied.Mount == ""
 	// Write-ahead: the marker must be durable before any guest write can be.
-	if volume.disk.RW {
+	if volume.disk.RW && !attachOnly {
 		if err := markVolumeDirty(volume.disk.Path); err != nil {
 			return fmt.Errorf("mark volume %q dirty: %w", volume.applied.Name, err)
 		}
 	}
 	if err := m.eng.DiskAttach(ctx, sb.VMName, volume.disk); err != nil {
 		return fmt.Errorf("attach volume %q: %w", volume.applied.Name, err)
 	}
+	if attachOnly {
+		return nil
+	}
 	if err := m.eng.MountVolume(ctx, sb.VsockSocket, volume.applied.Name, volume.applied.Mount, volume.disk.RW); err != nil {
 		return fmt.Errorf("setup volume %q: %w", volume.applied.Name, err)
 	}
@@ -222,10 +230,12 @@ func (m *Manager) applyVolume(ctx context.Context, sb *types.Sandbox, volume res
 // quiesceVolumes unmounts a claim's writable mounts in reverse order and
 // returns the teardown its VM removal must finish. A failed unmount never
 // blocks teardown: the image keeps its marker and waits for a recovering
-// writer. Runs while the guest is still live, before the VM is removed.
+// writer. Attach-only entries have no mount to bring down — removing the VM
+// closes their devices. Runs while the guest is still live, before the VM is
+// removed.
 func (m *Manager) quiesceVolumes(ctx context.Context, sb *types.Sandbox) volumeTeardown {
 	td := volumeTeardown{holds: sb.Volumes}
-	mounts := len(types.VolumeRWNames(sb.Volumes))
+	mounts := writableMounts(sb.Volumes)
 	if mounts == 0 {
 		return td
 	}
@@ -235,7 +245,7 @@ func (m *Manager) quiesceVolumes(ctx context.Context, sb *types.Sandbox) volumeT
 	defer cancel()
 	stuck := false
 	for _, volume := range slices.Backward(sb.Volumes) {
-		if !volume.RW() {
+		if !volume.RW() || volume.Mount == "" {
 			continue
 		}
 		if err := m.eng.UnmountVolume(ctx, sb.VsockSocket, volume.Mount); err != nil {
@@ -330,6 +340,16 @@ func quiesceBudget(mounts int) time.Duration {
 	return min(time.Duration(mounts+1)*engine.VolumeCallTimeout, volumeQuiesceMax)
 }
 
+func writableMounts(volumes []types.Volume) int {
+	mounts := 0
+	for _, volume := range volumes {
+		if volume.RW() && volume.Mount != "" {
+			mounts++
+		}
+	}
+	return mounts
+}
+
 func appliedVolumes(volumes []resolvedVolume) []types.Volume {
 	applied := make([]types.Volume, len(volumes))
 	for i, volume := range volumes {