diff --git a/README.md b/README.md index 576c484a..26088a73 100644 --- a/README.md +++ b/README.md @@ -33,9 +33,9 @@ performance) — source in baked into the base image - `sandboxd/` — per-node control plane (Go): warm pools refilled from golden snapshot exports (online-retunable), claim/release/hibernate/fork/promote/ - checkpoint HTTP API, signed preview URLs, the HTTP-upgrade byte relay to - silkd, usage + audit journals, /metrics, reap + restart reconcile, - memberlist mesh with redirect placement + checkpoint HTTP API, operator-catalog read-only dataset volumes, signed + preview URLs, the HTTP-upgrade byte relay to silkd, usage + audit journals, + /metrics, reap + restart reconcile, memberlist mesh with redirect placement - `sdk/go/` — Go SDK (stdlib-only): `Connect/New/Lookup`, `Exec/Run`, files, `Push/Pull`, sessions, `Find/Replace`, `Watch`, git verbs, `OpenPty`, `Fork/Hibernate/Promote/Checkpoint`, `DialPort/ProxyPort/PreviewURL`, @@ -54,8 +54,9 @@ performance) — source in - `e2e/` — in-process full-stack tests (real pool/engine/relay/SDK, fake cocoon+guest) plus bare-metal acceptance drivers under `cmd/`: `demo`, `smoke`, `meshsmoke`, `crossnode`, `coldproof`, `egresssmoke`, - `interceptsmoke`, `lifecycle` (idle→hibernate→archive), `androidsmoke`, - `browsersmoke`, and the `pullbench`/`pushbench`/`rpcbench` perf drivers + `interceptsmoke`, `volumesmoke`, `lifecycle` (idle→hibernate→archive), + `androidsmoke`, `browsersmoke`, and the `pullbench`/`pushbench`/`rpcbench` + perf drivers - `boot/kernel/` — kernel version pin (`VERSION` + matching tarball `SHA256`, bump both together) + config fragment (amd64: over `x86_64_defconfig` + `kvm_guest.config`; arm64: over `defconfig` + `sandbox-arm64.config`) @@ -102,6 +103,8 @@ TEMPLATE=rt:24.04 scripts/sandboxd-e2e.sh # `ip link add br0 type bridge` with no uplink is enough (NIC, not network). # SANDBOXD_BIN/DEMO_BIN/SMOKE_BIN point at prebuilt binaries for nodes # without a Go toolchain. +# VOLUME_IMAGE=/absolute/dataset.img enables the read-only sharing proof; the +# image contains volume-e2e.txt. Prebuilt runs also set VOLUME_SMOKE_BIN. ``` ## CI diff --git a/docs/cluster.md b/docs/cluster.md index 97dae109..2557d25b 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -2,10 +2,11 @@ A cluster is a set of sandboxd nodes joined through a [hashicorp/memberlist](https://github.com/hashicorp/memberlist) SWIM mesh. -Gossip carries only placement hints — per-pool warm counts and each node's -data-plane address. Per-sandbox state never leaves its owning node, so a -stale view costs at most one extra redirect, never correctness. A single -node with no seeds is a valid mesh of one. +Gossip carries only placement hints — per-pool warm counts, promoted-template +hashes, available volume names, and each node's data-plane address. +Per-sandbox state never leaves its owning node, so a stale view costs at most +one extra redirect, never correctness. A single node with no seeds is a valid +mesh of one. ## Joining @@ -57,6 +58,32 @@ Node death is honest: a dead node's sandboxes die with it (memory state is node-local by design). SWIM detects the death and peers stop redirecting to it. +### Read-only volumes and placement + +A volume name has one fleet-wide meaning and access list, while catalog +membership is node-local and deliberately excluded from the cluster config +digest. Nodes gossip only their currently available catalog names: host paths +and access lists never leave the node. After config load the set appears on the +next gossip tick; later image distribution or removal is detected the same way. +The node epoch bumps only when the advertised name set changes. + +A volume claim may consume an ordinary warm VM because attach happens after the +pop and before finalization. Warm candidates retain their normal ranking, but a +candidate must advertise every requested volume. If the entry node cannot serve +all requested names, it redirects once to a peer advertising their intersection. +A promoted-template claim uses the intersection of template owners and volume +owners first. If that advertised intersection is empty, a volume holder gets one +chance to prove it can resolve the template from a shared store. The target +retries with `no_redirect` plus the carried `require_promoted` intent and +validates both resources before provisioning, so a node-local template still +fails without a second hop even while template gossip is one tick stale. + +`GET /v1/volumes` and the SDK discovery calls return the gossiped union filtered +through the answering node's fleet-uniform access lists. `nodes` counts members +advertising each name, while `available` and `size_bytes` describe only the +answering node's image. No node address or dataset-to-host mapping is returned; +claim placement resolves the holder. + ## Querying members `GET /v1/info` (root `api_token`) reports this node's pools plus the peer @@ -119,9 +146,9 @@ Gossip is eventually consistent: a template promoted a moment ago may be invisible to name-based calls for about a gossip tick (the claim fails cold — retry), and one deleted a moment ago may still redirect to a 404. Correctness is never violated; only the name-based convenience lags. The -handle `Sandbox.Promote` returns is still owner-bound — `template.New` and -`template.Delete` dial the owner directly, no gossip involved — and is the -race-free choice immediately after a promote. +handle `Sandbox.Promote` returns is still owner-bound. `template.Delete` and a +`template.New` without volumes dial the owner directly; a volume claim may use +one placement redirect to a node that can resolve both resources. ## Checkpoints on a cluster @@ -179,10 +206,10 @@ fully from it: | state | source of truth | survives restart | |---|---|---| -| operator config (`tenants`, `secrets`, egress policies, `bridges`/`networks`, `mesh`, `preview_secret`, `egress_ca`) | `config.json` (human/deploy-tool owned) | re-read at boot | +| operator config (`tenants`, `volumes`, `secrets`, egress policies, `bridges`/`networks`, `mesh`, `preview_secret`, `egress_ca`) | `config.json` (human/deploy-tool owned) | re-read at boot | | API-applied pool targets (`PUT /v1/pools`) | `/pools.json` (machine owned) | yes | | claims | the claims journal + `Reconcile` | yes | -| placement hints (warm counts, template sets) | gossip | rebuilt | +| placement hints (warm counts, template and volume sets) | gossip | rebuilt | | checkpoint ownership | a live per-request probe (no gossip); a healed replica is this node's own persisted copy, aged out by `checkpoint_ttl_hours` | yes | Pools are managed API-first. The first time a node takes `PUT /v1/pools`, it @@ -235,3 +262,6 @@ the mesh. - `cluster_key` set if the gossip network is not otherwise trusted - pool changes via `Client.SetPoolsCluster` (or per-node `SetPools`); the applied set persists to `pools.json` and survives restart +- keep each volume name's dataset identity and access list identical across the + fleet, distribute its immutable image to every node meant to advertise it, + and verify the fleet view and holder count with `GET /v1/volumes` diff --git a/docs/deploy.md b/docs/deploy.md index 1d28609a..6ed34bdd 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -7,8 +7,9 @@ the cocoon CLI and needs a template image with silkd baked in. - Linux with KVM (`/dev/kvm`) - [cocoon](https://github.com/cocoonstack/cocoon) **v0.5.2 or newer** installed - and working (`cocoon vm run` boots a Cloud Hypervisor VM). v0.5.2 adds the - parallel-clone and snapshot/store performance work the + and working (`cocoon vm run` boots a Cloud Hypervisor VM). v0.5.2 includes + the disk hot-attach used by read-only volumes and the parallel-clone and + snapshot/store performance work the [performance](performance.md) numbers assume. sandboxd logs a warning at startup when the detected cocoon is below v0.5.2 (a dev/`master-` build is assumed current) @@ -37,6 +38,13 @@ The scalar egress-attachment keys are retired: rename `"bridge": "br0"` to starting the new binary — config loading rejects the old spellings loudly rather than silently dropping the egress lane. +Read-only dataset volumes require a lockstep rollout. Upgrade every sandboxd +node and cocoon to the required version before enabling the catalog or shipping +an SDK that requests volumes. Mixed-version serving is unsupported. Once a +volume claim has finalized, do not roll a node back to an older sandboxd until +all volume claims are gone; the older daemon cannot preserve their capture +semantics from `claims.json`. + ## Configuration sandboxd reads one JSON file (`-config`, default @@ -51,7 +59,13 @@ sandboxd reads one JSON file (`-config`, default "no_direct_io": true, "advertise_addr": "10.0.0.5:7777", "bridges": ["br0"], + "volumes": [ + {"name": "imagenet", "path": "/srv/datasets/imagenet.img"}, + {"name": "weights-llama", "path": "/srv/datasets/llama.img", "directio": "on"}, + {"name": "acme-corpus", "path": "/srv/datasets/acme.img", "tenants": ["acme"]} + ], "api_token": "…", + "tenants": [{"name": "acme", "token": "…"}], "mesh": { "node_id": "node-a", "bind": "10.0.0.5:7946", @@ -74,9 +88,10 @@ sandboxd reads one JSON file (`-config`, default | `no_direct_io` | false | use buffered writable disks for Cloud Hypervisor cold boots and clones; recommended for dense ephemeral pools to avoid direct-I/O CoW journal contention | | `advertise_addr` | = `listen` | the host:port clients reach this node at; returned as a claim's owner address and gossiped to peers. Must be routable when `listen` is a wildcard | | `bridges` / `networks` | unset | egress-lane attachment: a list of host bridge devices, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. A Linux bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so an N-entry list raises the node's egress ceiling to N×1024 — VMs spread over the list by a stable hash of the VM name, so size it with headroom (the spread is statistical, not exact). `bridges` keeps the raw TAP-on-bridge attachment (taps in the root netns, no per-VM network namespace or CNI plugin execution); `networks` runs the CNI chain per VM. [Guarded egress](egress.md) needs `bridges` and rejects a CNI network at load | +| `volumes` | unset | node-local catalog of operator-managed read-only dataset images: `[ {"name":"imagenet","path":"/srv/datasets/imagenet.img","directio":"off","tenants":["acme"]} ]`. Names match `^[a-z][a-z0-9_-]{0,19}$` and cannot start with `cocoon-`; paths are absolute; `directio` is `on`, `off`, or `auto` and defaults to `off`. `tenants` is an optional access list: empty means every authenticated scope, while every listed name must exist in the node's `tenants` config. Root always has access. The catalog is intentionally not part of the cluster digest | | `egress_ca` | unset | [HTTPS-interception](egress.md#https-interception) PKI: `root_cert` (the cluster root baked into intercepted guests; may bundle old+new roots during rotation) plus this node's `intermediate_cert`/`intermediate_key` from `sandboxd ca issue-intermediate`. Required when any pool rule sets `intercept` | | `api_token` | unset | the operator (root) credential: when set, guards the node-level endpoints (Bearer) with full access, including release-by-id cleanup. Per-sandbox tokens guard ordinary sandbox-scoped calls | -| `tenants` | unset | multi-tenant tokens next to `api_token`: `[{"name": "acme", "token": "…", "max_claims": 50}]`. A tenant token reaches the resource-creating verbs (claim, fork, promote, checkpoint, preview) and everything it creates is stamped with the tenant name; operator surfaces (`GET /v1/sandboxes` and the per-id reads under it, `GET /v1/info`, `PUT /v1/pools`, `POST/DELETE /v1/drain`, `/metrics`) answer it 403. `max_claims` (0 = unlimited) caps that tenant's live claims next to the node-wide cap. Requires `api_token` set (operator surfaces need it). Names and tokens must be unique, tokens distinct from `api_token`. On a cluster all nodes must carry the same tenants set (the SDK replays a tenant token across a redirect; a peer missing that tenant answers 401), and per-node caps mean a tenant's effective cluster limit is `max_claims` × nodes. Empty = exactly the single-token behavior | +| `tenants` | unset | multi-tenant tokens next to `api_token`: `[{"name": "acme", "token": "…", "max_claims": 50}]`. A tenant token reaches the resource-creating verbs (claim, fork, promote, checkpoint, preview), catalog discovery, and its own sandbox/checkpoint listings; everything it creates is stamped with the tenant name. Root-only surfaces (per-id sandbox reads, `GET /v1/info`, `PUT /v1/pools`, `POST/DELETE /v1/drain`, `/metrics`) answer it 403. `max_claims` (0 = unlimited) caps that tenant's live claims next to the node-wide cap. Requires `api_token` set. Names and tokens must be unique, tokens distinct from `api_token`. On a cluster all nodes must carry the same tenants set (the SDK replays whichever token authorized a redirect), and per-node caps mean a tenant's effective cluster limit is `max_claims` × nodes. Empty = exactly the single-token behavior | | `max_fork_count` | 16 | children a single `fork` may create; each is a full-RAM VM, so this bounds one request's memory blast radius to the node's capacity | | `refill_concurrency` | 0 (auto) | concurrent VM provisioning budget, shared by warm-pool refills, fork clones, and the reap/hibernate/reconcile engine batches. 0 sizes it from the node: `NumCPU*2/3` clamped to [4, 256] — a 384-core node gets 256; small nodes keep a floor of 4 | | `preview_listen` | (off) | address for a preview HTTP server that serves guest ports under signed URLs; needs `preview_secret` | @@ -87,7 +102,7 @@ sandboxd reads one JSON file (`-config`, default | `checkpoint_ttl_hours` | 0 (keep forever) | ages out checkpoints older than this; the sweep runs hourly and at startup. Explicit deletes never wait for it. Must be nonzero and match fleet-wide when `checkpoint_peer_heal` is on — it is the expiry eligibility point for a healed replica a delete broadcast missed, after which its next successful hourly sweep removes it; persistent sweep failure extends retention until one succeeds, so it is not a hard ceiling | | `checkpoint_peer_heal` | false | on a cluster, lets a node pull a checkpoint it lacks from a peer — found via a live probe, not gossip — rather than failing the branch; see [placement lifecycle](cluster.md#checkpoints-on-a-cluster). Three requirements, all enforced at config load: a nonempty `api_token` (the blob transfer between peers authenticates with it; without one the raw record stream would be open), `mesh.cluster_key` set (the pull presents the fleet `api_token` to an address learned from the peer probe, so the gossip layer carrying that address must itself be authenticated), and `checkpoint_ttl_hours` nonzero (a replica a delete broadcast missed becomes eligible for expiry after it, and its next successful hourly sweep removes it — so it is the finite eligibility point, not an exact ceiling). A shared checkpoint store (`checkpoint_store` kind `s3`) ignores this setting — every node already resolves every checkpoint directly, so there is nothing to heal | | `warm_max` (pool entry) | 0 (static) | turns on the demand-adaptive watermark for that pool: the warm target rises from `warm` toward `warm_max` while claims arrive faster than the measured provision lead covers, and decays back over ~a minute of silence | -| `max_claims` | 0 (unlimited) | node-wide cap on live claims; claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster, a claim is first redirected to a warm peer) | +| `max_claims` | 0 (unlimited) | node-wide cap on live claims; claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster, normal warm-candidate placement applies, with volume claims limited to candidates holding every requested volume) | | `audit_log` | false | append every relayed request frame's op + addressing fields (never payloads) to `/audit.jsonl`, size-rotated with one `.1` backup. Records are `{t, id, op}` plus whichever addressing fields the op carries (`argv`, `path`, `dest`, `from`, `to`, `url`, `session`, `port`); preview accesses record as op `preview_dial`. A request frame whose first line exceeds 4 KiB is skipped, never truncated | | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a claim with no data-plane connection for this long is hibernated; the next call wakes it transparently. Per-pool `idle_hibernate_seconds` (in a pool entry) does the same for that pool's claims — pooled keys ignore the node-wide value. Opt-in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice | | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call restores it transparently (a checkpoint restore's latency). Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool | @@ -105,6 +120,62 @@ fragment the warm pools): | `large` | 4 | 4G | | `xlarge` | 4 | 8G | +### Read-only dataset volumes + +Each catalog path must name an immutable disk image containing a mountable +whole-device filesystem. A missing path produces a startup warning and fails +only claims that request it, allowing images to be distributed after sandboxd +starts. Do not replace, truncate, or delete an image while it is attached; +publish a new catalog name or path instead. + +For example, build a whole-device ext4 image directly from a prepared tree, +then make the published file host-read-only: + +```bash +truncate -s 200G /srv/datasets/imagenet.img +mkfs.ext4 -F -d /srv/datasets/imagenet-root /srv/datasets/imagenet.img +chmod 0444 /srv/datasets/imagenet.img +``` + +Use the same dataset identity and access list for a volume name on every node, +then distribute its immutable image to each node that should advertise it. +sandboxd gossips only catalog names; it never copies content or gossips paths or +access lists. + +A claim requests up to eight unique names and may set an absolute, clean custom +mount for each; the default is `/volumes/`. Mounts must stay outside the +guest OS directories and cannot duplicate or nest within one claim. Volume +mounts may shadow an existing populated guest directory for that claim's life. + +A volume claim may consume an ordinary warm Cloud Hypervisor VM. sandboxd +attaches after the warm pop or provision, polls `/sys/block/*/serial` for the +attach name for up to 2 seconds, then mounts the device read-only before +finalizing the claim. Both the Cloud Hypervisor attachment and the guest +filesystem mount are read-only. Setup failure destroys the VM; a popped warm VM +is refilled normally. Firecracker volume claims are rejected. + +Warm candidates retain their normal ranking, but a candidate for a volume +claim must hold every requested image. If the entry node cannot serve them all, +it redirects once to such a holder. A promoted-template claim prefers a node +advertising both resources; when a shared store has not yet made that capability +visible in gossip, a volume holder self-verifies the template before +provisioning. + +An empty catalog `tenants` list allows every authenticated scope; a nonempty +list limits the image to those tenants, while root always bypasses it. Removing +a tenant therefore requires removing every catalog reference in the same edit. +`GET /v1/volumes` reports the caller-visible fleet union and holder count, plus +the answering node's current local availability, without exposing host paths or +node addresses. Applied names and effective mounts are persisted with the +claim. Such a claim cannot hibernate, fork, checkpoint, or promote; the idle +hibernate sweep leaves it running. Release removes the VM but never deletes the +operator-owned backing image. With `directio=off`, readers share the host page +cache; use `directio=on` when cache interference matters. +A dataset mounted into an egress-lane sandbox can be uploaded wherever that +tenant's egress policy permits, so treat the ACL and egress policy as one access +decision. Image replication, write-enabled dataset disks, detach, and +refcounting remain out of scope. + ### A fuller config The block above is the minimum. A production node with tenants, guarded @@ -232,8 +303,8 @@ Three token kinds. The root `api_token` has full access — operators and single-tenant deployments need nothing else. Tenant tokens (the `tenants` list) create and manage their own resources: claims, forks, checkpoints, promoted templates, and preview URLs are stamped with the tenant name; -checkpoint listings filter to the caller's tenant, and a tenant can delete -only its own checkpoints and templates (root sees and deletes everything). +sandbox and checkpoint listings filter to the caller's tenant, and a tenant +can delete only its own checkpoints and templates (root sees everything). Operator surfaces stay root-only — a tenant token there is authenticated but not authorized, so it answers 403 (a wrong token stays 401). Per-sandbox tokens are unchanged: whoever holds a sandbox's token drives that sandbox. @@ -296,6 +367,13 @@ The repository's `scripts/sandboxd-e2e.sh` runs the full loop on a real node (golden build → warm pool → claim tiers → the complete verb smoke → reap → restart reconcile); set `BRIDGE=` to include the egress lane. +To include the read-only volume proof, put a nonempty `volume-e2e.txt` in the +filesystem image and run `VOLUME_IMAGE=/srv/datasets/imagenet.img +scripts/sandboxd-e2e.sh`. The script verifies two concurrent mounts, read-only +enforcement, warm-pool consumption, and an unchanged source checksum. On a node +using prebuilt binaries, also set `VOLUME_SMOKE_BIN` beside `SANDBOXD_BIN`, +`DEMO_BIN`, and `SMOKE_BIN`. + ## Preview URLs `preview_listen` starts a second HTTP server that serves a sandbox's guest diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 598586b9..45b822c2 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -7,7 +7,7 @@ All bodies are JSON. Three token kinds: resource-creating verbs (claim, fork, promote, checkpoint create/claim, preview mint) and the tenant-scoped listings/deletes below; everything a tenant creates is stamped with its name. Operator surfaces - (`GET /v1/sandboxes` and the per-id reads under it, `GET /v1/info`, + (the per-id sandbox reads, `GET /v1/info`, `PUT /v1/pools`, `POST/DELETE /v1/drain`, `GET /metrics`, `GET /v1/checkpoints/{id}/blob`) answer a tenant token `403` — authenticated but not authorized; an unknown @@ -25,21 +25,33 @@ Auth: `Authorization: Bearer ` (when configured). ```json {"template": "base:24.04", "net": "none", "size": "small", - "ttl_seconds": 300, "claim_ref": "namespace/workload", "no_redirect": false} + "ttl_seconds": 300, + "volumes": [{"name": "imagenet"}, {"name": "weights", "mount": "/models"}], + "claim_ref": "namespace/workload", "no_redirect": false, + "require_promoted": false} ``` - `net` defaults to `none`, `size` to `small` - `ttl_seconds` 0 means the server default (5 minutes); capped at 24h. The owning node reaps the sandbox after the TTL even if the client vanishes -- `claim_ref` is an optional opaque caller reference echoed by the root-only +- `claim_ref` is an optional opaque caller reference echoed by the scoped sandbox index; the aggregated apiserver uses `/` - `no_redirect` is set by the SDK when retrying at a redirect target +- `require_promoted` is an internal redirect field. When a redirect response + sets it, copy it into the `no_redirect` retry so the target cannot cold-boot + a promoted template name if its gossip view is stale +- `volumes` is an ordered list of at most eight unique catalog names. `mount` + defaults to `/volumes/`; a custom value must be absolute and clean, + outside the guest OS tree, unique, and non-nesting within the request. Volumes + are read-only and require Cloud Hypervisor Success: ```json {"id": "sb_…", "token": "…", "deadline": "2026-07-06T00:05:00Z", - "owner_addr": "10.0.0.5:7777", "template_digest": "sha256:…"} + "owner_addr": "10.0.0.5:7777", "template_digest": "sha256:…", + "volumes": [{"name": "imagenet", "mount": "/volumes/imagenet"}, + {"name": "weights", "mount": "/models"}]} ``` A claim cloned from a promoted template carries `template_digest`, the exact @@ -51,26 +63,60 @@ A claim branched from a checkpoint (fork children included) additionally carries `"from_checkpoint": "ck_…"` — the lineage edge for reconstructing the checkpoint tree. +`volumes` reports the names and effective mounts applied and persisted at +finalization. sandboxd attaches each disk read-only, polls +`/sys/block/*/serial` for its attach name for up to 2 seconds, and mounts the +filesystem read-only before returning. A custom mount may shadow an existing +populated guest directory for the claim's life. + Redirects (mutually exclusive with the fields above) name peers to retry at — sent on a warm miss with warm peers, when the node lacks a golden for the key but gossip names a template owner, and when the node is at `max_claims` but a peer reports warm capacity: ```json -{"redirect": ["10.0.0.6:7777", "10.0.0.7:7777"]} +{"redirect": ["10.0.0.6:7777", "10.0.0.7:7777"], + "require_promoted": true} ``` Retry the same body (+`no_redirect: true`) at each candidate until one -answers. +answers. Preserve `require_promoted: true` when the redirect carries it; +ordinary redirects omit the field. + +A volume claim may consume an ordinary warm VM. Normal candidate ranking still +applies, but every candidate must advertise all requested volumes. A +promoted-template volume claim first uses a node that advertises both resources. +If none does, a volume holder may self-verify a shared template store; a +`no_redirect` target validates both resources before provisioning. The +redirect's `require_promoted` bit makes that validation independent of one-tick +gossip lag. Redirect responses never carry `volumes`. A tenant token claims the same way; the sandbox is stamped with the tenant name (attributed in the usage journal and counted against the tenant's -`max_claims`). +`max_claims`). A catalog access list may restrict an entry to named tenants; +an unknown and a forbidden volume return the same error text. + +Errors: 400 unknown template axis, invalid/duplicate volumes, or a volume that +is unknown or forbidden (the latter two are deliberately indistinguishable), +Firecracker with volumes, or bad body; 401 bad api token; 409 egress requested +on a node without an egress attachment; 429 node at `max_claims`, the calling +tenant at its own `max_claims`, or the node draining; 500 provisioning failed. + +## GET /v1/volumes + +Auth: node API token (root or tenant). Lists the fleet catalog entries the +caller may use, without host paths or holder addresses: -Errors: 400 unknown template axis / bad body; 401 bad api token; 409 egress -requested on a node without an egress attachment; 429 node at `max_claims`, -the calling tenant at its own `max_claims`, or the node draining (a redirect -to a warm peer is tried first on a cluster); 500 provisioning failed. +```json +{"volumes": [{"name": "imagenet", "default_mount": "/volumes/imagenet", + "size_bytes": 214748364800, "available": true, "nodes": 3}]} +``` + +Root sees every entry; a tenant sees unrestricted entries plus those whose +access list names it. The response is the gossiped union: `nodes` counts members +advertising the name. `size_bytes` and `available` are a best-effort stat of the +answering node's image, so a peer-only entry remains discoverable with +`available: false`. Membership is eventually consistent by one gossip tick. ## POST /v1/sandboxes/{id}/release @@ -88,8 +134,8 @@ snapshot point and the stop coincident). Idempotent on an already-hibernated sandbox. The TTL keeps running: a hibernated sandbox is still reaped (VM and snapshot) at its deadline. When to hibernate is the caller's policy — the node only provides the transition. 204 on success, 404 unknown id or wrong -token, 409 on the egress lane (egress-lane sandboxes never hibernate; see -[egress](egress.md)). +token, 409 on the egress lane or when volumes are attached (neither kind of +sandbox hibernates; see [egress](egress.md)). ## POST /v1/sandboxes/{id}/wake @@ -123,7 +169,7 @@ All-or-nothing: on error no child survived. 200 with one claim per child: Children inherit the parent's tenant and count against its `max_claims`, whoever calls. 400 invalid count or body, 401 bad api token, 404 unknown id -or wrong sandbox token, 409 egress-lane parent (the lane never forks, +or wrong sandbox token, 409 egress-lane or volume parent (neither forks, checkpoints, or promotes; see [egress](egress.md)), 429 node or the parent's tenant at `max_claims`, or the node draining. @@ -161,7 +207,7 @@ digest; changing any exported path or bytes changes it. 400 invalid name, 401 bad api token, 409 when the name collides with a configured pool, the template is owned by another tenant, or the sandbox is -on the egress lane (see [egress](egress.md)), 404 unknown id or wrong +on the egress lane or has volumes attached (see [egress](egress.md)), 404 unknown id or wrong sandbox token. ## DELETE /v1/templates?template=…&net=…&size=… @@ -195,7 +241,7 @@ pool on a node without an egress attachment. ## POST /v1/drain Auth: root only (tenant tokens get 403). Cordons the node for maintenance: claim/fork/branch answer -429 `node draining` (on a cluster the warm-peer redirect is tried first, and +429 `node draining` (on a cluster a non-volume claim tries a warm-peer redirect first, and gossip stops naming this node within a tick as its warm counts hit zero), unclaimed warm VMs are destroyed, and live claims keep serving until release or TTL. Pool ownership is untouched — no pools.json write, no config change. @@ -226,7 +272,8 @@ Auth: node API token; body `{"token": "", "name": "..."}` answers `200 {"checkpoint": {id, name, sandbox_id, key, tenant?, created_at}}` — `tenant` records the calling tenant, absent for root. 400 bad body or name, 401 bad api token, 404 unknown id or wrong sandbox -token, 409 egress-lane sandbox (see [egress](egress.md)). +token, 409 egress-lane sandbox or one with volumes attached (see +[egress](egress.md)). ## POST /v1/checkpoints/{id}/claim @@ -316,9 +363,10 @@ an SDK caller should set. ## GET /v1/sandboxes -Auth: root only (tenant tokens get 403). The operator index: `{"sandboxes": -[{id, key, deadline, hibernated, archived?, from_checkpoint?, claim_ref?}]}` — -never tokens. +Auth: node API token. Root sees every live claim; a tenant sees only its own. +The index is `{"sandboxes": [{id, key, deadline, hibernated, archived?, +from_checkpoint?, claim_ref?, volumes?: [{name, mount}]}]}` — never sandbox +tokens, volume host paths, or catalog access lists. ## GET /v1/sandboxes/{id} @@ -360,7 +408,9 @@ Always on: every lifecycle transition appends one JSONL event to "claim|hibernate|wake|fork|checkpoint|promote|release|reap|archive|unarchive|archive_delete|egress", "id": "sb_…", "vm": "sbx-…"}` plus `key` and `tenant` (the pool key and owning tenant, claim events), `children` (fork) and `ref` (the promoted -template / checkpoint id, or the egress host). The file rotates at +template / checkpoint id, or the egress host). A volume claim also carries +`volumes`, the applied catalog names (mounts and host paths are not billing +dimensions). The file rotates at 64 MiB keeping one `.1` backup, so a tailing collector never loses a window silently. Folding rules: billable compute seconds per sandbox = Σ(claim→release/reap) − Σ(hibernate→wake); hibernated storage seconds = diff --git a/docs/sdk-python.md b/docs/sdk-python.md index 8d9e60bd..b4048f07 100644 --- a/docs/sdk-python.md +++ b/docs/sdk-python.md @@ -86,22 +86,49 @@ sb = client.lookup(id, token) # asks the entry node, then each mesh peer ```python sb = client.new("ghcr.io/cocoonstack/sandbox/rt:24.04", - net="egress", size="medium", ttl_seconds=600) + net="egress", size="medium", ttl_seconds=600, + volumes=["imagenet", {"name": "weights", "mount": "/models"}]) ``` | parameter | values | default | meaning | |---|---|---|---| | `net` | `"none"`, `"egress"` | `"none"` | Cloud Hypervisor network shape: `none` disables the NIC and uses vsock-only I/O; `egress` attaches a bridge/CNI NIC | | `size` | `"small"`, `"medium"`, `"large"`, `"xlarge"` | `"small"` | resource tier: 1cpu/512M, 2cpu/1G, 4cpu/4G, 4cpu/8G | +| `volumes` | bare names or `{name, mount?}` mappings | `None` | attach and mount up to eight unique read-only dataset disks; an omitted mount defaults to `/volumes/`; accepted by `Client.new` and `Template.new` | | `ttl_seconds` | int | server default 5m | sandbox TTL, server-capped at 24h. The node reaps the sandbox after the TTL even if the client vanishes | `new` returns when the sandbox's silkd answers: a warm hit is milliseconds, -a cold key can take the full boot. The handle exposes `sb.id`, `sb.token`, +a cold key can take the full boot. A volume claim may consume an ordinary warm +VM and returns only after every requested disk is mounted; `sb.volumes` contains +dictionaries with the finalized name and effective mount. Custom mounts must be +absolute and clean, stay outside the guest OS tree, and cannot duplicate or +nest. The handle +exposes `sb.id`, `sb.token`, `sb.owner`, `sb.deadline`, and `sb.from_checkpoint` (the lineage edge when branched). `sb.template_digest` is the exact content identity when the claim cloned a promoted template; it is empty for other sources. `Sandbox` is a context manager; `sb.close()` releases it (releasing one already gone is not an error — double-release and reap races stay silent). +Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Checkpoint +branches do not accept volumes in this version. + +The caller-visible constraints are deliberate: volume claims may consume a +warm VM, remain non-capturable, mount read-only, and require Cloud Hypervisor. + +`client.volumes()` returns the fleet entries this token may use: + +```python +for volume in client.volumes(): + print(volume["name"], volume["default_mount"], + volume["size_bytes"], volume["available"], volume["nodes"]) +``` + +Discovery returns the gossiped union and holder count; availability and size +describe the connected node. Warm candidates retain normal ranking, filtered to +nodes advertising every requested name. A promoted-template claim prefers a +peer advertising both the template and every volume; when that intersection is +empty, one volume holder may self-verify access to a shared template store +before provisioning. ## Hibernating @@ -146,7 +173,8 @@ tpl.delete() # caller owns the lifecycle Templates are keyed by (name, the sandbox's network lane, its size); on the default local-disk backend they live on the owning node (a shared store makes every node resolve them); the returned `Template` handle is -bound there, so its `new`/`delete` always reach it. The name-based calls +bound there. Its `delete` and volume-less `new` reach that node; +`new(volumes=...)` may follow one volume-placement redirect. The name-based calls (`client.new("myproj:v1")`, `client.delete_template(...)`) route cluster-wide via template gossip and lag a promote/delete by about a gossip tick — prefer the handle right after promoting (see diff --git a/docs/sdk.md b/docs/sdk.md index 0795c047..7d81cb45 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -131,6 +131,9 @@ binds to whichever confirms ownership first. sb, err := client.New(ctx, "base:24.04", sandbox.WithNetwork(sandbox.NetEgress), sandbox.WithSize(sandbox.Medium), + sandbox.WithVolumes( + sandbox.Volume{Name: "imagenet"}, + sandbox.Volume{Name: "weights", Mount: "/models"}), sandbox.WithTimeout(10*time.Minute)) defer sb.Close() ``` @@ -139,16 +142,44 @@ defer sb.Close() |---|---|---|---| | `WithNetwork(n)` | `NetNone`, `NetEgress` | `NetNone` | Cloud Hypervisor network shape: `NetNone` disables the NIC and uses vsock-only I/O; `NetEgress` attaches a bridge/CNI NIC | | `WithSize(s)` | `Small`, `Medium`, `Large`, `XLarge` | `Small` | resource tier: 1cpu/512M, 2cpu/1G, 4cpu/4G, 4cpu/8G | +| `WithVolumes(volumes...)` | `Volume{Name, Mount?}` entries | none | attach and mount up to eight unique read-only dataset disks; `Mount` defaults to `/volumes/`; supported by `Client.New` and `Template.New` | | `WithTimeout(d)` | duration | server default 5m | sandbox TTL, rounded up to seconds, server-capped at 24h. The node reaps the sandbox after the TTL even if the client vanishes | `New` returns when the sandbox's silkd answers: a warm hit is milliseconds, -a cold key can take the full boot. `Sandbox.ID`, `Sandbox.Deadline`, and +a cold key can take the full boot. A volume claim may consume an ordinary warm +VM and returns only after every requested disk is mounted; the finalized name +and effective mount are available in `Sandbox.Volumes`. Custom mounts must be +absolute and clean, stay outside the guest OS tree, and cannot duplicate or +nest. +`Sandbox.ID`, `Sandbox.Deadline`, and `Sandbox.FromCheckpoint` (the lineage edge when branched) are exported. `Sandbox.TemplateDigest` is the exact content identity when the claim cloned a promoted template; it is empty for other sources. `Owner()` names the owning node, and `Token()` returns the per-sandbox bearer to persist with `ID` for a later `Lookup`; `Close()` releases the sandbox (releasing one already gone is not an error, and `Close` is bounded internally so it stays defer-friendly). +Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Passing +`WithVolumes` to `Checkpoint.New` returns a local error because checkpoint +branches do not support volumes in this version. + +The caller-visible constraints are deliberate: volume claims may consume a +warm VM, remain non-capturable, mount read-only, and require Cloud Hypervisor. + +Discover the fleet entries this token may use before planning a claim: + +```go +catalog, err := client.Volumes(ctx) // []sandbox.VolumeInfo +for _, volume := range catalog { + fmt.Println(volume.Name, volume.DefaultMount, volume.SizeBytes, volume.Available, volume.Nodes) +} +``` + +Discovery returns the gossiped union and holder count; availability and size +describe the connected node. Warm candidates retain normal ranking, filtered to +nodes advertising every requested name. A promoted-template claim prefers a +peer advertising both the template and every volume; when that intersection is +empty, one volume holder may self-verify access to a shared template store +before provisioning. ## Hibernating @@ -208,7 +239,8 @@ re-promoted after the node is upgraded. **On the default local-disk backend templates live on one node**, and on a cluster the parent claim may have been redirected — the returned `Template` -handle is bound to the owning node, so its `New`/`Delete` always reach it. +handle is bound to the owning node. Its `Delete` and volume-less `New` reach +that node; `New(WithVolumes(...))` may follow one volume-placement redirect. The name-based calls (`client.New("myproj:v1")`, `client.DeleteTemplate(...)` with `WithNetwork`/`WithSize` when non-default) route cluster-wide via the diff --git a/docs/security.md b/docs/security.md index 4a12c946..b1e6bbe1 100644 --- a/docs/security.md +++ b/docs/security.md @@ -85,6 +85,28 @@ or partitioned at that moment keeps its own replica branchable until Tenants are isolated at the API layer — listings filter, deletes answer 404 rather than confirming existence, and operator surfaces answer tenants 403. +## Read-only dataset volumes + +The volume catalog is an operator-owned data boundary. A volume name and its +access list must mean the same thing fleet-wide, although membership is +node-local. An empty entry `tenants` list permits every authenticated scope; a +nonempty list permits only those configured tenants, while the root token always +has access. Config load rejects an access-list name that is not a configured +tenant. Claim lookup returns byte-identical errors for an unknown and a +forbidden name, and catalog discovery filters before replying, so a tenant +cannot enumerate restricted entries by probing. Gossip carries names only. +Neither gossip, discovery, persisted claims, usage events, nor the sandbox index +exposes host image paths or access lists. + +Read-only is integrity protection for the shared image, not confidentiality. +Mounting a dataset into an egress-lane sandbox gives that sandbox an export path +to every destination its tenant egress policy permits; review the volume access +list and egress policy together. With `directio=off`, concurrent readers share +the host page cache, which improves reuse but lets one tenant's large scan evict +another's cached pages. `directio=on` is the per-volume mitigation; v1 has no +per-tenant cache quota or accounting. Operators must keep an attached image +immutable and publish a new name/path for new content. + ## Known limitations Facts to plan around, stated so the boundary is honest: diff --git a/e2e/cmd/androidsmoke/main.go b/e2e/cmd/androidsmoke/main.go index 5c2a6342..63448bd3 100644 --- a/e2e/cmd/androidsmoke/main.go +++ b/e2e/cmd/androidsmoke/main.go @@ -14,6 +14,7 @@ import ( "strings" "time" + "github.com/cocoonstack/sandbox/e2e/internal/harness" sandbox "github.com/cocoonstack/sandbox/sdk/go" ) @@ -44,17 +45,12 @@ func main() { func run(addr, token, template string) error { ctx, cancel := context.WithTimeout(context.Background(), 20*time.Minute) defer cancel() - client, err := sandbox.Connect(addr, sandbox.WithAPIToken(token)) - if err != nil { - return err - } - start := time.Now() - sb, err := client.New(ctx, template, + _, sb, err := harness.Claim(ctx, addr, token, template, sandbox.WithNetwork(sandbox.NetNone), sandbox.WithSize(sandbox.XLarge), sandbox.WithTimeout(30*time.Minute)) if err != nil { - return fmt.Errorf("claim: %w", err) + return err } defer func() { _ = sb.Close() }() fmt.Printf(" claim: android xlarge up in %.1fs (silkd probed)\n", time.Since(start).Seconds()) diff --git a/e2e/cmd/browsersmoke/main.go b/e2e/cmd/browsersmoke/main.go index 817cfd10..c4553a0d 100644 --- a/e2e/cmd/browsersmoke/main.go +++ b/e2e/cmd/browsersmoke/main.go @@ -15,6 +15,7 @@ import ( "strings" "time" + "github.com/cocoonstack/sandbox/e2e/internal/harness" sandbox "github.com/cocoonstack/sandbox/sdk/go" ) @@ -41,17 +42,12 @@ func main() { func run(addr, token, template string) error { ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute) defer cancel() - client, err := sandbox.Connect(addr, sandbox.WithAPIToken(token)) - if err != nil { - return err - } - start := time.Now() - sb, err := client.New(ctx, template, + _, sb, err := harness.Claim(ctx, addr, token, template, sandbox.WithNetwork(sandbox.NetNone), sandbox.WithSize(sandbox.Large), sandbox.WithTimeout(20*time.Minute)) if err != nil { - return fmt.Errorf("claim: %w", err) + return err } defer func() { _ = sb.Close() }() fmt.Printf(" claim: browser large up in %.1fs (silkd probed)\n", time.Since(start).Seconds()) diff --git a/e2e/cmd/rpcbench/main.go b/e2e/cmd/rpcbench/main.go index bf112299..a5c1d18d 100644 --- a/e2e/cmd/rpcbench/main.go +++ b/e2e/cmd/rpcbench/main.go @@ -125,6 +125,15 @@ func statRPC(conn net.Conn) error { return nil } +// bufferedConn reads through the handshake reader so bytes coalesced behind +// the 101 are never lost. +type bufferedConn struct { + net.Conn + r *bufio.Reader +} + +func (b bufferedConn) Read(p []byte) (int, error) { return b.r.Read(p) } + // dialAgent mirrors the SDK's hand-rolled upgrade (unexported there). func dialAgent(ctx context.Context, addr, id, token string) (net.Conn, error) { var d net.Dialer @@ -158,15 +167,6 @@ func dialAgent(ctx context.Context, addr, id, token string) (net.Conn, error) { return bufferedConn{Conn: raw, r: br}, nil } -// bufferedConn reads through the handshake reader so bytes coalesced behind -// the 101 are never lost. -type bufferedConn struct { - net.Conn - r *bufio.Reader -} - -func (b bufferedConn) Read(p []byte) (int, error) { return b.r.Read(p) } - func report(label string, samples []time.Duration) { slices.Sort(samples) pct := func(p float64) time.Duration { return samples[int(p*float64(len(samples)-1))] } diff --git a/e2e/cmd/smoke/main.go b/e2e/cmd/smoke/main.go index c9d3f5d6..b8392f14 100644 --- a/e2e/cmd/smoke/main.go +++ b/e2e/cmd/smoke/main.go @@ -574,24 +574,6 @@ func smokePortForward(ctx context.Context, sb *sandbox.Sandbox) error { return nil } -func wantHibernated(ctx context.Context, client *sandbox.Client, n int) error { - info, err := client.Info(ctx) - if err != nil { - return err - } - if info.Hibernated != n { - return fmt.Errorf("hibernated count %d, want %d", info.Hibernated, n) - } - return nil -} - -func want(got, exp string) error { - if got != exp { - return fmt.Errorf("got %q, want %q", got, exp) - } - return nil -} - // smokeLsp proves the LSP broker on hardware: the base sandbox answers a // typed not_found (no manifests), and a python-flavor sandbox serves a real // pylsp session — initialize, didOpen, hover — over the relay. @@ -650,6 +632,66 @@ func smokeLsp(ctx context.Context, client *sandbox.Client, base *sandbox.Sandbox return nil } +func smokeProcs(ctx context.Context, sb *sandbox.Sandbox) error { + pid, err := sb.Spawn(ctx, sandbox.Cmd{Argv: []string{"sh", "-c", "echo bg-mark; sleep 0.3; echo late"}}) + if err != nil { + return fmt.Errorf("spawn: %w", err) + } + procs, err := sb.Ps(ctx) + if err != nil { + return fmt.Errorf("ps: %w", err) + } + if !slices.ContainsFunc(procs, func(p wire.ProcInfo) bool { return p.PID == pid && p.Detached }) { + return fmt.Errorf("ps does not list spawned pid %d: %+v", pid, procs) + } + var out bytes.Buffer + code, exited, err := sb.Attach(ctx, pid, &out, nil) + if err != nil || !exited || code != 0 { + return fmt.Errorf("attach: code=%d exited=%v, %v", code, exited, err) + } + if !strings.Contains(out.String(), "bg-mark") || !strings.Contains(out.String(), "late") { + return fmt.Errorf("attach output %q missing marks", out.String()) + } + out.Reset() + if code, exited, err = sb.Logs(ctx, pid, &out, nil); err != nil || !exited || code != 0 { + return fmt.Errorf("logs after exit: code=%d exited=%v, %v", code, exited, err) + } + if !strings.Contains(out.String(), "bg-mark") { + return fmt.Errorf("logs replay %q missing mark", out.String()) + } + + // A long-runner dies to kill; its pid must be gone from the next attach. + pid, err = sb.Spawn(ctx, sandbox.Cmd{Argv: []string{"sleep", "30"}}) + if err != nil { + return fmt.Errorf("spawn sleeper: %w", err) + } + if err = sb.Kill(ctx, pid, 0); err != nil { + return fmt.Errorf("kill: %w", err) + } + if code, exited, err = sb.Attach(ctx, pid, nil, nil); err != nil || !exited || code == 0 { + return fmt.Errorf("attach killed: code=%d exited=%v, %v (want non-zero exit)", code, exited, err) + } + return nil +} + +func wantHibernated(ctx context.Context, client *sandbox.Client, n int) error { + info, err := client.Info(ctx) + if err != nil { + return err + } + if info.Hibernated != n { + return fmt.Errorf("hibernated count %d, want %d", info.Hibernated, n) + } + return nil +} + +func want(got, exp string) error { + if got != exp { + return fmt.Errorf("got %q, want %q", got, exp) + } + return nil +} + func isSilkdKind(err error, kind string) bool { var er *wire.ErrorResp return errors.As(err, &er) && er.Kind == kind @@ -706,47 +748,3 @@ func lspReadResponse(r *bufio.Reader, id int) (json.RawMessage, error) { return msg.Result, nil } } - -// smokeProcs drives the detached-process surface: spawn, ps, logs replay, -// attach-until-exit, and kill. -func smokeProcs(ctx context.Context, sb *sandbox.Sandbox) error { - pid, err := sb.Spawn(ctx, sandbox.Cmd{Argv: []string{"sh", "-c", "echo bg-mark; sleep 0.3; echo late"}}) - if err != nil { - return fmt.Errorf("spawn: %w", err) - } - procs, err := sb.Ps(ctx) - if err != nil { - return fmt.Errorf("ps: %w", err) - } - if !slices.ContainsFunc(procs, func(p wire.ProcInfo) bool { return p.PID == pid && p.Detached }) { - return fmt.Errorf("ps does not list spawned pid %d: %+v", pid, procs) - } - var out bytes.Buffer - code, exited, err := sb.Attach(ctx, pid, &out, nil) - if err != nil || !exited || code != 0 { - return fmt.Errorf("attach: code=%d exited=%v, %v", code, exited, err) - } - if !strings.Contains(out.String(), "bg-mark") || !strings.Contains(out.String(), "late") { - return fmt.Errorf("attach output %q missing marks", out.String()) - } - out.Reset() - if code, exited, err = sb.Logs(ctx, pid, &out, nil); err != nil || !exited || code != 0 { - return fmt.Errorf("logs after exit: code=%d exited=%v, %v", code, exited, err) - } - if !strings.Contains(out.String(), "bg-mark") { - return fmt.Errorf("logs replay %q missing mark", out.String()) - } - - // A long-runner dies to kill; its pid must be gone from the next attach. - pid, err = sb.Spawn(ctx, sandbox.Cmd{Argv: []string{"sleep", "30"}}) - if err != nil { - return fmt.Errorf("spawn sleeper: %w", err) - } - if err = sb.Kill(ctx, pid, 0); err != nil { - return fmt.Errorf("kill: %w", err) - } - if code, exited, err = sb.Attach(ctx, pid, nil, nil); err != nil || !exited || code == 0 { - return fmt.Errorf("attach killed: code=%d exited=%v, %v (want non-zero exit)", code, exited, err) - } - return nil -} diff --git a/e2e/cmd/volumesmoke/main.go b/e2e/cmd/volumesmoke/main.go new file mode 100644 index 00000000..fe450f83 --- /dev/null +++ b/e2e/cmd/volumesmoke/main.go @@ -0,0 +1,120 @@ +// volumesmoke validates shared read-only dataset mounts on a live Cloud Hypervisor node. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "os" + "path" + "slices" + "strings" + "sync" + "time" + + "github.com/cocoonstack/sandbox/e2e/internal/harness" + sandbox "github.com/cocoonstack/sandbox/sdk/go" +) + +func main() { + addr := flag.String("addr", "127.0.0.1:7777", "sandboxd address") + token := flag.String("token", "", "node api token") + template := flag.String("template", "rt:24.04", "template ref") + volume := flag.String("volume", "", "catalog volume name") + probe := flag.String("probe", "volume-e2e.txt", "non-empty file inside the volume") + flag.Parse() + + if err := run(*addr, *token, *template, *volume, *probe); err != nil { + fmt.Fprintln(os.Stderr, "volumesmoke:", err) + os.Exit(1) + } +} + +func run(addr, token, template, volume, probe string) error { + if volume == "" { + return errors.New("volume is required") + } + if probe == "" || probe == "." || path.IsAbs(probe) || path.Clean(probe) != probe || probe == ".." || strings.HasPrefix(probe, "../") { + return fmt.Errorf("probe %q must be a clean relative file path", probe) + } + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + + const ( + mountA = "/datasets/e2e-a" + mountB = "/datasets/e2e-b" + ) + start := time.Now() + client, sbA, err := harness.Claim(ctx, addr, token, template, + sandbox.WithNetwork(sandbox.NetNone), + sandbox.WithVolumes(sandbox.Volume{Name: volume, Mount: mountA}), + ) + claimA := time.Since(start) + if err != nil { + return err + } + start = time.Now() + sbB, err := client.New(ctx, template, + sandbox.WithNetwork(sandbox.NetNone), + sandbox.WithVolumes(sandbox.Volume{Name: volume, Mount: mountB}), + ) + claimB := time.Since(start) + if err != nil { + _ = sbA.Close() + return fmt.Errorf("claim at %s: %w", mountB, err) + } + released := false + defer func() { + if !released { + _ = sbA.Close() + _ = sbB.Close() + } + }() + if want := []sandbox.Volume{{Name: volume, Mount: mountA}}; !slices.Equal(sbA.Volumes, want) { + return fmt.Errorf("first claim volumes %+v, want %+v", sbA.Volumes, want) + } + if want := []sandbox.Volume{{Name: volume, Mount: mountB}}; !slices.Equal(sbB.Volumes, want) { + return fmt.Errorf("second claim volumes %+v, want %+v", sbB.Volumes, want) + } + fmt.Printf("volume=%s claim_a=%.1fms claim_b=%.1fms\n", volume, float64(claimA.Microseconds())/1000, float64(claimB.Microseconds())/1000) + + sandboxes := []*sandbox.Sandbox{sbA, sbB} + mounts := []string{mountA, mountB} + var ( + wg sync.WaitGroup + outputs [2]string + errs [2]error + ) + startReads := make(chan struct{}) + for i := range sandboxes { + wg.Go(func() { + <-startReads + outputs[i], errs[i] = sandboxes[i].Exec(ctx, "cat", path.Join(mounts[i], probe)) + }) + } + close(startReads) + wg.Wait() + if readErr := errors.Join(errs[:]...); readErr != nil { + return fmt.Errorf("concurrent volume reads: %w", readErr) + } + if outputs[0] == "" || outputs[0] != outputs[1] { + return fmt.Errorf("concurrent reads differ: first_bytes=%d second_bytes=%d", len(outputs[0]), len(outputs[1])) + } + + _, err = sbA.Exec(ctx, "touch", path.Join(mountA, ".sandboxd-write-probe")) + var exitErr *sandbox.ExitError + if err == nil { + return errors.New("write to read-only volume succeeded") + } + if !errors.As(err, &exitErr) || !strings.Contains(strings.ToLower(exitErr.Stderr), "read-only file system") { + return fmt.Errorf("write failed without EROFS: %w", err) + } + + if err := errors.Join(sbA.Close(), sbB.Close()); err != nil { + return fmt.Errorf("release volume claims: %w", err) + } + released = true + fmt.Printf("VOLUME PASS concurrent_read_bytes=%d read_only=true released=true\n", len(outputs[0])) + return nil +} diff --git a/e2e/e2e_test.go b/e2e/e2e_test.go index c0de464e..5db58214 100644 --- a/e2e/e2e_test.go +++ b/e2e/e2e_test.go @@ -7,6 +7,7 @@ package e2e import ( "net/http/httptest" "os" + "path/filepath" "slices" "strings" "testing" @@ -20,7 +21,7 @@ import ( sandbox "github.com/cocoonstack/sandbox/sdk/go" ) -var testKey = types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall} +var testKey = types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall, Engine: types.EngineCH} func TestEndToEnd(t *testing.T) { stack := startStack(t, "node-token", config.PoolSpec{PoolKey: testKey, Warm: 1}) @@ -207,7 +208,7 @@ func TestTwoTenantFlow(t *testing.T) { {Name: "acme", Token: "acme-tok", MaxClaims: 1}, {Name: "beta", Token: "beta-tok"}, } - stack := startTenantStack(t, "node-token", tenants) + stack := startTenantStack(t, "node-token", tenants, nil) acme, err := sandbox.Connect(stack.addr, sandbox.WithAPIToken("acme-tok")) if err != nil { t.Fatalf("connect acme: %v", err) @@ -282,6 +283,47 @@ func TestWrongAPITokenRejected(t *testing.T) { } } +func TestVolumesEndToEnd(t *testing.T) { + image := filepath.Join(t.TempDir(), "dataset.img") + if err := os.WriteFile(image, []byte("dataset-bytes"), 0o600); err != nil { + t.Fatalf("write image: %v", err) + } + stack := startTenantStack(t, "node-token", nil, + []config.VolumeSpec{{Name: "dataset", Path: image, DirectIO: "off"}}, + config.PoolSpec{PoolKey: testKey, Warm: 1}) + waitFor(t, func() bool { + infos, _ := stack.mgr.Info() + return len(infos) == 1 && infos[0].Warm >= 1 + }) + warmBefore := stack.mgr.Counters().ClaimsWarm + + sb, err := stack.client.New(t.Context(), "rt:24.04", + sandbox.WithVolumes(sandbox.Volume{Name: "dataset", Mount: "/datasets/e2e"})) + if err != nil { + t.Fatalf("volume claim: %v", err) + } + defer sb.Close() + if want := []sandbox.Volume{{Name: "dataset", Mount: "/datasets/e2e"}}; !slices.Equal(sb.Volumes, want) { + t.Errorf("claim volumes %+v, want %+v", sb.Volumes, want) + } + if counters := stack.mgr.Counters(); counters.ClaimsWarm != warmBefore+1 { + infos, _ := stack.mgr.Info() + t.Errorf("counters=%+v pools=%+v, want warm claims %d", counters, infos, warmBefore+1) + } + + infos, err := stack.client.Volumes(t.Context()) + if err != nil { + t.Fatalf("Volumes: %v", err) + } + want := []sandbox.VolumeInfo{{ + Name: "dataset", DefaultMount: "/volumes/dataset", + SizeBytes: int64(len("dataset-bytes")), Available: true, Nodes: 1, + }} + if !slices.Equal(infos, want) { + t.Errorf("catalog %+v, want %+v", infos, want) + } +} + type stack struct { client *sandbox.Client mgr *pool.Manager @@ -290,10 +332,10 @@ type stack struct { func startStack(t *testing.T, apiToken string, pools ...config.PoolSpec) *stack { t.Helper() - return startTenantStack(t, apiToken, nil, pools...) + return startTenantStack(t, apiToken, nil, nil, pools...) } -func startTenantStack(t *testing.T, apiToken string, tenants []config.TenantSpec, pools ...config.PoolSpec) *stack { +func startTenantStack(t *testing.T, apiToken string, tenants []config.TenantSpec, volumes []config.VolumeSpec, pools ...config.PoolSpec) *stack { t.Helper() // Short prefix: the sockets under it must fit darwin's 104-byte sun_path. dir, err := os.MkdirTemp("", "sbx") @@ -307,7 +349,7 @@ func startTenantStack(t *testing.T, apiToken string, tenants []config.TenantSpec if err != nil { t.Fatalf("secrets: %v", err) } - mgr, err := pool.NewManager(t.Context(), &config.Config{DataDir: dir, Pools: pools, Tenants: tenants}, eng, secrets) + mgr, err := pool.NewManager(t.Context(), &config.Config{DataDir: dir, Pools: pools, Tenants: tenants, Volumes: volumes}, eng, secrets) if err != nil { t.Fatalf("setup manager: %v", err) } diff --git a/e2e/fakeengine_test.go b/e2e/fakeengine_test.go index 48235969..da81291a 100644 --- a/e2e/fakeengine_test.go +++ b/e2e/fakeengine_test.go @@ -109,6 +109,10 @@ func (f *fakeEngine) DialGuestPort(context.Context, string, uint16) (net.Conn, e func (f *fakeEngine) InstallCACert(context.Context, string, []byte) error { return nil } +func (f *fakeEngine) DiskAttach(context.Context, string, engine.VolumeSpec) error { return nil } + +func (f *fakeEngine) MountVolume(context.Context, string, string, string) error { return nil } + func (f *fakeEngine) create(name string) (string, error) { f.mu.Lock() defer f.mu.Unlock() diff --git a/protocol/wire/frame.go b/protocol/wire/frame.go index d223ebe7..dbcd7a5e 100644 --- a/protocol/wire/frame.go +++ b/protocol/wire/frame.go @@ -98,7 +98,7 @@ var ( // responseDecoders maps each type tag to a decoder. Two-stage dispatch is // required because the same key differs in shape across variants (info's // procs is a count, ps's procs is a list). - responseDecoders = map[string]func([]byte) (Response, error){ + responseDecoders = map[string]respDecoder{ "started": decodeResp[Started], "stdout": fastBulk("stdout", decodeResp[Stdout], func(d []byte) Response { return &Stdout{Data: d} }), "stderr": fastBulk("stderr", decodeResp[Stderr], func(d []byte) Response { return &Stderr{Data: d} }), @@ -129,6 +129,8 @@ type Request interface{ Op() string } // Response is a server→client frame; RespType is its wire tag. type Response interface{ RespType() string } +type respDecoder func([]byte) (Response, error) + // B64 carries request payload bytes. It exists because silkd's deserializer // requires a base64 string and rejects null — which is exactly what // encoding/json emits for a nil []byte. Decoding needs no counterpart: @@ -694,7 +696,7 @@ func AppendBulkRequest(buf []byte, op string, data []byte) []byte { // fastBulk slices the base64 data out of a canonical bulk frame, skipping the // json.Unmarshal that dominates downloads; any other shape falls back to slow. -func fastBulk(tag string, slow func([]byte) (Response, error), mk func([]byte) Response) func([]byte) (Response, error) { +func fastBulk(tag string, slow respDecoder, mk func([]byte) Response) respDecoder { head := []byte(`{"type":"` + tag + `","data":"`) return func(line []byte) (Response, error) { after, ok := bytes.CutPrefix(line, head) diff --git a/sandboxd/config/config.go b/sandboxd/config/config.go index dc5da56a..61addf89 100644 --- a/sandboxd/config/config.go +++ b/sandboxd/config/config.go @@ -12,6 +12,7 @@ import ( "net" "net/netip" "os" + "path/filepath" "runtime" "slices" "strconv" @@ -111,13 +112,18 @@ type TenantSpec struct { Egress *egress.Policy `json:"egress,omitempty"` } -// MeshConfig configures cluster membership. Two v1 constraints: all nodes -// must share the same APIToken AND the same Tenants set (the SDK replays -// whichever token authorized a claim across a redirect, so a peer missing -// that tenant answers 401), and a -// node serving the egress lane can only redirect egress claims to peers if it -// too has an egress attachment (a no-egress node answers 409 rather than -// redirecting). +// VolumeSpec declares one operator-managed read-only dataset disk. +type VolumeSpec struct { + Name string `json:"name"` + Path string `json:"path"` + DirectIO string `json:"directio,omitempty"` + Tenants []string `json:"tenants,omitempty"` +} + +// MeshConfig configures cluster membership. Two v1 constraints: every node +// shares one APIToken and Tenants set (a redirect replays the claim's token, +// so a peer missing that tenant answers 401), and only a node with its own +// egress attachment can take a redirected egress claim. type MeshConfig struct { NodeID string `json:"node_id"` // unique name; defaults to Bind Bind string `json:"bind"` // memberlist host:port @@ -258,6 +264,9 @@ type Config struct { // Defaults to 16. MaxForkCount int `json:"max_fork_count,omitempty"` + // Volumes is the node-local catalog of operator-managed dataset images. + Volumes []VolumeSpec `json:"volumes,omitempty"` + // RefillConcurrency caps concurrent VM provisioning node-wide — warm-pool // refills, fork clones, and reap/hibernate/reconcile batches share one // budget; 0 auto-scales with the node's CPU count. @@ -329,6 +338,9 @@ func (c *Config) applyDefaults() { c.Pools[i].Warm = cmp.Or(c.Pools[i].Warm, defaultWarm) c.Pools[i].PoolKey = c.Pools[i].Defaulted() } + for i := range c.Volumes { + c.Volumes[i].DirectIO = cmp.Or(c.Volumes[i].DirectIO, types.DirectIOOff) + } } func (c *Config) validate() error { @@ -390,6 +402,9 @@ func (c *Config) validate() error { if err := c.validateTenants(); err != nil { return err } + if err := c.validateVolumes(); err != nil { + return err + } if err := c.validateMesh(); err != nil { return err } @@ -400,6 +415,35 @@ func (c *Config) validate() error { return c.validateEgress(secrets) } +func (c *Config) validateVolumes() error { + names := make(map[string]struct{}, len(c.Volumes)) + tenants := make(map[string]struct{}, len(c.Tenants)) + for _, tenant := range c.Tenants { + tenants[tenant.Name] = struct{}{} + } + for _, volume := range c.Volumes { + if !types.ValidVolumeName(volume.Name) { + return fmt.Errorf("volume name %q must match %s and not start with cocoon-", volume.Name, types.VolumeNameRe) + } + if _, ok := names[volume.Name]; ok { + return fmt.Errorf("duplicate volume name %q", volume.Name) + } + names[volume.Name] = struct{}{} + if !filepath.IsAbs(volume.Path) { + return fmt.Errorf("volume %q path must be absolute", volume.Name) + } + if !types.ValidDirectIO(volume.DirectIO) { + return fmt.Errorf("volume %q directio must be on, off, or auto, got %q", volume.Name, volume.DirectIO) + } + for _, tenant := range volume.Tenants { + if _, ok := tenants[tenant]; !ok { + return fmt.Errorf("volume %q references unknown tenant %q", volume.Name, tenant) + } + } + } + return nil +} + // validateMesh fails at load what would otherwise only surface at startMesh. func (c *Config) validateMesh() error { if c.Mesh == nil { diff --git a/sandboxd/config/config_test.go b/sandboxd/config/config_test.go index f9ea8c22..63242397 100644 --- a/sandboxd/config/config_test.go +++ b/sandboxd/config/config_test.go @@ -4,6 +4,7 @@ import ( "encoding/base64" "os" "path/filepath" + "slices" "strings" "testing" @@ -32,6 +33,11 @@ func TestClusterDigest(t *testing.T) { if keyed.ClusterDigest("ca-fp") == keyedDiff.ClusterDigest("ca-fp") { t.Error("with a cluster_key the api_token must be covered by the digest") } + withVolume := *base + withVolume.Volumes = []VolumeSpec{{Name: "imagenet", Path: "/srv/datasets/imagenet.img", DirectIO: types.DirectIOOff}} + if withVolume.ClusterDigest("ca-fp") != d { + t.Error("node-local volume catalog must not change the cluster digest") + } } func TestLoadAppliesDefaults(t *testing.T) { @@ -124,6 +130,12 @@ func TestLoadRejectsInvalid(t *testing.T) { {"checkpoint peer heal without cluster key", `{"checkpoint_peer_heal":true,"pools":[],"mesh":{"bind":"node1:7946"}}`, "requires an encrypted mesh"}, {"checkpoint peer heal without ttl", `{"checkpoint_peer_heal":true,"pools":[],"mesh":{"bind":"node1:7946","cluster_key":"MDEyMzQ1Njc4OWFiY2RlZg=="}}`, "requires checkpoint_ttl_hours"}, {"checkpoint peer heal without api_token", `{"checkpoint_peer_heal":true,"pools":[],"mesh":{"bind":"node1:7946","cluster_key":"MDEyMzQ1Njc4OWFiY2RlZg=="},"checkpoint_ttl_hours":1}`, "requires api_token"}, + {"bad volume name", `{"pools":[],"volumes":[{"name":"ImageNet","path":"/srv/datasets/a.img"}]}`, "volume name"}, + {"reserved volume name", `{"pools":[],"volumes":[{"name":"cocoon-data","path":"/srv/datasets/a.img"}]}`, "not start with cocoon-"}, + {"duplicate volume name", `{"pools":[],"volumes":[{"name":"data","path":"/srv/datasets/a.img"},{"name":"data","path":"/srv/datasets/b.img"}]}`, "duplicate volume"}, + {"relative volume path", `{"pools":[],"volumes":[{"name":"data","path":"datasets/a.img"}]}`, "path must be absolute"}, + {"bad volume directio", `{"pools":[],"volumes":[{"name":"data","path":"/srv/datasets/a.img","directio":"yes"}]}`, "directio must be"}, + {"unknown volume tenant", `{"api_token":"root","pools":[],"tenants":[{"name":"beta","token":"b"}],"volumes":[{"name":"data","path":"/srv/datasets/a.img","tenants":["acme"]}]}`, `volume "data" references unknown tenant "acme"`}, } { t.Run(tt.name, func(t *testing.T) { _, err := Load(writeConfig(t, tt.body)) @@ -134,6 +146,33 @@ func TestLoadRejectsInvalid(t *testing.T) { } } +func TestLoadAcceptsVolumes(t *testing.T) { + path := writeConfig(t, `{"pools":[],"volumes":[ + {"name":"imagenet","path":"/srv/datasets/imagenet.img"}, + {"name":"weights-llama","path":"/srv/datasets/llama.img","directio":"on"}]}`) + cfg, err := Load(path) + if err != nil { + t.Fatalf("Load: %v", err) + } + if len(cfg.Volumes) != 2 || cfg.Volumes[0].DirectIO != types.DirectIOOff || + cfg.Volumes[1].DirectIO != types.DirectIOOn { + t.Errorf("volumes = %+v", cfg.Volumes) + } +} + +func TestLoadAcceptsVolumeTenantAccessList(t *testing.T) { + path := writeConfig(t, `{"api_token":"root","pools":[], + "tenants":[{"name":"acme","token":"a"},{"name":"beta","token":"b"}], + "volumes":[{"name":"corpus","path":"/srv/datasets/corpus.img","tenants":["acme"]}]}`) + cfg, err := Load(path) + if err != nil { + t.Fatalf("Load: %v", err) + } + if got := cfg.Volumes[0].Tenants; !slices.Equal(got, []string{"acme"}) { + t.Errorf("volume tenants = %v, want [acme]", got) + } +} + func TestLoadAcceptsTenants(t *testing.T) { path := writeConfig(t, `{"api_token":"root","pools":[], "tenants":[{"name":"acme","token":"t1","max_claims":50},{"name":"beta","token":"t2"}]}`) diff --git a/sandboxd/egress/bench_test.go b/sandboxd/egress/bench_test.go index 0c9b38b6..6ac3a146 100644 --- a/sandboxd/egress/bench_test.go +++ b/sandboxd/egress/bench_test.go @@ -135,7 +135,7 @@ func benchSession(b *testing.B, proxyAddr string, roots *x509.CertPool) *tls.Con func benchRoundTrip(b *testing.B, tc *tls.Conn, br *bufio.Reader) { b.Helper() - req, err := http.NewRequest(http.MethodGet, "https://example.com/x", nil) + req, err := http.NewRequestWithContext(b.Context(), http.MethodGet, "https://example.com/x", nil) if err != nil { b.Fatalf("build request: %v", err) } diff --git a/sandboxd/egress/proxy.go b/sandboxd/egress/proxy.go index b18a177b..b96142eb 100644 --- a/sandboxd/egress/proxy.go +++ b/sandboxd/egress/proxy.go @@ -12,6 +12,8 @@ import ( "slices" "strings" "sync" + + "github.com/cocoonstack/sandbox/sandboxd/utils" ) // hopHeaders are hop-by-hop and proxy-scoped headers this hop owns: stripped @@ -258,19 +260,13 @@ func splice(a, b net.Conn) { go func() { defer close(done) _, _ = io.Copy(a, b) - closeWrite(a) + utils.CloseWrite(a) }() _, _ = io.Copy(b, a) - closeWrite(b) + utils.CloseWrite(b) <-done } -func closeWrite(conn net.Conn) { - if cw, ok := conn.(interface{ CloseWrite() error }); ok { - _ = cw.CloseWrite() - } -} - // denied answers a policy rejection with a typed 403 the guest's client // surfaces as a distinct failure rather than a hang. func denied(w http.ResponseWriter, host string) { diff --git a/sandboxd/egress/secrets.go b/sandboxd/egress/secrets.go index 58b9e2b0..deb43157 100644 --- a/sandboxd/egress/secrets.go +++ b/sandboxd/egress/secrets.go @@ -46,6 +46,11 @@ func (s SecretSpec) Validate() error { return nil } +type resolvedSecret struct { + header string + value string +} + // SecretStore is the resolved node-side credential registry, implementing the // Proxy's Secrets interface. Values live only here — never in a policy or a // gossiped struct. @@ -75,8 +80,3 @@ func (s *SecretStore) Header(name string) (header, value string, ok bool) { r, ok := s.byName[name] return r.header, r.value, ok } - -type resolvedSecret struct { - header string - value string -} diff --git a/sandboxd/engine/installca_test.go b/sandboxd/engine/installca_test.go index 7ab237f5..82a0ecda 100644 --- a/sandboxd/engine/installca_test.go +++ b/sandboxd/engine/installca_test.go @@ -9,6 +9,8 @@ import ( "strings" "sync" "testing" + + "github.com/cocoonstack/sandbox/protocol/wire" ) func TestInstallCACertWritesCertAndUpdates(t *testing.T) { @@ -61,15 +63,24 @@ func TestInstallCACertWriteErrorFrameFails(t *testing.T) { } type fakeSilkd struct { - mu sync.Mutex - writePath string - writeMode uint32 - writeData []byte - execArgv []string - execEnv map[string]string - execCode int32 - writeErr string - execErr string + mu sync.Mutex + writePath string + writeMode uint32 + writeData []byte + execArgv []string + execCalls [][]string + execEnv map[string]string + execCode int32 + execFailAt int + writeErr string + execErr string + block []wire.DirEntry + serial map[string][]byte + readMisses map[string]int + listErr string + readErr map[string]string + listCalls int + readCalls []string } func serveFakeSilkd(t *testing.T, path string) *fakeSilkd { @@ -79,7 +90,11 @@ func serveFakeSilkd(t *testing.T, path string) *fakeSilkd { t.Fatalf("listen: %v", err) } t.Cleanup(func() { _ = ln.Close() }) - f := &fakeSilkd{} + f := &fakeSilkd{ + serial: make(map[string][]byte), + readMisses: make(map[string]int), + readErr: make(map[string]string), + } go func() { for { conn, err := ln.Accept() @@ -118,6 +133,10 @@ func (f *fakeSilkd) serve(conn net.Conn) { switch req.Op { case "fs_write": f.handleWrite(conn, r, req.Path, req.Mode) + case "fs_list": + f.handleList(conn, req.Path) + case "fs_read": + f.handleRead(conn, req.Path) case "exec": f.handleExec(conn, req.Argv, req.Env) } @@ -156,8 +175,12 @@ func (f *fakeSilkd) handleWrite(conn net.Conn, r *bufio.Reader, path string, mod func (f *fakeSilkd) handleExec(conn net.Conn, argv []string, env map[string]string) { f.mu.Lock() f.execArgv = argv + f.execCalls = append(f.execCalls, argv) f.execEnv = env code, reply := f.execCode, f.execErr + if f.execFailAt > 0 && len(f.execCalls) != f.execFailAt { + code, reply = 0, "" + } f.mu.Unlock() if reply != "" { _, _ = io.WriteString(conn, `{"type":"error","kind":"internal","message":"`+reply+`"}`+"\n") @@ -166,3 +189,51 @@ func (f *fakeSilkd) handleExec(conn net.Conn, argv []string, env map[string]stri _, _ = io.WriteString(conn, `{"type":"started","pid":1}`+"\n") _, _ = fmt.Fprintf(conn, `{"type":"exit","code":%d}`+"\n", code) } + +func (f *fakeSilkd) handleList(conn net.Conn, path string) { + f.mu.Lock() + f.listCalls++ + entries, reply := append([]wire.DirEntry(nil), f.block...), f.listErr + f.mu.Unlock() + if reply != "" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) + return + } + if path != "/sys/block" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindNotFound, Message: path}) + return + } + writeFakeSilkdResponse(conn, &wire.Entries{Entries: entries}) + writeFakeSilkdResponse(conn, &wire.Done{}) +} + +func (f *fakeSilkd) handleRead(conn net.Conn, path string) { + f.mu.Lock() + f.readCalls = append(f.readCalls, path) + data, ok := f.serial[path] + reply := f.readErr[path] + if f.readMisses[path] > 0 { + f.readMisses[path]-- + ok = false + } + data = append([]byte(nil), data...) + f.mu.Unlock() + if reply != "" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) + return + } + if !ok { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindNotFound, Message: path}) + return + } + writeFakeSilkdResponse(conn, &wire.DataResp{Data: data}) + writeFakeSilkdResponse(conn, &wire.Done{}) +} + +func writeFakeSilkdResponse(conn net.Conn, resp wire.Response) { + buf, err := wire.EncodeResponse(resp) + if err != nil { + return + } + _, _ = conn.Write(append(buf, '\n')) +} diff --git a/sandboxd/engine/silkd.go b/sandboxd/engine/silkd.go index 3c8d275c..25174eee 100644 --- a/sandboxd/engine/silkd.go +++ b/sandboxd/engine/silkd.go @@ -20,18 +20,6 @@ type silkdSession struct { stop func() bool } -func (e *Engine) dialSilkdSession(ctx context.Context, vsockSocket string) (*silkdSession, error) { - conn, err := e.DialSilkd(ctx, vsockSocket) - if err != nil { - return nil, err - } - return &silkdSession{ - conn: conn, - sc: wire.NewFrameScanner(conn), - stop: context.AfterFunc(ctx, func() { _ = conn.Close() }), - }, nil -} - func (s *silkdSession) send(req wire.Request) error { buf, err := wire.EncodeRequest(req) if err != nil { @@ -57,3 +45,76 @@ func (s *silkdSession) close() { s.stop() _ = s.conn.Close() } + +func (e *Engine) dialSilkdSession(ctx context.Context, vsockSocket string) (*silkdSession, error) { + conn, err := e.DialSilkd(ctx, vsockSocket) + if err != nil { + return nil, err + } + return &silkdSession{ + conn: conn, + sc: wire.NewFrameScanner(conn), + stop: context.AfterFunc(ctx, func() { _ = conn.Close() }), + }, nil +} + +// silkdStream serves one request per dial: silkd handles a single request +// per connection. +func (e *Engine) silkdStream(ctx context.Context, vsockSocket string, req wire.Request, onFrame func(wire.Response) error) error { + s, err := e.dialSilkdSession(ctx, vsockSocket) + if err != nil { + return err + } + defer s.close() + if err := s.send(req); err != nil { + return err + } + for { + frame, err := s.recv() + if err != nil { + return err + } + switch resp := frame.(type) { + case *wire.Done: + return nil + case *wire.ErrorResp: + return fmt.Errorf("silkd %w", resp) + default: + if err := onFrame(resp); err != nil { + return err + } + } + } +} + +func (e *Engine) silkdList(ctx context.Context, vsockSocket, path string) ([]wire.DirEntry, error) { + var entries []wire.DirEntry + err := e.silkdStream(ctx, vsockSocket, wire.FsList{Path: path}, func(frame wire.Response) error { + resp, ok := frame.(*wire.Entries) + if !ok { + return fmt.Errorf("unexpected silkd frame %q", frame.RespType()) + } + entries = append(entries, resp.Entries...) + return nil + }) + if err != nil { + return nil, err + } + return entries, nil +} + +func (e *Engine) silkdReadFile(ctx context.Context, vsockSocket, path string) ([]byte, error) { + var data []byte + err := e.silkdStream(ctx, vsockSocket, wire.FsRead{Path: path}, func(frame wire.Response) error { + resp, ok := frame.(*wire.DataResp) + if !ok { + return fmt.Errorf("unexpected silkd frame %q", frame.RespType()) + } + data = append(data, resp.Data...) + return nil + }) + if err != nil { + return nil, err + } + return data, nil +} diff --git a/sandboxd/engine/volume.go b/sandboxd/engine/volume.go new file mode 100644 index 00000000..6d520b01 --- /dev/null +++ b/sandboxd/engine/volume.go @@ -0,0 +1,117 @@ +package engine + +import ( + "cmp" + "context" + "errors" + "fmt" + "strings" + "time" + + "github.com/cocoonstack/sandbox/protocol/wire" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +const ( + volumePollInterval = 10 * time.Millisecond + volumeProbeTimeout = 2 * time.Second + volumeSetupTimeout = 10 * time.Second +) + +// VolumeSpec describes one operator-owned disk image attached to a sandbox. +type VolumeSpec struct { + Name string + Path string + DirectIO string +} + +func (s VolumeSpec) directIO() (string, error) { + mode := cmp.Or(s.DirectIO, types.DirectIOOff) + if !types.ValidDirectIO(mode) { + return "", fmt.Errorf("volume directio must be on, off, or auto, got %q", s.DirectIO) + } + return mode, nil +} + +// DiskAttach hot-attaches an operator-owned disk read-only through cocoon. +func (e *Engine) DiskAttach(ctx context.Context, vmName string, spec VolumeSpec) error { + args, err := e.diskAttachArgs(vmName, spec) + if err != nil { + return err + } + _, err = e.run(ctx, args...) + return err +} + +// MountVolume discovers and mounts a hot-attached disk read-only at mount. +func (e *Engine) MountVolume(ctx context.Context, vsockSocket, name, mount string) error { + ctx, cancel := context.WithTimeout(ctx, volumeSetupTimeout) + defer cancel() + device, err := e.waitForVolumeDevice(ctx, vsockSocket, name) + if err != nil { + return fmt.Errorf("wait for volume device %s: %w", name, err) + } + if err := e.silkdExec(ctx, vsockSocket, "mkdir", "-p", "--", mount); err != nil { + return fmt.Errorf("create volume mount point %s: %w", mount, err) + } + if err := e.silkdExec(ctx, vsockSocket, "mount", "-o", "ro", "--", device, mount); err != nil { + return fmt.Errorf("mount volume %s: %w", name, err) + } + return nil +} + +func (e *Engine) waitForVolumeDevice(ctx context.Context, vsockSocket, name string) (string, error) { + ctx, cancel := context.WithTimeout(ctx, volumeProbeTimeout) + defer cancel() + ticker := time.NewTicker(volumePollInterval) + defer ticker.Stop() + for { + device, found, err := e.findVolumeDevice(ctx, vsockSocket, name) + if err != nil { + return "", cmp.Or(ctx.Err(), err) + } + if found { + return device, nil + } + select { + case <-ctx.Done(): + return "", ctx.Err() + case <-ticker.C: + } + } +} + +func (e *Engine) findVolumeDevice(ctx context.Context, vsockSocket, name string) (string, bool, error) { + entries, err := e.silkdList(ctx, vsockSocket, "/sys/block") + if err != nil { + return "", false, err + } + for _, entry := range entries { + serial, err := e.silkdReadFile(ctx, vsockSocket, "/sys/block/"+entry.Name+"/serial") + var respErr *wire.ErrorResp + if errors.As(err, &respErr) && respErr.Kind == wire.KindNotFound { + continue + } + if err != nil { + return "", false, err + } + if strings.TrimSpace(string(serial)) == name { + return "/dev/" + entry.Name, true, nil + } + } + return "", false, nil +} + +func (e *Engine) diskAttachArgs(vmName string, spec VolumeSpec) ([]string, error) { + directIO, err := spec.directIO() + if err != nil { + return nil, err + } + return []string{ + "vm", "disk", "attach", vmName, + "--path", spec.Path, + argName, spec.Name, + "--readonly", + "--directio", directIO, + }, nil +} diff --git a/sandboxd/engine/volume_test.go b/sandboxd/engine/volume_test.go new file mode 100644 index 00000000..2ad84f98 --- /dev/null +++ b/sandboxd/engine/volume_test.go @@ -0,0 +1,169 @@ +package engine + +import ( + "context" + "errors" + "slices" + "strings" + "testing" + "time" + + "github.com/cocoonstack/sandbox/protocol/wire" +) + +func TestDiskAttachArgsReadOnlyAndDirectIO(t *testing.T) { + for _, tt := range []struct { + name string + directIO string + wantIO string + }{ + {"default buffered", "", "off"}, + {"direct", "on", "on"}, + {"auto", "auto", "auto"}, + } { + t.Run(tt.name, func(t *testing.T) { + e := New("cocoon", nil, nil, false, "") + args, err := e.diskAttachArgs("sbx-1", VolumeSpec{ + Name: "imagenet", Path: "/srv/datasets/imagenet.img", DirectIO: tt.directIO, + }) + if err != nil { + t.Fatalf("diskAttachArgs: %v", err) + } + want := []string{ + "vm", "disk", "attach", "sbx-1", + "--path", "/srv/datasets/imagenet.img", + "--name", "imagenet", + "--readonly", + "--directio", tt.wantIO, + } + if !slices.Equal(args, want) { + t.Errorf("args = %v, want %v", args, want) + } + }) + } +} + +func TestDiskAttachArgsRejectBadOptions(t *testing.T) { + _, err := New("cocoon", nil, nil, false, "").diskAttachArgs("sbx-1", VolumeSpec{DirectIO: "maybe"}) + if err == nil || !strings.Contains(err.Error(), "on, off, or auto") { + t.Errorf("got %v, want directio validation error", err) + } +} + +func TestMountVolumeUsesSysfsAndReadOnlyMount(t *testing.T) { + path := sockPath(t) + fake := serveFakeSilkd(t, path) + configureVolumeDevices(fake) + if err := New("cocoon", nil, nil, false, "").MountVolume( + t.Context(), path, "imagenet", "/datasets/training", + ); err != nil { + t.Fatalf("MountVolume: %v", err) + } + + fake.mu.Lock() + defer fake.mu.Unlock() + wantExec := [][]string{ + {"mkdir", "-p", "--", "/datasets/training"}, + {"mount", "-o", "ro", "--", "/dev/vdc", "/datasets/training"}, + } + if !slices.EqualFunc(fake.execCalls, wantExec, slices.Equal) { + t.Errorf("exec calls = %v, want %v", fake.execCalls, wantExec) + } + wantReads := []string{ + "/sys/block/vda/serial", + "/sys/block/vdb/serial", + "/sys/block/vdc/serial", + } + if fake.listCalls != 1 || !slices.Equal(fake.readCalls, wantReads) { + t.Errorf("sysfs calls = list:%d read:%v, want list:1 read:%v", fake.listCalls, fake.readCalls, wantReads) + } + if fake.execEnv["PATH"] == "" { + t.Error("guest exec PATH is empty") + } +} + +func TestMountVolumeWaitsForDelayedSysfsSerial(t *testing.T) { + path := sockPath(t) + fake := serveFakeSilkd(t, path) + configureVolumeDevices(fake) + fake.mu.Lock() + fake.readMisses["/sys/block/vdc/serial"] = 1 + fake.mu.Unlock() + if err := New("cocoon", nil, nil, false, "").MountVolume( + t.Context(), path, "imagenet", "/datasets/training", + ); err != nil { + t.Fatalf("MountVolume: %v", err) + } + + fake.mu.Lock() + defer fake.mu.Unlock() + if fake.listCalls != 2 { + t.Errorf("fs_list calls = %d, want 2", fake.listCalls) + } + if got := fake.readCalls[len(fake.readCalls)-1]; got != "/sys/block/vdc/serial" { + t.Errorf("last fs_read = %q, want target serial", got) + } +} + +func TestMountVolumeStopsAtFailedStage(t *testing.T) { + for _, tt := range []struct { + name string + prepare func(*fakeSilkd) + wantErr string + wantExec int + }{ + {"list", func(f *fakeSilkd) { f.listErr = "list failed" }, "wait for volume device imagenet", 0}, + {"read", func(f *fakeSilkd) { f.readErr["/sys/block/vda/serial"] = "read failed" }, "wait for volume device imagenet", 0}, + {"mkdir", func(f *fakeSilkd) { f.execCode, f.execFailAt = 3, 1 }, "create volume mount point", 1}, + {"mount", func(f *fakeSilkd) { f.execCode, f.execFailAt = 3, 2 }, "mount volume imagenet", 2}, + } { + t.Run(tt.name, func(t *testing.T) { + path := sockPath(t) + fake := serveFakeSilkd(t, path) + configureVolumeDevices(fake) + fake.mu.Lock() + tt.prepare(fake) + fake.mu.Unlock() + err := New("cocoon", nil, nil, false, "").MountVolume( + t.Context(), path, "imagenet", "/datasets/training", + ) + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("got %v, want %q failure", err, tt.wantErr) + } + fake.mu.Lock() + defer fake.mu.Unlock() + if len(fake.execCalls) != tt.wantExec { + t.Errorf("exec calls = %v, want %d", fake.execCalls, tt.wantExec) + } + }) + } +} + +func TestMountVolumeDeviceProbeIsBoundedAndCancelable(t *testing.T) { + if volumeProbeTimeout != 2*time.Second { + t.Fatalf("volume probe timeout = %s, want 2s", volumeProbeTimeout) + } + path := sockPath(t) + fake := serveFakeSilkd(t, path) + configureVolumeDevices(fake) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + err := New("cocoon", nil, nil, false, "").MountVolume( + ctx, path, "missing", "/datasets/training", + ) + if !errors.Is(err, context.Canceled) { + t.Errorf("error = %v, want context cancellation", err) + } +} + +func configureVolumeDevices(f *fakeSilkd) { + f.mu.Lock() + defer f.mu.Unlock() + f.block = []wire.DirEntry{ + {Name: "vda", Kind: wire.FileKindSymlink}, + {Name: "vdb", Kind: wire.FileKindSymlink}, + {Name: "vdc", Kind: wire.FileKindSymlink}, + } + f.serial["/sys/block/vda/serial"] = []byte("root\n") + f.serial["/sys/block/vdc/serial"] = []byte("imagenet\n") +} diff --git a/sandboxd/main.go b/sandboxd/main.go index 071c13d3..965fd486 100644 --- a/sandboxd/main.go +++ b/sandboxd/main.go @@ -70,6 +70,11 @@ func main() { if err != nil { logger.Fatalf(ctx, err, "load config") } + for _, volume := range cfg.Volumes { + if _, statErr := os.Stat(volume.Path); statErr != nil { + logger.Warnf(ctx, "volume %s path %s unavailable at startup: %v", volume.Name, volume.Path, statErr) + } + } eng := engine.New(cfg.CocoonBin, cfg.Bridges, cfg.Networks, cfg.NoDirectIO, cfg.RestoreMode) if v, warn := eng.VersionWarning(ctx); warn != "" { logger.Warn(ctx, warn) @@ -104,7 +109,9 @@ func main() { } defer func() { _ = msh.Shutdown() }() placer = msh - mgr.SetTemplateNotifier(func() { msh.UpdateSelf(ctx, mgr.WarmCounts(), mgr.TemplateHashes()) }) + mgr.SetTemplateNotifier(func() { + msh.UpdateSelf(ctx, mgr.WarmCounts(), mgr.TemplateHashes(), mgr.VolumeNames()) + }) clusterKey, err := cfg.Mesh.DecodedKey() if err != nil { logger.Fatalf(ctx, err, "decode mesh cluster key") @@ -194,6 +201,7 @@ func startMesh(ctx context.Context, cfg *config.Config, mgr *pool.Manager) (*mes } // Publish the config digest before Join, so the first gossip carries it. msh.SetSelfDigest(cfg.ClusterDigest(mgr.EgressCAFingerprint())) + msh.UpdateSelf(ctx, mgr.WarmCounts(), mgr.TemplateHashes(), mgr.VolumeNames()) if err := msh.Join(mc.Join); err != nil { _ = msh.Shutdown() return nil, err @@ -201,8 +209,8 @@ func startMesh(ctx context.Context, cfg *config.Config, mgr *pool.Manager) (*mes return msh, nil } -// gossipNodeState republishes this node's warm-pool counts and template set -// every tick so the mesh's placement view tracks refill and promotes. +// gossipNodeState republishes this node's warm-pool counts, templates, and +// locally available volumes every tick so placement tracks filesystem changes. func gossipNodeState(ctx context.Context, msh *mesh.Mesh, mgr *pool.Manager) { t := time.NewTicker(gossipInterval) defer t.Stop() @@ -211,7 +219,7 @@ func gossipNodeState(ctx context.Context, msh *mesh.Mesh, mgr *pool.Manager) { case <-ctx.Done(): return case <-t.C: - msh.UpdateSelf(ctx, mgr.WarmCounts(), mgr.TemplateHashes()) + msh.UpdateSelf(ctx, mgr.WarmCounts(), mgr.TemplateHashes(), mgr.VolumeNames()) } } } diff --git a/sandboxd/mesh/mesh.go b/sandboxd/mesh/mesh.go index 5eefcc37..8536aede 100644 --- a/sandboxd/mesh/mesh.go +++ b/sandboxd/mesh/mesh.go @@ -1,9 +1,8 @@ -// Package mesh gossips per-node warm-pool counts over a hashicorp/memberlist -// SWIM cluster, so any node can redirect a claim to a peer that already holds -// a warm sandbox for the requested pool key. Gossip carries only placement -// hints — per-sandbox state stays node-local — so a stale view costs at most -// one extra redirect, never correctness. A single node with no seeds is a -// valid mesh of one. +// Package mesh gossips per-node warm counts, promoted templates, and available +// volume names over a hashicorp/memberlist SWIM cluster. Gossip carries only +// placement hints — per-sandbox state stays node-local — so a stale view costs +// at most one failed redirect, never correctness. A single node with no seeds +// is a valid mesh of one. package mesh import ( @@ -33,6 +32,7 @@ type NodeState struct { Epoch uint64 `json:"epoch"` Pools map[string]int `json:"pools"` // PoolKey hash → warm count Templates []string `json:"templates,omitempty"` // promoted-template key hashes on disk + Volumes []string `json:"volumes,omitempty"` // locally available dataset names Digest string `json:"digest,omitempty"` // cluster-invariant config digest } @@ -103,15 +103,14 @@ func (m *Mesh) Join(seeds []string) error { return nil } -// UpdateSelf republishes this node's warm-pool counts and promoted-template -// set, bumping the epoch so peers adopt the new view. An unchanged view is -// not republished. templates must arrive sorted: the unchanged compare is -// order-sensitive. -func (m *Mesh) UpdateSelf(ctx context.Context, pools map[string]int, templates []string) { +// UpdateSelf republishes this node's warm-pool counts, promoted-template set, +// and locally available volumes. An unchanged view does not bump the epoch. +// templates and volumes must arrive sorted: the compare is order-sensitive. +func (m *Mesh) UpdateSelf(ctx context.Context, pools map[string]int, templates, volumes []string) { m.updateMu.Lock() defer m.updateMu.Unlock() m.mu.Lock() - if maps.Equal(m.self.Pools, pools) && slices.Equal(m.self.Templates, templates) { + if maps.Equal(m.self.Pools, pools) && slices.Equal(m.self.Templates, templates) && slices.Equal(m.self.Volumes, volumes) { m.mu.Unlock() return } @@ -128,6 +127,7 @@ func (m *Mesh) UpdateSelf(ctx context.Context, pools map[string]int, templates [ m.self.Epoch = epoch m.self.Pools = pools m.self.Templates = templates + m.self.Volumes = volumes m.view[m.self.NodeID] = m.self m.mu.Unlock() } @@ -162,6 +162,77 @@ func (m *Mesh) ConfigMismatches() int { // chosen power-of-two-choices to avoid herding every waiter onto one node. // Self is never a candidate — the caller has already missed locally. func (m *Mesh) Candidates(keyHash string) []string { + return m.warmCandidates(keyHash, func(NodeState) bool { return true }) +} + +// VolumeCandidates returns warm peers that advertise every requested volume. +func (m *Mesh) VolumeCandidates(keyHash string, names []string) []string { + return m.warmCandidates(keyHash, func(st NodeState) bool { return containsAll(st.Volumes, names) }) +} + +// TemplateOwners returns up to two peer addresses whose gossiped template +// set contains keyHash — the redirect targets for a name-based claim or +// delete of a template this node does not hold. Self is excluded: the caller +// has already checked its own disk. +func (m *Mesh) TemplateOwners(keyHash string) []string { + return m.owners(func(st NodeState) bool { return slices.Contains(st.Templates, keyHash) }) +} + +// VolumeOwners returns peers that currently advertise every requested volume. +// Self is excluded because the caller checks local availability first. +func (m *Mesh) VolumeOwners(names []string) []string { + return m.owners(func(st NodeState) bool { return containsAll(st.Volumes, names) }) +} + +// TemplateVolumeOwners returns peers that hold both the promoted template and +// every requested volume, avoiding an incorrect intersection after truncation. +func (m *Mesh) TemplateVolumeOwners(keyHash string, names []string) []string { + return m.owners(func(st NodeState) bool { + return slices.Contains(st.Templates, keyHash) && containsAll(st.Volumes, names) + }) +} + +// VolumeHolders counts every member advertising each volume, including self. +func (m *Mesh) VolumeHolders() map[string]int { + m.mu.Lock() + defer m.mu.Unlock() + holders := map[string]int{} + for _, st := range m.view { + for _, name := range st.Volumes { + holders[name]++ + } + } + return holders +} + +// Members returns the current cluster view (self included). +func (m *Mesh) Members() []NodeState { + m.mu.Lock() + defer m.mu.Unlock() + return slices.Collect(maps.Values(m.view)) +} + +// PeerAddrs returns the data-plane addresses of the other nodes, for a +// client-side Lookup scatter. +func (m *Mesh) PeerAddrs() []string { + m.mu.Lock() + defer m.mu.Unlock() + addrs := make([]string, 0, len(m.view)) + for id, st := range m.view { + if id != m.self.NodeID { + addrs = append(addrs, st.Addr) + } + } + return addrs +} + +// Shutdown leaves the mesh and stops the member. +func (m *Mesh) Shutdown() error { + _ = m.ml.Leave(leaveTimeout) + return m.ml.Shutdown() +} + +func (m *Mesh) warmCandidates(keyHash string, match func(NodeState) bool) []string { m.mu.Lock() type cand struct { addr string @@ -172,7 +243,7 @@ func (m *Mesh) Candidates(keyHash string) []string { if id == m.self.NodeID { continue } - if st.Pools[keyHash] > 0 { + if st.Pools[keyHash] > 0 && match(st) { pool = append(pool, cand{st.Addr, st.Pools[keyHash]}) } } @@ -196,58 +267,25 @@ func (m *Mesh) Candidates(keyHash string) []string { return []string{a.addr, b.addr} } -// TemplateOwners returns up to two peer addresses whose gossiped template -// set contains keyHash — the redirect targets for a name-based claim or -// delete of a template this node does not hold. Self is excluded: the caller -// has already checked its own disk. -func (m *Mesh) TemplateOwners(keyHash string) []string { +func (m *Mesh) persistEpoch(epoch uint64) error { + return storeEpoch(m.epochPath, epoch) +} + +func (m *Mesh) owners(match func(NodeState) bool) []string { m.mu.Lock() var owners []string for id, st := range m.view { - if id != m.self.NodeID && slices.Contains(st.Templates, keyHash) { + if id != m.self.NodeID && match(st) { owners = append(owners, st.Addr) } } m.mu.Unlock() - // Which two survive truncation is already jittered by map iteration - // order; unlike Candidates there is no warmth to rank by. if len(owners) > 2 { owners = owners[:2] } return owners } -// Members returns the current cluster view (self included). -func (m *Mesh) Members() []NodeState { - m.mu.Lock() - defer m.mu.Unlock() - return slices.Collect(maps.Values(m.view)) -} - -// PeerAddrs returns the data-plane addresses of the other nodes, for a -// client-side Lookup scatter. -func (m *Mesh) PeerAddrs() []string { - m.mu.Lock() - defer m.mu.Unlock() - addrs := make([]string, 0, len(m.view)) - for id, st := range m.view { - if id != m.self.NodeID { - addrs = append(addrs, st.Addr) - } - } - return addrs -} - -// Shutdown leaves the mesh and stops the member. -func (m *Mesh) Shutdown() error { - _ = m.ml.Leave(leaveTimeout) - return m.ml.Shutdown() -} - -func (m *Mesh) persistEpoch(epoch uint64) error { - return storeEpoch(m.epochPath, epoch) -} - // forget drops a departed node from the placement view so redirects stop // targeting a dead peer; SWIM detected the death, the view must follow. func (m *Mesh) forget(nodeID string) { @@ -291,6 +329,12 @@ func short(digest string) string { return digest } +func containsAll(have, need []string) bool { + return len(need) > 0 && !slices.ContainsFunc(need, func(name string) bool { + return !slices.Contains(have, name) + }) +} + var _ memberlist.Delegate = (*delegate)(nil) // delegate carries this node's full view on each memberlist push/pull sync, so diff --git a/sandboxd/mesh/mesh_test.go b/sandboxd/mesh/mesh_test.go index 17e327d7..20e4968a 100644 --- a/sandboxd/mesh/mesh_test.go +++ b/sandboxd/mesh/mesh_test.go @@ -4,6 +4,7 @@ import ( "fmt" "io" "log" + "maps" "path/filepath" "slices" "testing" @@ -31,7 +32,7 @@ func TestMergeKeepsHigherEpoch(t *testing.T) { func TestMergeNeverOverwritesSelf(t *testing.T) { m := newTestMesh(t, "a") - m.UpdateSelf(t.Context(), map[string]int{"k": 3}, nil) + m.UpdateSelf(t.Context(), map[string]int{"k": 3}, nil, nil) // A peer claiming to be "a" must not clobber our authoritative self entry. m.merge([]NodeState{{NodeID: "a", Addr: "evil:9999", Epoch: 999, Pools: map[string]int{"k": 0}}}) @@ -44,7 +45,7 @@ func TestMergeNeverOverwritesSelf(t *testing.T) { func TestCandidatesExcludeSelfAndEmpty(t *testing.T) { m := newTestMesh(t, "a") - m.UpdateSelf(t.Context(), map[string]int{"k": 5}, nil) // self has warm, but is never a candidate + m.UpdateSelf(t.Context(), map[string]int{"k": 5}, nil, nil) // self has warm, but is never a candidate m.merge([]NodeState{ {NodeID: "b", Addr: "b:7777", Epoch: 1, Pools: map[string]int{"k": 2}}, {NodeID: "c", Addr: "c:7777", Epoch: 1, Pools: map[string]int{"k": 0}}, // no warm @@ -62,7 +63,7 @@ func TestCandidatesExcludeSelfAndEmpty(t *testing.T) { func TestTemplateOwnersExcludeSelfAndUnknown(t *testing.T) { m := newTestMesh(t, "a") - m.UpdateSelf(t.Context(), nil, []string{"tpl"}) // self holds it, but is never an owner candidate + m.UpdateSelf(t.Context(), nil, []string{"tpl"}, nil) // self holds it, but is never an owner candidate m.merge([]NodeState{ {NodeID: "b", Addr: "b:7777", Epoch: 1, Templates: []string{"tpl", "other"}}, {NodeID: "c", Addr: "c:7777", Epoch: 1, Templates: []string{"other"}}, @@ -87,7 +88,7 @@ func TestForgetPrunesDeadNode(t *testing.T) { t.Errorf("candidates after forget %v, want nil (b pruned)", got) } // forgetting self is a no-op. - m.UpdateSelf(t.Context(), map[string]int{"k": 1}, nil) + m.UpdateSelf(t.Context(), map[string]int{"k": 1}, nil, nil) m.forget("a") if len(m.Members()) != 1 { t.Error("forget removed self") @@ -121,7 +122,7 @@ func TestTwoNodeClusterGossipsPools(t *testing.T) { if err := b.mesh.Join([]string{a.addr}); err != nil { t.Fatalf("join: %v", err) } - a.mesh.UpdateSelf(t.Context(), map[string]int{"kk": 4}, []string{"tpl-hash"}) + a.mesh.UpdateSelf(t.Context(), map[string]int{"kk": 4}, []string{"tpl-hash"}, []string{"dataset"}) // Push/pull sync propagates a's warm counts and template set to b within // a few intervals. @@ -129,8 +130,10 @@ func TestTwoNodeClusterGossipsPools(t *testing.T) { for time.Now().Before(deadline) { cands := b.mesh.Candidates("kk") owners := b.mesh.TemplateOwners("tpl-hash") + volumeOwners := b.mesh.VolumeOwners([]string{"dataset"}) if len(cands) == 1 && cands[0] == "node-a:7777" && - len(owners) == 1 && owners[0] == "node-a:7777" { + len(owners) == 1 && owners[0] == "node-a:7777" && + len(volumeOwners) == 1 && volumeOwners[0] == "node-a:7777" { return } time.Sleep(100 * time.Millisecond) @@ -138,6 +141,72 @@ func TestTwoNodeClusterGossipsPools(t *testing.T) { t.Fatalf("node-b never learned node-a's state: view=%+v", b.mesh.Members()) } +func TestVolumeOwnersRequireEveryNameAndExcludeSelf(t *testing.T) { + m := newTestMesh(t, "a") + m.UpdateSelf(t.Context(), nil, nil, []string{"dataset", "weights"}) + m.merge([]NodeState{ + {NodeID: "b", Addr: "b:7777", Epoch: 1, Volumes: []string{"dataset", "weights"}}, + {NodeID: "c", Addr: "c:7777", Epoch: 1, Volumes: []string{"dataset"}}, + {NodeID: "d", Addr: "d:7777", Epoch: 1, Volumes: []string{"weights"}}, + }) + + if owners := m.VolumeOwners([]string{"dataset", "weights"}); !slices.Equal(owners, []string{"b:7777"}) { + t.Errorf("owners=%v, want only full holder b", owners) + } + if owners := m.VolumeOwners([]string{"absent"}); owners != nil { + t.Errorf("unknown owners=%v, want nil", owners) + } + if owners := m.VolumeOwners(nil); owners != nil { + t.Errorf("empty request owners=%v, want nil", owners) + } +} + +func TestVolumeCandidatesRequireWarmAndEveryVolume(t *testing.T) { + m := newTestMesh(t, "a") + m.merge([]NodeState{ + {NodeID: "both", Addr: "both:7777", Epoch: 1, Pools: map[string]int{"k": 2}, Volumes: []string{"dataset", "weights"}}, + {NodeID: "partial", Addr: "partial:7777", Epoch: 1, Pools: map[string]int{"k": 3}, Volumes: []string{"dataset"}}, + {NodeID: "cold", Addr: "cold:7777", Epoch: 1, Pools: map[string]int{"k": 0}, Volumes: []string{"dataset", "weights"}}, + }) + + if got := m.VolumeCandidates("k", []string{"dataset", "weights"}); !slices.Equal(got, []string{"both:7777"}) { + t.Errorf("candidates=%v, want only warm full holder", got) + } + if got := m.VolumeCandidates("missing", []string{"dataset"}); got != nil { + t.Errorf("missing pool candidates=%v, want nil", got) + } +} + +func TestTemplateVolumeOwnersUseTrueIntersection(t *testing.T) { + m := newTestMesh(t, "a") + m.merge([]NodeState{ + {NodeID: "template", Addr: "template:7777", Epoch: 1, Templates: []string{"tpl"}}, + {NodeID: "volume", Addr: "volume:7777", Epoch: 1, Volumes: []string{"dataset"}}, + {NodeID: "both", Addr: "both:7777", Epoch: 1, Templates: []string{"tpl"}, Volumes: []string{"dataset"}}, + }) + + if owners := m.TemplateVolumeOwners("tpl", []string{"dataset"}); !slices.Equal(owners, []string{"both:7777"}) { + t.Errorf("owners=%v, want only intersection holder", owners) + } + if owners := m.TemplateVolumeOwners("missing", []string{"dataset"}); owners != nil { + t.Errorf("missing template owners=%v, want nil", owners) + } +} + +func TestVolumeHoldersCountSelfAndPeers(t *testing.T) { + m := newTestMesh(t, "a") + m.UpdateSelf(t.Context(), nil, nil, []string{"dataset"}) + m.merge([]NodeState{ + {NodeID: "b", Addr: "b:7777", Epoch: 1, Volumes: []string{"dataset", "weights"}}, + {NodeID: "c", Addr: "c:7777", Epoch: 1, Volumes: []string{"weights"}}, + }) + + want := map[string]int{"dataset": 2, "weights": 2} + if got := m.VolumeHolders(); !maps.Equal(got, want) { + t.Errorf("holders=%v, want %v", got, want) + } +} + func newTestMesh(t *testing.T, id string) *Mesh { t.Helper() return &Mesh{ diff --git a/sandboxd/mesh/state_test.go b/sandboxd/mesh/state_test.go index 699dc392..3813b3e4 100644 --- a/sandboxd/mesh/state_test.go +++ b/sandboxd/mesh/state_test.go @@ -52,7 +52,7 @@ func TestUpdateSelfPersistFailKeepsOldState(t *testing.T) { before := m.self.Epoch // A non-existent dir makes the durable write fail: the epoch must not publish. m.epochPath = filepath.Join(dir, "gone", "mesh-epoch") - m.UpdateSelf(t.Context(), map[string]int{"k": 1}, nil) + m.UpdateSelf(t.Context(), map[string]int{"k": 1}, nil, nil) if m.self.Epoch != before { t.Errorf("self epoch advanced to %d on a failed persist, want %d held", m.self.Epoch, before) } @@ -65,7 +65,7 @@ func TestUpdateSelfPersistsEpoch(t *testing.T) { dir := t.TempDir() m := newBoundMesh(t, dir) before := m.self.Epoch - m.UpdateSelf(t.Context(), map[string]int{"k": 1}, nil) + m.UpdateSelf(t.Context(), map[string]int{"k": 1}, nil, nil) if got := loadEpoch(filepath.Join(dir, "mesh-epoch")); got <= before { t.Errorf("persisted epoch %d did not advance past the seed %d", got, before) } @@ -78,8 +78,8 @@ func TestUpdateSelfConcurrentDropsNothing(t *testing.T) { a := map[string]int{"a": i + 1} b := map[string]int{"b": i + 1} var wg sync.WaitGroup - wg.Go(func() { m.UpdateSelf(t.Context(), a, nil) }) - wg.Go(func() { m.UpdateSelf(t.Context(), b, nil) }) + wg.Go(func() { m.UpdateSelf(t.Context(), a, nil, nil) }) + wg.Go(func() { m.UpdateSelf(t.Context(), b, nil, nil) }) wg.Wait() // Serialized updates with distinct payloads must both land: the loser // of the old TOCTOU race silently dropped its payload and one bump. @@ -90,6 +90,23 @@ func TestUpdateSelfConcurrentDropsNothing(t *testing.T) { } } +func TestUpdateSelfBumpsOnlyWhenVolumesChange(t *testing.T) { + m := newBoundMesh(t, t.TempDir()) + base := m.self.Epoch + m.UpdateSelf(t.Context(), nil, nil, []string{"dataset"}) + if m.self.Epoch != base+1 { + t.Fatalf("first volume publish epoch=%d, want %d", m.self.Epoch, base+1) + } + m.UpdateSelf(t.Context(), nil, nil, []string{"dataset"}) + if m.self.Epoch != base+1 { + t.Errorf("unchanged volume republished at epoch=%d", m.self.Epoch) + } + m.UpdateSelf(t.Context(), nil, nil, []string{"dataset", "weights"}) + if m.self.Epoch != base+2 { + t.Errorf("changed volume epoch=%d, want %d", m.self.Epoch, base+2) + } +} + func TestConfigDigestMismatch(t *testing.T) { m := newTestMesh(t, "self") m.SetSelfDigest("self-digest") diff --git a/sandboxd/pool/archive_test.go b/sandboxd/pool/archive_test.go index 1c4d5f98..d26957e9 100644 --- a/sandboxd/pool/archive_test.go +++ b/sandboxd/pool/archive_test.go @@ -306,7 +306,7 @@ func TestArchiveKeepsForeverWhenDeleteZero(t *testing.T) { func TestArchiveRetentionPurge(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) - sb, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "") + sb, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil) if err != nil { t.Fatalf("claim: %v", err) } diff --git a/sandboxd/pool/checkpoint.go b/sandboxd/pool/checkpoint.go index 099f3b4e..bc294b61 100644 --- a/sandboxd/pool/checkpoint.go +++ b/sandboxd/pool/checkpoint.go @@ -42,6 +42,9 @@ func (m *Manager) Checkpoint(ctx context.Context, id string, cred Cred, name, te if name != "" && !types.NameRe.MatchString(name) { return types.Checkpoint{}, fmt.Errorf("%w: %q must match %s", ErrBadName, name, types.NameRe) } + if hasAppliedVolumes(sb) { + return types.Checkpoint{}, ErrVolumeCapture + } if !sb.Key.Capturable() { return types.Checkpoint{}, ErrNoEgressFork } @@ -279,8 +282,7 @@ func (m *Manager) healCheckpoint(ctx context.Context, ckptID string) (types.Chec // runHeal is healCheckpoint's flight body: it stages and pulls WITHOUT // holding ckptID's record lock — a heal budget runs up to 30 minutes, and // holding the lock across it would pin every same-id operation behind an -// uncancellable wait. The lock covers only the fast final steps: the veto -// check, re-validate, publish. +// uncancellable wait. func (m *Manager) runHeal(ckptID string) (types.Checkpoint, error) { select { case m.healSem <- struct{}{}: diff --git a/sandboxd/pool/checkpoint_test.go b/sandboxd/pool/checkpoint_test.go index 4c93bd4e..a924463a 100644 --- a/sandboxd/pool/checkpoint_test.go +++ b/sandboxd/pool/checkpoint_test.go @@ -120,11 +120,11 @@ func TestRetentionSweepsExpiredCheckpoints(t *testing.T) { func TestCheckpointTenantIsolation(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - srcA, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "") + srcA, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil) if err != nil { t.Fatalf("acme claim: %v", err) } - srcB, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "beta", "") + srcB, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "beta", "", nil) if err != nil { t.Fatalf("beta claim: %v", err) } diff --git a/sandboxd/pool/claim.go b/sandboxd/pool/claim.go index 3403e4e9..d4b9b146 100644 --- a/sandboxd/pool/claim.go +++ b/sandboxd/pool/claim.go @@ -25,14 +25,18 @@ const ( // ErrNoWarm means the pool is empty (the caller may redirect or provision). // tenant attributes the claim; empty means the operator (root). claimRef is an // opaque caller reference recorded on the claim; empty means none. -func (m *Manager) ClaimWarm(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string) (*types.Sandbox, error) { +func (m *Manager) ClaimWarm(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) { start := time.Now() if err := m.validate(key); err != nil { return nil, err } - if err := m.overQuota(1, tenant); err != nil { + volumeSpecs, err := m.resolveVolumes(key, tenant, volumes) + if err != nil { return nil, err } + if quotaErr := m.overQuota(1, tenant); quotaErr != nil { + return nil, quotaErr + } m.mu.Lock() var sb *types.Sandbox if p := m.pools[key]; p != nil { @@ -47,6 +51,10 @@ func (m *Manager) ClaimWarm(ctx context.Context, key types.PoolKey, ttl time.Dur return nil, ErrNoWarm } m.kickRefill() + if volumeErr := m.applyVolumes(ctx, sb, volumeSpecs); volumeErr != nil { + m.destroy(ctx, sb.VMName) + return nil, volumeErr + } sb.Tenant = tenant sb.ClaimRef = claimRef out, err := m.finalize(ctx, sb, ttl) @@ -59,36 +67,14 @@ func (m *Manager) ClaimWarm(ctx context.Context, key types.PoolKey, ttl time.Dur // ClaimProvision creates a claim-ready sandbox (golden clone or cold boot). // claimRef is an opaque caller reference recorded on the claim; empty means none. -func (m *Manager) ClaimProvision(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string) (*types.Sandbox, error) { - start := time.Now() - if err := m.validate(key); err != nil { - return nil, err - } - if err := m.overQuota(1, tenant); err != nil { - return nil, err - } - golden, templateDigest, release, err := m.resolveGolden(ctx, key) - if err != nil { - return nil, fmt.Errorf("resolve template: %w", err) - } - sb, err := m.provision(ctx, key, golden) - release() - if err != nil { - return nil, err - } - sb.TemplateDigest = templateDigest - sb.Tenant = tenant - sb.ClaimRef = claimRef - out, err := m.finalize(ctx, sb, ttl) - if err == nil { - if golden != "" { - m.counters.claimsClone.Add(1) - } else { - m.counters.claimsCold.Add(1) - } - m.counters.claimNanos.Add(uint64(time.Since(start))) //nolint:gosec // durations are positive - } - return out, err +func (m *Manager) ClaimProvision(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) { + return m.claimProvision(ctx, key, ttl, tenant, claimRef, volumes, false) +} + +// ClaimProvisionPromoted requires key to resolve from a promoted template and +// never falls through to a cold image boot. +func (m *Manager) ClaimProvisionPromoted(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) { + return m.claimProvision(ctx, key, ttl, tenant, claimRef, volumes, true) } // Release destroys a claimed sandbox after authorizing cred. @@ -288,7 +274,11 @@ func (m *Manager) finalizeBatch(ctx context.Context, sbs []*types.Sandbox, ttl t // Usage lands only after the whole batch armed: a rollback must not leave // claim events with no terminal release/reap in the billing stream. for _, sb := range sbs { - m.recordUsage(ctx, usageEvent{Event: "claim", ID: sb.ID, VMName: sb.VMName, KeyHash: sb.Key.Hash(), Tenant: sb.Tenant}) //nolint:goconst // event name; other occurrences are test assertions + m.recordUsage(ctx, usageEvent{ + Event: "claim", //nolint:goconst // event name; other occurrences are test assertions + ID: sb.ID, VMName: sb.VMName, + KeyHash: sb.Key.Hash(), Tenant: sb.Tenant, Volumes: types.VolumeNames(sb.Volumes), + }) } return nil } @@ -471,6 +461,50 @@ func (m *Manager) authed(id, token string) (*types.Sandbox, bool) { return sb, true } +func (m *Manager) claimProvision(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume, requirePromoted bool) (*types.Sandbox, error) { + start := time.Now() + if err := m.validate(key); err != nil { + return nil, err + } + volumeSpecs, err := m.resolveVolumes(key, tenant, volumes) + if err != nil { + return nil, err + } + if quotaErr := m.overQuota(1, tenant); quotaErr != nil { + return nil, quotaErr + } + golden, err := m.resolveGolden(ctx, key) + if err != nil { + return nil, fmt.Errorf("resolve template: %w", err) + } + if requirePromoted && !golden.promoted { + golden.release() + return nil, ErrVolumeUnavailable + } + sb, err := m.provision(ctx, key, golden.dir) + golden.release() + if err != nil { + return nil, err + } + if volumeErr := m.applyVolumes(ctx, sb, volumeSpecs); volumeErr != nil { + m.destroy(ctx, sb.VMName) + return nil, volumeErr + } + sb.TemplateDigest = golden.templateDigest + sb.Tenant = tenant + sb.ClaimRef = claimRef + out, err := m.finalize(ctx, sb, ttl) + if err == nil { + if golden.dir != "" { + m.counters.claimsClone.Add(1) + } else { + m.counters.claimsCold.Add(1) + } + m.counters.claimNanos.Add(uint64(time.Since(start))) //nolint:gosec // durations are positive + } + return out, err +} + func stampIdentity(sb *types.Sandbox, ttl time.Duration) { sb.ID = "sb_" + randHex(8) sb.Token = randHex(16) diff --git a/sandboxd/pool/claims.go b/sandboxd/pool/claims.go index 696cee0f..5542df8f 100644 --- a/sandboxd/pool/claims.go +++ b/sandboxd/pool/claims.go @@ -7,6 +7,7 @@ import ( "io/fs" "os" "path/filepath" + "slices" "sync" "sync/atomic" "time" @@ -25,25 +26,25 @@ type claimSnapshot struct { // copied by value so commit's marshal runs off the manager mutex without // racing the live record's Transition mutex and lastActivity. type claimDTO struct { - ID string `json:"id"` - VMName string `json:"vm_name"` - Key types.PoolKey `json:"key"` - Token string `json:"token,omitempty"` - Deadline time.Time `json:"deadline,omitzero"` - Tenant string `json:"tenant,omitempty"` - ClaimRef string `json:"claim_ref,omitempty"` - VsockSocket string `json:"vsock_socket,omitempty"` - TAP string `json:"tap,omitempty"` - HibernateSnap string `json:"hibernate_snap,omitempty"` - PendingSnap string `json:"pending_snap,omitempty"` - ArchiveCk string `json:"archive_ck,omitempty"` - FromCheckpoint string `json:"from_checkpoint,omitempty"` + ID string `json:"id"` + VMName string `json:"vm_name"` + Key types.PoolKey `json:"key"` + Token string `json:"token,omitempty"` + Deadline time.Time `json:"deadline,omitzero"` + Tenant string `json:"tenant,omitempty"` + ClaimRef string `json:"claim_ref,omitempty"` + Volumes []types.Volume `json:"volumes,omitempty"` + VsockSocket string `json:"vsock_socket,omitempty"` + TAP string `json:"tap,omitempty"` + HibernateSnap string `json:"hibernate_snap,omitempty"` + PendingSnap string `json:"pending_snap,omitempty"` + ArchiveCk string `json:"archive_ck,omitempty"` + FromCheckpoint string `json:"from_checkpoint,omitempty"` } // claimStore persists claimed sandboxes across daemon restarts. Warm pool // VMs are deliberately not persisted: they are cheap to rebuild and unsafe -// to trust after an unsupervised gap. snapshot()/commit() document the -// split-write that keeps the manager mutex off the marshal and the syscalls. +// to trust after an unsupervised gap. type claimStore struct { path string @@ -79,7 +80,8 @@ func (s *claimStore) snapshot(claims map[string]*types.Sandbox) claimSnapshot { for id, sb := range claims { dtos[id] = claimDTO{ ID: sb.ID, VMName: sb.VMName, Key: sb.Key, Token: sb.Token, - Deadline: sb.Deadline, Tenant: sb.Tenant, ClaimRef: sb.ClaimRef, VsockSocket: sb.VsockSocket, + Deadline: sb.Deadline, Tenant: sb.Tenant, ClaimRef: sb.ClaimRef, + Volumes: slices.Clone(sb.Volumes), VsockSocket: sb.VsockSocket, TAP: sb.TAP, HibernateSnap: sb.HibernateSnap, PendingSnap: sb.PendingSnap, ArchiveCk: sb.ArchiveCk, FromCheckpoint: sb.FromCheckpoint, } diff --git a/sandboxd/pool/claims_test.go b/sandboxd/pool/claims_test.go index 975e9493..c13d2418 100644 --- a/sandboxd/pool/claims_test.go +++ b/sandboxd/pool/claims_test.go @@ -1,9 +1,10 @@ package pool import ( - "maps" "os" "path/filepath" + "reflect" + "slices" "sync" "testing" "time" @@ -14,7 +15,7 @@ import ( func TestStoreRoundTrip(t *testing.T) { s := newClaimStore(t.TempDir()) claims := map[string]*types.Sandbox{ - "sb_a": {ID: "sb_a", VMName: "sbx-1", Key: testKey, Token: "t1", Deadline: time.Now().Add(time.Minute).UTC(), ClaimRef: "ns/workload", VsockSocket: "/v/1"}, + "sb_a": {ID: "sb_a", VMName: "sbx-1", Key: testKey, Token: "t1", Deadline: time.Now().Add(time.Minute).UTC(), ClaimRef: "ns/workload", Volumes: []types.Volume{{Name: "dataset-a", Mount: "/datasets/a"}}, VsockSocket: "/v/1"}, "sb_b": {ID: "sb_b", VMName: "sbx-2", Key: testKey, Token: "t2"}, } @@ -25,11 +26,32 @@ func TestStoreRoundTrip(t *testing.T) { if err != nil { t.Fatalf("load: %v", err) } - if !maps.EqualFunc(got, claims, func(a, b *types.Sandbox) bool { return *a == *b }) { + if !reflect.DeepEqual(got, claims) { t.Errorf("got %+v, want %+v", got, claims) } } +func TestClaimStoreSnapshotDetachesVolumes(t *testing.T) { + s := newClaimStore(t.TempDir()) + sb := &types.Sandbox{ID: "sb_a", Volumes: []types.Volume{{Name: "dataset-a", Mount: "/datasets/a"}}} + snap := s.snapshot(map[string]*types.Sandbox{sb.ID: sb}) + sb.Volumes[0].Mount = "/mutated" + if err := s.commit(snap); err != nil { + t.Fatalf("commit: %v", err) + } + got, err := s.load() + if err != nil { + t.Fatalf("load: %v", err) + } + loaded := got[sb.ID] + if loaded == nil { + t.Fatal("detached snapshot omitted claim") + } + if !slices.Equal(loaded.Volumes, []types.Volume{{Name: "dataset-a", Mount: "/datasets/a"}}) { + t.Errorf("volumes %v, want detached snapshot", loaded.Volumes) + } +} + func TestStoreLoadMissingFile(t *testing.T) { got, err := newClaimStore(t.TempDir()).load() if err != nil { diff --git a/sandboxd/pool/drain_test.go b/sandboxd/pool/drain_test.go index f20277e1..2332633b 100644 --- a/sandboxd/pool/drain_test.go +++ b/sandboxd/pool/drain_test.go @@ -26,10 +26,10 @@ func TestDrainRefusesClaimsTrimsWarmAndUncordonRefills(t *testing.T) { m.Drain(t.Context()) - if _, err := m.ClaimWarm(t.Context(), testKey, 0, "", ""); !errors.Is(err, ErrQuota) { + if _, err := m.ClaimWarm(t.Context(), testKey, 0, "", "", nil); !errors.Is(err, ErrQuota) { t.Fatalf("ClaimWarm during drain: %v, want ErrQuota", err) } - if _, err := m.ClaimProvision(t.Context(), testKey, 0, "", ""); !errors.Is(err, ErrQuota) { + if _, err := m.ClaimProvision(t.Context(), testKey, 0, "", "", nil); !errors.Is(err, ErrQuota) { t.Fatalf("ClaimProvision during drain: %v, want ErrQuota", err) } infos, g := m.Info() diff --git a/sandboxd/pool/egress.go b/sandboxd/pool/egress.go index f22622f5..5bd9544a 100644 --- a/sandboxd/pool/egress.go +++ b/sandboxd/pool/egress.go @@ -48,37 +48,6 @@ var ( } ) -// newEgressDialer builds the proxy's upstream dialer: internal targets are -// blocked (the proxy must not be an SSRF), then exactly the node-named -// prefixes re-admitted. -func newEgressDialer(allow []netip.Prefix) *net.Dialer { - return &net.Dialer{Control: func(_, address string, _ syscall.RawConn) error { - host, _, err := net.SplitHostPort(address) - if err != nil { - return err - } - ip, err := netip.ParseAddr(host) - if err != nil { - return fmt.Errorf("egress: unresolved address %q", host) - } - ip = ip.Unmap() - if nat64Range.Contains(ip) { - b := ip.As16() - ip = netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]}) - } - // After the NAT64 unwrap (a v4-in-v6 target matches as the v4 it is), - // before the block, so a named prefix wins. - if slices.ContainsFunc(allow, func(p netip.Prefix) bool { return p.Contains(ip) }) { - return nil - } - if !ip.IsGlobalUnicast() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || - slices.ContainsFunc(internalRanges, func(p netip.Prefix) bool { return p.Contains(ip) }) { - return fmt.Errorf("egress: blocked internal address %s", ip) - } - return nil - }} -} - // egressListener is one sandbox's egress accept point: an http.Server serving // egress.Proxy over the per-sandbox UDS the VMM connects when the guest dials // CID2:egressPort. @@ -233,6 +202,37 @@ func (m *Manager) effectivePolicy(sb *types.Sandbox) (egress.Evaluator, bool) { } } +// newEgressDialer builds the proxy's upstream dialer: internal targets are +// blocked (the proxy must not be an SSRF), then exactly the node-named +// prefixes re-admitted. +func newEgressDialer(allow []netip.Prefix) *net.Dialer { + return &net.Dialer{Control: func(_, address string, _ syscall.RawConn) error { + host, _, err := net.SplitHostPort(address) + if err != nil { + return err + } + ip, err := netip.ParseAddr(host) + if err != nil { + return fmt.Errorf("egress: unresolved address %q", host) + } + ip = ip.Unmap() + if nat64Range.Contains(ip) { + b := ip.As16() + ip = netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]}) + } + // After the NAT64 unwrap (a v4-in-v6 target matches as the v4 it is), + // before the block, so a named prefix wins. + if slices.ContainsFunc(allow, func(p netip.Prefix) bool { return p.Contains(ip) }) { + return nil + } + if !ip.IsGlobalUnicast() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || + slices.ContainsFunc(internalRanges, func(p netip.Prefix) bool { return p.Contains(ip) }) { + return fmt.Errorf("egress: blocked internal address %s", ip) + } + return nil + }} +} + // parsePrefixes turns the allow-list into prefixes; config validation already // rejected malformed entries, so a leftover is dropped rather than failing dials. func parsePrefixes(cidrs []string) []netip.Prefix { diff --git a/sandboxd/pool/fork.go b/sandboxd/pool/fork.go index 8326270d..d7613620 100644 --- a/sandboxd/pool/fork.go +++ b/sandboxd/pool/fork.go @@ -5,7 +5,6 @@ import ( "fmt" "os" "path/filepath" - "strings" "time" "github.com/cocoonstack/sandbox/sandboxd/types" @@ -25,6 +24,9 @@ func (m *Manager) Fork(ctx context.Context, id string, cred Cred, count int, ttl if count < 1 || count > m.maxFork { return nil, fmt.Errorf("%w: %d not in 1..%d", ErrBadCount, count, m.maxFork) } + if hasAppliedVolumes(sb) { + return nil, ErrVolumeCapture + } if !sb.Key.Capturable() { return nil, ErrNoEgressFork } @@ -69,16 +71,16 @@ func (m *Manager) forkClones(ctx context.Context, sb *types.Sandbox, count int) // forkSource decides and captures the fork source under the transition lock // (a racing hibernate cannot swap the snapshot out from under the fan-out), // returning the per-child provisioner and its cleanup. -func (m *Manager) forkSource(ctx context.Context, sb *types.Sandbox) (func(string) (types.VMRecord, error), func(), error) { +func (m *Manager) forkSource(ctx context.Context, sb *types.Sandbox) (vmProvisioner, func(), error) { sb.Transition.Lock() defer sb.Transition.Unlock() if sb.HibernateSnap == "" { - snap := forkPrefix + strings.TrimPrefix(sb.VMName, vmPrefix) + "-" + randHex(3) - if err := m.eng.SnapshotSave(ctx, sb.VMName, snap); err != nil { + snap, cleanup, err := m.sourceSnap(ctx, sb) + if err != nil { return nil, nil, err } return func(name string) (types.VMRecord, error) { return m.eng.CloneSnap(ctx, snap, name, sb.Key) }, - func() { m.dropSnap(ctx, snap) }, nil + cleanup, nil } dir, err := os.MkdirTemp(m.dataDir, "fork-") if err != nil { diff --git a/sandboxd/pool/fork_test.go b/sandboxd/pool/fork_test.go index 552f7792..49d8412c 100644 --- a/sandboxd/pool/fork_test.go +++ b/sandboxd/pool/fork_test.go @@ -230,7 +230,7 @@ func TestForkChildrenInheritTenantAndQuota(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) m.tenantMax = map[string]int{"acme": 3} - parent, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "") + parent, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil) if err != nil { t.Fatalf("claim: %v", err) } diff --git a/sandboxd/pool/hibernate.go b/sandboxd/pool/hibernate.go index 0fbc7b15..c97994aa 100644 --- a/sandboxd/pool/hibernate.go +++ b/sandboxd/pool/hibernate.go @@ -40,6 +40,9 @@ func (m *Manager) WakeAgentSocket(ctx context.Context, id, token string) (string // hibernateLocked is Hibernate's body; the caller holds sb.Transition. func (m *Manager) hibernateLocked(ctx context.Context, sb *types.Sandbox) error { + if hasAppliedVolumes(sb) { + return ErrVolumeCapture + } if sb.Key.Net == types.NetEgress { // cocoon resumes the guest before its fresh tap can be re-locked, so a // woken egress guest would egress unlocked; keep the lane live instead. @@ -200,9 +203,9 @@ func (m *Manager) idleOnce(ctx context.Context) { for _, sb := range m.claimed { idle := m.idleDefault if p, pooled := m.activePool(sb.Key); pooled { - idle = p.idle // pooled keys never take the node default + idle = p.idle } - if idle <= 0 || sb.HibernateSnap != "" || sb.ArchiveCk != "" || now.Sub(sb.LastSeen()) < idle { + if idle <= 0 || hasAppliedVolumes(sb) || sb.HibernateSnap != "" || sb.ArchiveCk != "" || now.Sub(sb.LastSeen()) < idle { continue } victims = append(victims, victim{sb.ID, sb.Token}) diff --git a/sandboxd/pool/idle_test.go b/sandboxd/pool/idle_test.go index f96b5ee5..71f59190 100644 --- a/sandboxd/pool/idle_test.go +++ b/sandboxd/pool/idle_test.go @@ -46,7 +46,7 @@ func TestIdleOncePolicyScope(t *testing.T) { // An unpooled key (template claim shape) takes the node default. unpooled := types.PoolKey{Template: "tpl:v1", Net: types.NetNone, Size: types.SizeSmall} - sb2, err := m.ClaimProvision(t.Context(), unpooled, time.Hour, "", "") + sb2, err := m.ClaimProvision(t.Context(), unpooled, time.Hour, "", "", nil) if err != nil { t.Fatalf("provision: %v", err) } diff --git a/sandboxd/pool/journal.go b/sandboxd/pool/journal.go index e3234443..df640224 100644 --- a/sandboxd/pool/journal.go +++ b/sandboxd/pool/journal.go @@ -23,6 +23,7 @@ type usageEvent struct { VMName string `json:"vm,omitempty"` KeyHash string `json:"key,omitempty"` // claim only Tenant string `json:"tenant,omitempty"` // claim only + Volumes []string `json:"volumes,omitempty"` // claim only Children []string `json:"children,omitempty"` // fork only Reference string `json:"ref,omitempty"` // promote: template; checkpoint: ckpt id } diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 4d063ef5..07525ae5 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -43,7 +43,7 @@ const ( coldProbeTimeout = 90 * time.Second // A refill has no caller waiting and holds a target-accounting slot for // its whole probe; a clone answers in ~1s even saturated, so a silent one - // is replaced, not waited on. Claims keep the generous deadline. + // is replaced, not waited on. warmProbeTimeout = 5 * time.Second // One list; a wrong answer only costs an extra sweep. removeVerifyTimeout = 15 * time.Second @@ -82,6 +82,8 @@ const ( var ( ErrBadKey = errors.New("invalid pool key") ErrBadCount = errors.New("invalid fork count") + ErrBadVolume = errors.New("invalid volume request") + ErrVolumeUnavailable = fmt.Errorf("%w: unknown or unavailable volume", ErrBadVolume) ErrNoWarm = errors.New("no warm sandbox for key") ErrUnknownSandbox = errors.New("unknown sandbox or bad token") ErrUnknownTemplate = errors.New("unknown promoted template") @@ -90,6 +92,7 @@ var ( ErrNoEgress = errors.New("node has no egress attachment (bridge or network)") ErrNoEgressHibernate = errors.New("egress-lane sandboxes do not hibernate") ErrNoEgressFork = errors.New("egress-lane sandboxes cannot fork, checkpoint, or promote: a resumed guest egresses before its fresh tap can be locked") + ErrVolumeCapture = errors.New("sandboxes with volumes cannot hibernate, fork, checkpoint, or promote") ErrQuota = errors.New("node claim quota reached") errWokeMeanwhile = errors.New("woke between sweep and hibernate") @@ -113,16 +116,19 @@ type Engine interface { Probe(ctx context.Context, vsockSocket string, timeout time.Duration) error DialGuestPort(ctx context.Context, vsockSocket string, port uint16) (net.Conn, error) InstallCACert(ctx context.Context, vsockSocket string, certPEM []byte) error + DiskAttach(ctx context.Context, vmName string, spec engine.VolumeSpec) error + MountVolume(ctx context.Context, vsockSocket, name, mount string) error } // SandboxSummary is the ops view of one live claim — no tokens. type SandboxSummary struct { - ID string `json:"id"` - Key types.PoolKey `json:"key"` - Deadline time.Time `json:"deadline"` - Hibernated bool `json:"hibernated"` - Archived bool `json:"archived,omitempty"` - FromCheckpoint string `json:"from_checkpoint,omitempty"` + ID string `json:"id"` + Key types.PoolKey `json:"key"` + Deadline time.Time `json:"deadline"` + Hibernated bool `json:"hibernated"` + Archived bool `json:"archived,omitempty"` + FromCheckpoint string `json:"from_checkpoint,omitempty"` + Volumes []types.Volume `json:"volumes,omitempty"` // ClaimRef echoes the caller reference recorded at claim time (the // aggregated apiserver's k8s "/"), so the operator index // can map this sandbox back to the name it was claimed under. Empty for @@ -238,6 +244,7 @@ type Manager struct { egress bool maxFork int store *claimStore + volumes map[string]catalogVolume poolStore *poolStore configSeedHash string // config pools' hash, to warn when a file edit is overridden @@ -374,6 +381,7 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg lockEgress: len(cfg.Bridges) > 0, maxFork: maxFork, store: newClaimStore(cfg.DataDir), + volumes: make(map[string]catalogVolume, len(cfg.Volumes)), poolStore: newPoolStore(cfg.DataDir), pools: make(map[types.PoolKey]*pool, len(cfg.Pools)), claimed: map[string]*types.Sandbox{}, @@ -395,6 +403,12 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg refillKick: make(chan struct{}, 1), healSem: make(chan struct{}, maxConcurrentHeals), } + for _, volume := range cfg.Volumes { + m.volumes[volume.Name] = catalogVolume{ + disk: engine.VolumeSpec{Name: volume.Name, Path: volume.Path, DirectIO: volume.DirectIO}, + tenants: slices.Clone(volume.Tenants), + } + } if err := os.MkdirAll(m.goldensDir(), 0o750); err != nil { return nil, fmt.Errorf("create goldens dir: %w", err) } @@ -568,12 +582,15 @@ func (m *Manager) Info() ([]PoolInfo, Gauges) { return pools, g } -// Sandboxes lists the live claims, for the operator index. -func (m *Manager) Sandboxes() []SandboxSummary { +// Sandboxes lists live claims visible to tenant; empty tenant means root. +func (m *Manager) Sandboxes(tenant string) []SandboxSummary { m.mu.Lock() defer m.mu.Unlock() out := make([]SandboxSummary, 0, len(m.claimed)) for _, sb := range m.claimed { + if !tenantOwns(tenant, sb.Tenant) { + continue + } out = append(out, summarize(sb)) } slices.SortFunc(out, func(a, b SandboxSummary) int { return strings.Compare(a.ID, b.ID) }) @@ -655,6 +672,7 @@ func summarize(sb *types.Sandbox) SandboxSummary { ID: sb.ID, Key: sb.Key, Deadline: sb.Deadline, Hibernated: sb.HibernateSnap != "", Archived: sb.ArchiveCk != "", FromCheckpoint: sb.FromCheckpoint, ClaimRef: sb.ClaimRef, + Volumes: slices.Clone(sb.Volumes), } } @@ -694,6 +712,10 @@ func tenantOwns(tenant, owner string) bool { return tenant == "" || tenant == owner } +func hasAppliedVolumes(sb *types.Sandbox) bool { + return len(sb.Volumes) > 0 +} + // logSweepResult reports one background-sweep outcome; benign races stay silent. func logSweepResult(ctx context.Context, logger *log.Fields, err error, okMsg, failMsg string) { switch { diff --git a/sandboxd/pool/pool_test.go b/sandboxd/pool/pool_test.go index a71d36f8..9247688d 100644 --- a/sandboxd/pool/pool_test.go +++ b/sandboxd/pool/pool_test.go @@ -705,9 +705,9 @@ func newTestManager(t *testing.T, eng *fakeEngine, pools ...config.PoolSpec) *Ma // claimAny composes warm-then-provision the way the server does around the // redirect decision; production has no single-call form. func claimAny(ctx context.Context, m *Manager, key types.PoolKey, ttl time.Duration) (*types.Sandbox, error) { - sb, err := m.ClaimWarm(ctx, key, ttl, "", "") + sb, err := m.ClaimWarm(ctx, key, ttl, "", "", nil) if errors.Is(err, ErrNoWarm) { - return m.ClaimProvision(ctx, key, ttl, "", "") + return m.ClaimProvision(ctx, key, ttl, "", "", nil) } return sb, err } @@ -762,6 +762,9 @@ type fakeEngine struct { colds []string removes []string probeTimeouts []time.Duration + volumeSpecs []engine.VolumeSpec + volumeMounts []types.Volume + volumeOps []string hibernates, restores, snapRemoves []string snapSaves, exports, snapshots []string @@ -769,6 +772,9 @@ type fakeEngine struct { caInstalls []string // vsock sockets InstallCACert was called on staleReconciles []string // VM names ReconcileStaleCreate was called on installCAErr error + diskAttachErr error + diskAttachCancel context.CancelFunc + mountVolumeErr error stopped map[string]bool creating map[string]bool // VMs List reports in the creating state staleOutcome engine.StaleCreateOutcome @@ -808,6 +814,7 @@ func (f *fakeEngine) RunCold(_ context.Context, name string, _ types.PoolKey) (t f.mu.Lock() defer f.mu.Unlock() f.colds = append(f.colds, name) + f.volumeOps = append(f.volumeOps, "provision") if f.runColdErr != nil { return types.VMRecord{}, f.runColdErr } @@ -967,6 +974,7 @@ func (f *fakeEngine) List(_ context.Context, filters ...string) ([]types.VMRecor func (f *fakeEngine) Probe(_ context.Context, _ string, timeout time.Duration) error { f.mu.Lock() f.probeTimeouts = append(f.probeTimeouts, timeout) + f.volumeOps = append(f.volumeOps, "probe") stall := f.probeStall err := f.probeErr f.mu.Unlock() @@ -987,10 +995,30 @@ func (f *fakeEngine) InstallCACert(_ context.Context, vsockSocket string, _ []by return f.installCAErr } +func (f *fakeEngine) DiskAttach(_ context.Context, _ string, spec engine.VolumeSpec) error { + f.mu.Lock() + defer f.mu.Unlock() + f.volumeSpecs = append(f.volumeSpecs, spec) + f.volumeOps = append(f.volumeOps, "attach:"+spec.Name) + if f.diskAttachCancel != nil { + f.diskAttachCancel() + } + return f.diskAttachErr +} + +func (f *fakeEngine) MountVolume(_ context.Context, _, name, mount string) error { + f.mu.Lock() + defer f.mu.Unlock() + f.volumeMounts = append(f.volumeMounts, types.Volume{Name: name, Mount: mount}) + f.volumeOps = append(f.volumeOps, "mount:"+name+":"+mount) + return f.mountVolumeErr +} + func (f *fakeEngine) clone(from, name string) (types.VMRecord, error) { f.mu.Lock() f.clones = append(f.clones, name) f.cloneFroms = append(f.cloneFroms, from) + f.volumeOps = append(f.volumeOps, "provision") call := len(f.clones) stall, err, failNth := f.cloneStall, f.cloneErr, f.cloneFailNth f.mu.Unlock() diff --git a/sandboxd/pool/promote_test.go b/sandboxd/pool/promote_test.go index d0fad7ec..916454e0 100644 --- a/sandboxd/pool/promote_test.go +++ b/sandboxd/pool/promote_test.go @@ -200,20 +200,20 @@ func TestResolveGoldenSkipsPromotedEgressTemplate(t *testing.T) { if _, err = m.commitTemplate(t.Context(), staging, id, ""); err != nil { t.Fatalf("seed template: %v", err) } - dir, _, release, err := m.resolveGolden(t.Context(), egKey) + golden, err := m.resolveGolden(t.Context(), egKey) if err != nil { t.Fatalf("resolveGolden: %v", err) } - release() - if dir != "" { - t.Errorf("resolveGolden resumed a promoted egress template %q; want cold-boot", dir) + golden.release() + if golden.dir != "" { + t.Errorf("resolveGolden resumed a promoted egress template %q; want cold-boot", golden.dir) } } func TestTemplateTenantScopedDelete(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - parent, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "") + parent, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil) if err != nil { t.Fatalf("claim: %v", err) } @@ -269,7 +269,7 @@ func TestPromoteFailsClosedOnMetaError(t *testing.T) { } eng := newFakeEngine() m := newTestManager(t, eng) - a, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "") + a, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil) if err != nil { t.Fatalf("claim: %v", err) } @@ -292,7 +292,7 @@ func TestPromoteRefusesCrossTenantOverwrite(t *testing.T) { m := newTestManager(t, eng) claim := func(tenant string) *types.Sandbox { t.Helper() - sb, err := m.ClaimProvision(t.Context(), testKey, time.Hour, tenant, "") + sb, err := m.ClaimProvision(t.Context(), testKey, time.Hour, tenant, "", nil) if err != nil { t.Fatalf("claim %q: %v", tenant, err) } diff --git a/sandboxd/pool/refill.go b/sandboxd/pool/refill.go index 6818daf4..bc1099ea 100644 --- a/sandboxd/pool/refill.go +++ b/sandboxd/pool/refill.go @@ -17,6 +17,8 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) +type vmProvisioner func(name string) (types.VMRecord, error) + // kickRefill nudges Run past its refill ticker after a warm claim; the // 1-buffered channel coalesces bursts and never blocks the claim path. func (m *Manager) kickRefill() { @@ -320,7 +322,7 @@ func (m *Manager) provision(ctx context.Context, key types.PoolKey, golden strin }) } -func (m *Manager) provisionVM(ctx context.Context, key types.PoolKey, probeTimeout time.Duration, create func(name string) (types.VMRecord, error)) (*types.Sandbox, error) { +func (m *Manager) provisionVM(ctx context.Context, key types.PoolKey, probeTimeout time.Duration, create vmProvisioner) (*types.Sandbox, error) { sb, err := m.startVM(ctx, key, create) if err != nil { return nil, err @@ -328,7 +330,7 @@ func (m *Manager) provisionVM(ctx context.Context, key types.PoolKey, probeTimeo return m.readyVM(ctx, sb, time.Now().Add(probeTimeout)) } -func (m *Manager) startVM(ctx context.Context, key types.PoolKey, create func(name string) (types.VMRecord, error)) (*types.Sandbox, error) { +func (m *Manager) startVM(ctx context.Context, key types.PoolKey, create vmProvisioner) (*types.Sandbox, error) { name := vmName(key) rec, err := create(name) if err != nil { @@ -361,7 +363,7 @@ func (m *Manager) readyBounded(ctx context.Context, sb *types.Sandbox, deadline return m.readyVM(probeCtx, sb, deadline) } -func (m *Manager) cloneBatch(ctx context.Context, count int, key types.PoolKey, create func(string) (types.VMRecord, error)) ([]*types.Sandbox, error) { +func (m *Manager) cloneBatch(ctx context.Context, count int, key types.PoolKey, create vmProvisioner) ([]*types.Sandbox, error) { children := make([]*types.Sandbox, count) errs := make([]error, count) var wg sync.WaitGroup diff --git a/sandboxd/pool/telemetry_test.go b/sandboxd/pool/telemetry_test.go index b3287117..d573cf32 100644 --- a/sandboxd/pool/telemetry_test.go +++ b/sandboxd/pool/telemetry_test.go @@ -99,22 +99,22 @@ func TestTenantQuotaBindsPerTenant(t *testing.T) { m := newTestManager(t, eng) m.tenantMax = map[string]int{"acme": 1, "beta": 2} - first, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "") + first, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil) if err != nil { t.Fatalf("acme claim: %v", err) } if first.Tenant != "acme" { t.Errorf("tenant %q, want acme", first.Tenant) } - if _, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", ""); !errors.Is(err, ErrQuota) { + if _, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil); !errors.Is(err, ErrQuota) { t.Fatalf("acme past its cap: %v, want ErrQuota", err) } // The node-wide cap (unset here) stays untouched: other tenants and root // keep claiming while acme is full. - if _, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "beta", ""); err != nil { + if _, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "beta", "", nil); err != nil { t.Errorf("beta claim while acme is at cap: %v", err) } - if _, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "", ""); err != nil { + if _, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "", "", nil); err != nil { t.Errorf("root claim while acme is at cap: %v", err) } counts := m.TenantClaims() @@ -124,7 +124,7 @@ func TestTenantQuotaBindsPerTenant(t *testing.T) { if err := m.Release(t.Context(), first.ID, Cred{Token: first.Token}); err != nil { t.Fatalf("Release: %v", err) } - if _, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", ""); err != nil { + if _, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil); err != nil { t.Errorf("acme claim after release: %v", err) } } @@ -132,7 +132,7 @@ func TestTenantQuotaBindsPerTenant(t *testing.T) { func TestTenantStampedInJournals(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - sb, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "") + sb, err := m.ClaimProvision(t.Context(), testKey, time.Hour, "acme", "", nil) if err != nil { t.Fatalf("claim: %v", err) } @@ -179,7 +179,7 @@ func TestSandboxesIndexOmitsTokens(t *testing.T) { sb.ClaimRef = "ns/workload" m.mu.Unlock() - list := m.Sandboxes() + list := m.Sandboxes("") if len(list) != 1 || list[0].ID != sb.ID || list[0].ClaimRef != "ns/workload" || list[0].Hibernated { t.Fatalf("index %+v", list) } diff --git a/sandboxd/pool/template.go b/sandboxd/pool/template.go index 407b1573..ccb9c45d 100644 --- a/sandboxd/pool/template.go +++ b/sandboxd/pool/template.go @@ -36,6 +36,9 @@ func (m *Manager) Promote(ctx context.Context, id string, cred Cred, template, t if !types.NameRe.MatchString(template) { return types.PoolKey{}, "", fmt.Errorf("%w: template %q must match %s", ErrBadKey, template, types.NameRe) } + if hasAppliedVolumes(sb) { + return types.PoolKey{}, "", ErrVolumeCapture + } if !sb.Key.Capturable() { return types.PoolKey{}, "", ErrNoEgressFork } @@ -148,6 +151,12 @@ func (m *Manager) HasGolden(ctx context.Context, key types.PoolKey) bool { if pooled { return true } + return m.HasPromotedTemplate(ctx, key) +} + +// HasPromotedTemplate reports whether the template store contains key. Unlike +// HasGolden it does not count a configured pool golden. +func (m *Manager) HasPromotedTemplate(ctx context.Context, key types.PoolKey) bool { id := store.TemplateID(key.Hash()) m.tplMu.Lock() _, cached := m.tplSet[id] @@ -250,11 +259,18 @@ func (m *Manager) checkTemplateOwner(ctx context.Context, id, tenant string) err return nil } +type goldenResolution struct { + dir string + templateDigest string + promoted bool + release func() +} + // resolveGolden resolves a key's clone source: the configured pool's local // golden (no release), else a promoted template fetched from the store; // empty dir cold-boots. Only a true absence cold-boots — a backend failure // propagates rather than silently booting a template name as an image ref. -func (m *Manager) resolveGolden(ctx context.Context, key types.PoolKey) (string, string, func(), error) { +func (m *Manager) resolveGolden(ctx context.Context, key types.PoolKey) (goldenResolution, error) { m.mu.Lock() var dir string if p := m.pools[key]; p != nil { @@ -262,10 +278,10 @@ func (m *Manager) resolveGolden(ctx context.Context, key types.PoolKey) (string, } m.mu.Unlock() if dir != "" { - return dir, "", func() {}, nil + return goldenResolution{dir: dir, release: func() {}}, nil } if key.Net == types.NetEgress { - return "", "", func() {}, nil // never resume a live-captured template on the egress lane; cold-boot instead + return goldenResolution{release: func() {}}, nil // never resume a live-captured template on the egress lane; cold-boot instead } id := store.TemplateID(key.Hash()) l := m.recLock(id) @@ -275,18 +291,23 @@ func (m *Manager) resolveGolden(ctx context.Context, key types.PoolKey) (string, l.RUnlock() m.recDone(id) if errors.Is(err, store.ErrNotFound) { - return "", "", func() {}, nil + return goldenResolution{release: func() {}}, nil } - return "", "", func() {}, err + return goldenResolution{release: func() {}}, err } var rec templateRecord if err := json.Unmarshal(meta, &rec); err != nil { release() l.RUnlock() m.recDone(id) - return "", "", func() {}, fmt.Errorf("decode template metadata: %w", err) - } - return dir, digest, func() { release(); l.RUnlock(); m.recDone(id) }, nil + return goldenResolution{release: func() {}}, fmt.Errorf("decode template metadata: %w", err) + } + return goldenResolution{ + dir: dir, + templateDigest: digest, + promoted: true, + release: func() { release(); l.RUnlock(); m.recDone(id) }, + }, nil } // publishTemplate exports snap into the store under the key's template id. diff --git a/sandboxd/pool/volume.go b/sandboxd/pool/volume.go new file mode 100644 index 00000000..30e22daf --- /dev/null +++ b/sandboxd/pool/volume.go @@ -0,0 +1,143 @@ +package pool + +import ( + "context" + "fmt" + "os" + "slices" + "strings" + + "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +type catalogVolume struct { + disk engine.VolumeSpec + tenants []string +} + +func (v catalogVolume) allowed(tenant string) bool { + return tenant == "" || len(v.tenants) == 0 || slices.Contains(v.tenants, tenant) +} + +type resolvedVolume struct { + disk engine.VolumeSpec + applied types.Volume +} + +// Volumes reports the caller-visible fleet catalog, projected through this +// node's ACL metadata and local path state. Empty tenant means root. +func (m *Manager) Volumes(tenant string, holders map[string]int) []types.VolumeInfo { + infos := make([]types.VolumeInfo, 0, len(m.volumes)) + for name, volume := range m.volumes { + if !volume.allowed(tenant) { + continue + } + info := types.VolumeInfo{Name: name, DefaultMount: types.DefaultVolumeMount(name), Nodes: holders[name]} + if st, err := os.Stat(volume.disk.Path); err == nil { + info.SizeBytes = st.Size() + info.Available = true + info.Nodes = max(info.Nodes, 1) + } + infos = append(infos, info) + } + // Root can safely see peer-only names. Tenant callers need the local catalog + // metadata so the access list can be applied without gossiping it. + if tenant == "" { + for name, nodes := range holders { + if _, ok := m.volumes[name]; !ok { + infos = append(infos, types.VolumeInfo{ + Name: name, + DefaultMount: types.DefaultVolumeMount(name), + Nodes: nodes, + }) + } + } + } + slices.SortFunc(infos, func(a, b types.VolumeInfo) int { return strings.Compare(a.Name, b.Name) }) + return infos +} + +// VolumeNames returns the sorted names this node can currently serve. +func (m *Manager) VolumeNames() []string { + names := make([]string, 0, len(m.volumes)) + for name, volume := range m.volumes { + if _, err := os.Stat(volume.disk.Path); err == nil { + names = append(names, name) + } + } + slices.Sort(names) + return names +} + +// VolumePlacement checks catalog access and reports whether every named image +// is currently available on this node. Empty tenant means root. +func (m *Manager) VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, error) { + if err := m.validate(key); err != nil { + return false, err + } + if len(names) == 0 { + return true, nil + } + local := true + for _, name := range names { + volume, ok := m.volumes[name] + if !ok { + if tenant == "" { + local = false + continue + } + return false, ErrVolumeUnavailable + } + if !volume.allowed(tenant) { + return false, ErrVolumeUnavailable + } + if _, err := os.Stat(volume.disk.Path); err != nil { + local = false + } + } + return local, nil +} + +func (m *Manager) resolveVolumes(key types.PoolKey, tenant string, requested []types.Volume) ([]resolvedVolume, error) { + if len(requested) == 0 { + return nil, nil + } + applied, err := types.ValidateVolumes(requested) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrBadVolume, err) + } + if key.Engine != types.EngineCH { + return nil, fmt.Errorf("%w: volumes require engine ch", ErrBadVolume) + } + resolved := make([]resolvedVolume, 0, len(applied)) + for _, volume := range applied { + entry, ok := m.volumes[volume.Name] + if !ok || !entry.allowed(tenant) { + return nil, ErrVolumeUnavailable + } + if _, statErr := os.Stat(entry.disk.Path); statErr != nil { + return nil, fmt.Errorf("volume %q path %q: %w", volume.Name, entry.disk.Path, statErr) + } + resolved = append(resolved, resolvedVolume{disk: entry.disk, applied: volume}) + } + return resolved, nil +} + +func (m *Manager) applyVolumes(ctx context.Context, sb *types.Sandbox, volumes []resolvedVolume) error { + if len(volumes) == 0 { + return nil + } + applied := make([]types.Volume, len(volumes)) + for i, volume := range volumes { + if err := m.eng.DiskAttach(ctx, sb.VMName, volume.disk); err != nil { + return fmt.Errorf("attach volume %q: %w", volume.applied.Name, err) + } + if err := m.eng.MountVolume(ctx, sb.VsockSocket, volume.applied.Name, volume.applied.Mount); err != nil { + return fmt.Errorf("setup volume %q: %w", volume.applied.Name, err) + } + applied[i] = volume.applied + } + sb.Volumes = applied + return nil +} diff --git a/sandboxd/pool/volume_capture_test.go b/sandboxd/pool/volume_capture_test.go new file mode 100644 index 00000000..aaf29f38 --- /dev/null +++ b/sandboxd/pool/volume_capture_test.go @@ -0,0 +1,131 @@ +package pool + +import ( + "errors" + "slices" + "testing" + "time" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func TestVolumeClaimRejectsDirectCapture(t *testing.T) { + eng := newFakeEngine() + m := newTestManager(t, eng) + sb := mustClaim(t, m, testKey) + sb.Volumes = []types.Volume{{Name: "dataset", Mount: "/datasets"}} + + tests := []struct { + name string + run func(*testing.T) error + }{ + {"direct hibernate", func(t *testing.T) error { return m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) }}, + {"fork", func(t *testing.T) error { + _, err := m.Fork(t.Context(), sb.ID, Cred{Token: sb.Token}, 1, time.Hour) + return err + }}, + {"checkpoint", func(t *testing.T) error { + _, err := m.Checkpoint(t.Context(), sb.ID, Cred{Token: sb.Token}, "", "") + return err + }}, + {"promote", func(t *testing.T) error { + _, _, err := m.Promote(t.Context(), sb.ID, Cred{Token: sb.Token}, "volume-source", "") + return err + }}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if err := tt.run(t); !errors.Is(err, ErrVolumeCapture) { + t.Errorf("error %v, want ErrVolumeCapture", err) + } + }) + } + if len(eng.hibernates) != 0 || len(eng.snapSaves) != 0 || len(eng.exports) != 0 { + t.Errorf("capture reached engine: hibernates=%v snapshots=%v exports=%v", eng.hibernates, eng.snapSaves, eng.exports) + } +} + +func TestVolumeClaimRunningWakeUnchanged(t *testing.T) { + eng := newFakeEngine() + m := newTestManager(t, eng) + sb := mustClaim(t, m, testKey) + sb.Volumes = []types.Volume{{Name: "dataset", Mount: "/datasets"}} + + sock, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) + if err != nil { + t.Fatalf("WakeAgentSocket: %v", err) + } + if sock != sb.VsockSocket { + t.Errorf("socket %q, want %q", sock, sb.VsockSocket) + } + if len(eng.restores) != 0 { + t.Errorf("running wake restored %v, want no restore", eng.restores) + } +} + +func TestReconcileRetainsVolumeCaptureGateWithoutCatalog(t *testing.T) { + eng := newFakeEngine() + dataDir := t.TempDir() + volumePath := writeVolumeImage(t, "dataset.img", "data") + m, err := NewManager(t.Context(), &config.Config{ + DataDir: dataDir, + Volumes: []config.VolumeSpec{{Name: "dataset", Path: volumePath, DirectIO: "off"}}, + }, eng, testSecrets(t)) + if err != nil { + t.Fatalf("setup manager: %v", err) + } + sb, err := m.ClaimProvision(t.Context(), testKey, 0, "", "", []types.Volume{{Name: "dataset", Mount: "/datasets"}}) + if err != nil { + t.Fatalf("ClaimProvision: %v", err) + } + + m2 := newTestManagerAt(t, eng, dataDir) + if err := m2.Reconcile(t.Context()); err != nil { + t.Fatalf("Reconcile: %v", err) + } + for _, tt := range []struct { + name string + run func() error + }{ + {"after restart hibernate", func() error { return m2.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) }}, + {"after restart fork", func() error { + _, forkErr := m2.Fork(t.Context(), sb.ID, Cred{Token: sb.Token}, 1, time.Hour) + return forkErr + }}, + {"after restart checkpoint", func() error { + _, checkpointErr := m2.Checkpoint(t.Context(), sb.ID, Cred{Token: sb.Token}, "", "") + return checkpointErr + }}, + {"after restart promote", func() error { + _, _, promoteErr := m2.Promote(t.Context(), sb.ID, Cred{Token: sb.Token}, "volume-source", "") + return promoteErr + }}, + } { + t.Run(tt.name, func(t *testing.T) { + if captureErr := tt.run(); !errors.Is(captureErr, ErrVolumeCapture) { + t.Errorf("error %v, want ErrVolumeCapture", captureErr) + } + }) + } + if got, ok := m2.claimed[sb.ID]; !ok || !slices.Equal(got.Volumes, []types.Volume{{Name: "dataset", Mount: "/datasets"}}) { + t.Errorf("reconciled claim=%+v, want persisted volume", got) + } + if len(eng.hibernates) != 0 || len(eng.snapSaves) != 0 || len(eng.exports) != 0 { + t.Errorf("capture reached engine: hibernates=%v snapshots=%v exports=%v", eng.hibernates, eng.snapSaves, eng.exports) + } +} + +func TestIdleOnceSkipsVolumeClaim(t *testing.T) { + eng := newFakeEngine() + m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, IdleHibernateSeconds: 1}) + sb := mustClaim(t, m, testKey) + sb.Volumes = []types.Volume{{Name: "dataset", Mount: "/datasets"}} + backdate(m, sb, 2*time.Second) + + m.idleOnce(t.Context()) + waitFor(t, func() bool { return !m.idleSweep.Load() }) + if len(eng.hibernates) != 0 || hibernated(m) != 0 { + t.Errorf("idle sweep hibernated volume claim: engine=%v count=%d", eng.hibernates, hibernated(m)) + } +} diff --git a/sandboxd/pool/volume_test.go b/sandboxd/pool/volume_test.go new file mode 100644 index 00000000..1b9d761b --- /dev/null +++ b/sandboxd/pool/volume_test.go @@ -0,0 +1,432 @@ +package pool + +import ( + "context" + "encoding/json" + "errors" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/cocoonstack/sandbox/sandboxd/config" + "github.com/cocoonstack/sandbox/sandboxd/types" +) + +func TestClaimProvisionAppliesVolumesInOrder(t *testing.T) { + first := writeVolumeImage(t, "imagenet.img", "first") + second := writeVolumeImage(t, "weights.img", "second") + eng := newFakeEngine() + m := newVolumeManager(t, eng, []config.VolumeSpec{ + {Name: "imagenet", Path: first, DirectIO: "off"}, + {Name: "weights", Path: second, DirectIO: "on"}, + }) + requested := []types.Volume{{Name: "weights", Mount: "/models"}, {Name: "imagenet"}} + wantApplied := []types.Volume{ + {Name: "weights", Mount: "/models"}, + {Name: "imagenet", Mount: "/volumes/imagenet"}, + } + + sb, err := m.ClaimProvision(t.Context(), testKey, 0, "", "", requested) + if err != nil { + t.Fatalf("ClaimProvision: %v", err) + } + if !slices.Equal(sb.Volumes, wantApplied) { + t.Errorf("volumes=%v, want %v", sb.Volumes, wantApplied) + } + wantOps := []string{ + "provision", "probe", + "attach:weights", "mount:weights:/models", + "attach:imagenet", "mount:imagenet:/volumes/imagenet", + } + if !slices.Equal(eng.volumeOps, wantOps) { + t.Errorf("operations=%v, want %v", eng.volumeOps, wantOps) + } + if !slices.Equal(eng.volumeMounts, wantApplied) { + t.Errorf("mounts=%v, want %v", eng.volumeMounts, wantApplied) + } + if got := eng.volumeSpecs; len(got) != 2 || got[0].Path != second || got[0].DirectIO != "on" || got[1].Path != first { + t.Errorf("attached specs=%+v", got) + } + persisted, err := newClaimStore(m.dataDir).load() + if err != nil { + t.Fatalf("load claims: %v", err) + } + if got := persisted[sb.ID]; got == nil || !slices.Equal(got.Volumes, wantApplied) { + t.Errorf("persisted volumes=%v, want %v", got, wantApplied) + } + if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("Release: %v", err) + } + for path, want := range map[string]string{first: "first", second: "second"} { + got, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read backing image: %v", err) + } + if string(got) != want { + t.Errorf("backing image %s=%q, want %q", path, got, want) + } + } +} + +func TestClaimWarmAppliesVolumesAndRefillsAfterFailure(t *testing.T) { + path := writeVolumeImage(t, "data.img", "data") + for _, tt := range []struct { + name string + mountErr error + wantClaim bool + }{ + {name: "success", wantClaim: true}, + {name: "mount failure", mountErr: errors.New("mount failed")}, + } { + t.Run(tt.name, func(t *testing.T) { + eng := newFakeEngine() + eng.mountVolumeErr = tt.mountErr + m := newVolumePoolManager(t, eng, t.TempDir(), []config.VolumeSpec{{Name: "data", Path: path}}) + warm := &types.Sandbox{VMName: "sbx-warm", Key: testKey, VsockSocket: "/vsock/warm"} + m.pools[testKey].warm = append(m.pools[testKey].warm, warm) + + sb, err := m.ClaimWarm(t.Context(), testKey, 0, "", "", []types.Volume{{Name: "data"}}) + if tt.wantClaim && err != nil { + t.Fatalf("ClaimWarm: %v", err) + } + if tt.wantClaim { + wantVolumes := []types.Volume{{Name: "data", Mount: "/volumes/data"}} + if sb.VMName != warm.VMName || !slices.Equal(sb.Volumes, wantVolumes) { + t.Errorf("sandbox=%+v, want warm VM with %v", sb, wantVolumes) + } + } else { + if err == nil { + t.Fatal("ClaimWarm succeeded") + } + if !eng.removed(warm.VMName) { + t.Errorf("removes=%v, want failed warm VM destroyed", eng.removedNames()) + } + } + if len(eng.colds)+len(eng.clones) != 0 { + t.Errorf("warm claim provisioned colds=%v clones=%v", eng.colds, eng.clones) + } + wantOps := []string{"attach:data", "mount:data:/volumes/data"} + if !slices.Equal(eng.volumeOps, wantOps) { + t.Errorf("operations=%v, want %v", eng.volumeOps, wantOps) + } + select { + case <-m.refillKick: + default: + t.Error("warm pop did not request refill") + } + }) + } +} + +func TestClaimProvisionVolumeFailureDestroysVM(t *testing.T) { + path := writeVolumeImage(t, "data.img", "data") + for _, tt := range []struct { + name string + setup func(*fakeEngine) + wantLast string + cancelOnAttach bool + }{ + {"attach", func(eng *fakeEngine) { eng.diskAttachErr = errors.New("attach failed") }, "attach:data", false}, + {"guest setup", func(eng *fakeEngine) { eng.mountVolumeErr = errors.New("mount failed") }, "mount:data:/volumes/data", false}, + {"canceled caller", func(eng *fakeEngine) { eng.diskAttachErr = errors.New("attach failed") }, "attach:data", true}, + } { + t.Run(tt.name, func(t *testing.T) { + eng := newFakeEngine() + tt.setup(eng) + m := newVolumeManager(t, eng, []config.VolumeSpec{{Name: "data", Path: path, DirectIO: "off"}}) + ctx := t.Context() + if tt.cancelOnAttach { + var cancel context.CancelFunc + ctx, cancel = context.WithCancel(ctx) + eng.diskAttachCancel = cancel + } + + _, err := m.ClaimProvision(ctx, testKey, 0, "", "", []types.Volume{{Name: "data"}}) + if err == nil { + t.Fatal("ClaimProvision succeeded") + } + if len(eng.volumeOps) == 0 || eng.volumeOps[len(eng.volumeOps)-1] != tt.wantLast { + t.Errorf("operations=%v, want last %q", eng.volumeOps, tt.wantLast) + } + if len(eng.removes) != 1 { + t.Errorf("removes=%v, want failed VM destroyed", eng.removes) + } + if _, gauges := m.Info(); gauges.Claimed != 0 { + t.Errorf("claimed=%d, want 0", gauges.Claimed) + } + if persisted, err := newClaimStore(m.dataDir).load(); err != nil || len(persisted) != 0 { + t.Errorf("persisted=%v err=%v, want no claim", persisted, err) + } + }) + } +} + +func TestClaimProvisionRejectsInvalidVolumesBeforeProvision(t *testing.T) { + path := writeVolumeImage(t, "data.img", "data") + tooMany := make([]types.Volume, types.MaxClaimVolumes+1) + for i := range tooMany { + tooMany[i].Name = string(rune('a' + i)) + } + for _, tt := range []struct { + name string + key types.PoolKey + catalog []config.VolumeSpec + volumes []types.Volume + }{ + {"invalid name", testKey, []config.VolumeSpec{{Name: "data", Path: path}}, []types.Volume{{Name: "Data"}}}, + {"reserved name", testKey, []config.VolumeSpec{{Name: "data", Path: path}}, []types.Volume{{Name: "cocoon-data"}}}, + {"duplicate", testKey, []config.VolumeSpec{{Name: "data", Path: path}}, []types.Volume{{Name: "data"}, {Name: "data"}}}, + {"too many", testKey, nil, tooMany}, + {"unknown", testKey, []config.VolumeSpec{{Name: "data", Path: path}}, []types.Volume{{Name: "other"}}}, + {"invalid mount", testKey, []config.VolumeSpec{{Name: "data", Path: path}}, []types.Volume{{Name: "data", Mount: "relative"}}}, + {"firecracker", types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall, Engine: types.EngineFC}, []config.VolumeSpec{{Name: "data", Path: path}}, []types.Volume{{Name: "data"}}}, + } { + t.Run(tt.name, func(t *testing.T) { + eng := newFakeEngine() + m := newVolumeManager(t, eng, tt.catalog) + if _, err := m.ClaimProvision(t.Context(), tt.key, 0, "", "", tt.volumes); !errors.Is(err, ErrBadVolume) { + t.Errorf("got %v, want ErrBadVolume", err) + } + if len(eng.colds)+len(eng.clones) != 0 { + t.Errorf("provisioned colds=%v clones=%v", eng.colds, eng.clones) + } + }) + } +} + +func TestClaimProvisionPromotedRejectsMissingTemplateBeforeProvision(t *testing.T) { + path := writeVolumeImage(t, "data.img", "data") + eng := newFakeEngine() + m := newVolumeManager(t, eng, []config.VolumeSpec{{Name: "data", Path: path}}) + + _, err := m.ClaimProvisionPromoted(t.Context(), testKey, 0, "", "", []types.Volume{{Name: "data"}}) + if !errors.Is(err, ErrVolumeUnavailable) { + t.Errorf("error=%v, want ErrVolumeUnavailable", err) + } + if len(eng.colds)+len(eng.clones) != 0 { + t.Errorf("provisioned colds=%v clones=%v", eng.colds, eng.clones) + } +} + +func TestClaimProvisionPromotedAppliesVolumesFromTemplate(t *testing.T) { + path := writeVolumeImage(t, "data.img", "data") + eng := newFakeEngine() + m := newVolumeManager(t, eng, []config.VolumeSpec{{Name: "data", Path: path}}) + parent := mustClaim(t, m, testKey) + key, digest, err := m.Promote(t.Context(), parent.ID, Cred{Token: parent.Token}, "tpl:volume", "") + if err != nil { + t.Fatalf("Promote: %v", err) + } + beforeColds, beforeClones := len(eng.colds), len(eng.clones) + eng.volumeOps = nil + + sb, err := m.ClaimProvisionPromoted(t.Context(), key, 0, "", "", []types.Volume{{Name: "data"}}) + if err != nil { + t.Fatalf("ClaimProvisionPromoted: %v", err) + } + if len(eng.colds) != beforeColds || len(eng.clones) != beforeClones+1 { + t.Errorf("provisioned colds=%v clones=%v, want one template clone and no cold boot", eng.colds, eng.clones) + } + wantOps := []string{"provision", "probe", "attach:data", "mount:data:/volumes/data"} + if !slices.Equal(eng.volumeOps, wantOps) { + t.Errorf("operations=%v, want %v", eng.volumeOps, wantOps) + } + if sb.TemplateDigest != digest { + t.Errorf("template digest=%q, want %q", sb.TemplateDigest, digest) + } +} + +func TestClaimProvisionVolumeACL(t *testing.T) { + path := writeVolumeImage(t, "data.img", "data") + eng := newFakeEngine() + m := newVolumeManager(t, eng, []config.VolumeSpec{ + {Name: "private", Path: path, Tenants: []string{"acme"}}, + {Name: "public", Path: path}, + }) + + _, forbiddenErr := m.ClaimProvision(t.Context(), testKey, 0, "beta", "", []types.Volume{{Name: "private"}}) + _, unknownErr := m.ClaimProvision(t.Context(), testKey, 0, "beta", "", []types.Volume{{Name: "unknown"}}) + if forbiddenErr == nil || unknownErr == nil || forbiddenErr.Error() != unknownErr.Error() { + t.Fatalf("forbidden=%q unknown=%q, want byte-identical errors", forbiddenErr, unknownErr) + } + if !errors.Is(forbiddenErr, ErrBadVolume) { + t.Errorf("forbidden error %v, want ErrBadVolume", forbiddenErr) + } + if len(eng.colds)+len(eng.clones) != 0 { + t.Errorf("rejected claims provisioned colds=%v clones=%v", eng.colds, eng.clones) + } + for _, tt := range []struct { + name, tenant, volume string + }{ + {"listed tenant", "acme", "private"}, + {"root", "", "private"}, + {"public", "beta", "public"}, + } { + t.Run(tt.name, func(t *testing.T) { + if _, err := m.ClaimProvision(t.Context(), testKey, 0, tt.tenant, "", []types.Volume{{Name: tt.volume}}); err != nil { + t.Errorf("ClaimProvision: %v", err) + } + }) + } +} + +func TestVolumesFiltersScopeAndStatsPaths(t *testing.T) { + publicPath := writeVolumeImage(t, "public.img", "public") + privatePath := writeVolumeImage(t, "private.img", "private") + missingPath := filepath.Join(t.TempDir(), "missing.img") + m := newVolumeManager(t, newFakeEngine(), []config.VolumeSpec{ + {Name: "public", Path: publicPath}, + {Name: "acme", Path: privatePath, Tenants: []string{"acme"}}, + {Name: "beta", Path: missingPath, Tenants: []string{"beta"}}, + }) + holders := map[string]int{"acme": 2, "beta": 3, "peer-only": 2, "public": 4} + + wantRoot := []types.VolumeInfo{ + {Name: "acme", DefaultMount: "/volumes/acme", SizeBytes: int64(len("private")), Available: true, Nodes: 2}, + {Name: "beta", DefaultMount: "/volumes/beta", Nodes: 3}, + {Name: "peer-only", DefaultMount: "/volumes/peer-only", Nodes: 2}, + {Name: "public", DefaultMount: "/volumes/public", SizeBytes: int64(len("public")), Available: true, Nodes: 4}, + } + if got := m.Volumes("", holders); !slices.Equal(got, wantRoot) { + t.Errorf("root volumes=%+v, want %+v", got, wantRoot) + } + wantAcme := []types.VolumeInfo{wantRoot[0], wantRoot[3]} + if got := m.Volumes("acme", holders); !slices.Equal(got, wantAcme) { + t.Errorf("acme volumes=%+v, want %+v", got, wantAcme) + } + wantBeta := []types.VolumeInfo{wantRoot[1], wantRoot[3]} + if got := m.Volumes("beta", holders); !slices.Equal(got, wantBeta) { + t.Errorf("beta volumes=%+v, want %+v", got, wantBeta) + } + if got := m.VolumeNames(); !slices.Equal(got, []string{"acme", "public"}) { + t.Errorf("available names=%v, want [acme public]", got) + } + if err := os.WriteFile(missingPath, []byte("beta"), 0o600); err != nil { + t.Fatalf("distribute missing image: %v", err) + } + if got := m.VolumeNames(); !slices.Equal(got, []string{"acme", "beta", "public"}) { + t.Errorf("names after image distribution=%v, want [acme beta public]", got) + } + if err := os.Remove(missingPath); err != nil { + t.Fatalf("remove distributed image: %v", err) + } + if got := m.VolumeNames(); !slices.Equal(got, []string{"acme", "public"}) { + t.Errorf("names after image removal=%v, want [acme public]", got) + } +} + +func TestVolumePlacementChecksAccessAndLocalAvailability(t *testing.T) { + path := writeVolumeImage(t, "data.img", "data") + m := newVolumeManager(t, newFakeEngine(), []config.VolumeSpec{ + {Name: "local", Path: path, Tenants: []string{"acme"}}, + {Name: "remote", Path: filepath.Join(t.TempDir(), "missing.img"), Tenants: []string{"acme"}}, + }) + + if local, err := m.VolumePlacement(testKey, "acme", []string{"local"}); err != nil || !local { + t.Errorf("local placement=(%v, %v), want true, nil", local, err) + } + if local, err := m.VolumePlacement(testKey, "acme", []string{"local", "remote"}); err != nil || local { + t.Errorf("partial local placement=(%v, %v), want false, nil", local, err) + } + _, forbiddenErr := m.VolumePlacement(testKey, "beta", []string{"local"}) + _, unknownErr := m.VolumePlacement(testKey, "beta", []string{"unknown"}) + if forbiddenErr == nil || unknownErr == nil || forbiddenErr.Error() != unknownErr.Error() { + t.Errorf("forbidden=%q unknown=%q, want byte-identical errors", forbiddenErr, unknownErr) + } + if local, err := m.VolumePlacement(testKey, "", []string{"peer-only"}); err != nil || local { + t.Errorf("root peer-only placement=(%v, %v), want false, nil", local, err) + } + badKey := types.PoolKey{Template: "rt:24.04", Net: "lan", Size: types.SizeSmall, Engine: types.EngineCH} + if _, err := m.VolumePlacement(badKey, "", []string{"local"}); !errors.Is(err, ErrBadKey) { + t.Errorf("invalid key error=%v, want ErrBadKey", err) + } +} + +func TestVolumeClaimUsageAndScopedSummaries(t *testing.T) { + path := writeVolumeImage(t, "data.img", "data") + m := newVolumeManager(t, newFakeEngine(), []config.VolumeSpec{{Name: "data", Path: path}}) + request := []types.Volume{{Name: "data", Mount: "/datasets"}} + wantApplied := []types.Volume{{Name: "data", Mount: "/datasets"}} + acme, err := m.ClaimProvision(t.Context(), testKey, 0, "acme", "", request) + if err != nil { + t.Fatalf("acme claim: %v", err) + } + _, betaErr := m.ClaimProvision(t.Context(), testKey, 0, "beta", "", request) + if betaErr != nil { + t.Fatalf("beta claim: %v", betaErr) + } + + if got := m.Sandboxes("acme"); len(got) != 1 || got[0].ID != acme.ID || !slices.Equal(got[0].Volumes, wantApplied) { + t.Errorf("acme summaries=%+v, want only its volume claim", got) + } + if got := m.Sandboxes(""); len(got) != 2 { + t.Errorf("root summaries=%+v, want both claims", got) + } + if got, ok := m.Sandbox(acme.ID); !ok || !slices.Equal(got.Volumes, wantApplied) { + t.Errorf("sandbox summary=%+v ok=%v, want applied volume", got, ok) + } + + raw, err := os.ReadFile(filepath.Join(m.dataDir, "usage.jsonl")) + if err != nil { + t.Fatalf("read usage journal: %v", err) + } + for line := range strings.SplitSeq(strings.TrimSpace(string(raw)), "\n") { + var event usageEvent + if err := json.Unmarshal([]byte(line), &event); err != nil { + t.Fatalf("decode usage event: %v", err) + } + if event.Event == "claim" && event.ID == acme.ID { + if !slices.Equal(event.Volumes, []string{"data"}) { + t.Errorf("claim event volumes=%v, want [data]", event.Volumes) + } + return + } + } + t.Fatal("acme claim usage event not found") +} + +func TestClaimProvisionRejectsMissingVolumePathBeforeProvision(t *testing.T) { + eng := newFakeEngine() + m := newVolumeManager(t, eng, []config.VolumeSpec{{Name: "data", Path: filepath.Join(t.TempDir(), "missing.img")}}) + + _, err := m.ClaimProvision(t.Context(), testKey, 0, "", "", []types.Volume{{Name: "data"}}) + if err == nil || !strings.Contains(err.Error(), "missing.img") { + t.Errorf("got %v, want missing-path error", err) + } + if len(eng.colds)+len(eng.clones) != 0 { + t.Errorf("provisioned colds=%v clones=%v", eng.colds, eng.clones) + } +} + +func newVolumeManager(t *testing.T, eng *fakeEngine, volumes []config.VolumeSpec) *Manager { + t.Helper() + m, err := NewManager(t.Context(), &config.Config{DataDir: t.TempDir(), Volumes: volumes}, eng, testSecrets(t)) + if err != nil { + t.Fatalf("setup manager: %v", err) + } + return m +} + +func newVolumePoolManager(t *testing.T, eng *fakeEngine, dataDir string, volumes []config.VolumeSpec) *Manager { + t.Helper() + m, err := NewManager(t.Context(), &config.Config{ + DataDir: dataDir, + Pools: []config.PoolSpec{{PoolKey: testKey, Warm: 1}}, + Volumes: volumes, + }, eng, testSecrets(t)) + if err != nil { + t.Fatalf("setup manager: %v", err) + } + return m +} + +func writeVolumeImage(t *testing.T, name, content string) string { + t.Helper() + path := filepath.Join(t.TempDir(), name) + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatalf("write volume image: %v", err) + } + return path +} diff --git a/sandboxd/server/metrics.go b/sandboxd/server/metrics.go index 638321b3..84597d92 100644 --- a/sandboxd/server/metrics.go +++ b/sandboxd/server/metrics.go @@ -87,7 +87,7 @@ func (s *Server) handleMetrics(w http.ResponseWriter, _ *http.Request) { } } -// handleSandboxes lists live claims for operator tooling — never tokens. -func (s *Server) handleSandboxes(w http.ResponseWriter, _ *http.Request) { - writeJSON(w, http.StatusOK, SandboxListResponse{Sandboxes: s.mgr.Sandboxes()}) +// handleSandboxes lists the live claims visible to the caller — never tokens. +func (s *Server) handleSandboxes(w http.ResponseWriter, r *http.Request) { + writeJSON(w, http.StatusOK, SandboxListResponse{Sandboxes: s.mgr.Sandboxes(tenantFrom(r.Context()))}) } diff --git a/sandboxd/server/relay.go b/sandboxd/server/relay.go index 73c10367..1502f15d 100644 --- a/sandboxd/server/relay.go +++ b/sandboxd/server/relay.go @@ -13,6 +13,7 @@ import ( "github.com/projecteru2/core/log" "github.com/cocoonstack/sandbox/sandboxd/pool" + "github.com/cocoonstack/sandbox/sandboxd/utils" ) // drainGrace bounds how long a finished relay waits for the client to @@ -130,7 +131,7 @@ func (s *Server) relay(ctx context.Context, id string, client net.Conn, clientBu }() _, _ = io.Copy(client, guest) - closeWrite(client) + utils.CloseWrite(client) // Wait for the client to consume the tail and close its side; the // deadline unblocks the splice goroutine if it never does. _ = client.SetReadDeadline(time.Now().Add(drainGrace)) @@ -189,11 +190,3 @@ func (t *auditTee) Read(p []byte) (int, error) { } return n, err } - -// closeWrite signals EOF to the client without tearing down its read -// direction, so the tail written above survives until the client drains it. -func closeWrite(conn net.Conn) { - if cw, ok := conn.(interface{ CloseWrite() error }); ok { - _ = cw.CloseWrite() - } -} diff --git a/sandboxd/server/server.go b/sandboxd/server/server.go index fd7c4f36..673ae5e4 100644 --- a/sandboxd/server/server.go +++ b/sandboxd/server/server.go @@ -11,6 +11,7 @@ import ( "context" "crypto/subtle" "errors" + "fmt" "io" "net" "net/http" @@ -40,11 +41,13 @@ var poolErrHTTP = []struct { msg string }{ {pool.ErrBadKey, http.StatusBadRequest, ""}, + {pool.ErrBadVolume, http.StatusBadRequest, ""}, {pool.ErrBadName, http.StatusBadRequest, ""}, {pool.ErrBadCount, http.StatusBadRequest, ""}, {pool.ErrNoEgress, http.StatusConflict, ""}, {pool.ErrNoEgressHibernate, http.StatusConflict, ""}, {pool.ErrNoEgressFork, http.StatusConflict, ""}, + {pool.ErrVolumeCapture, http.StatusConflict, ""}, {pool.ErrQuota, http.StatusTooManyRequests, ""}, {pool.ErrHealBusy, http.StatusServiceUnavailable, ""}, {pool.ErrPooledTemplate, http.StatusConflict, ""}, @@ -58,8 +61,9 @@ var poolErrHTTP = []struct { // parameters attribute created resources and scope listings/deletes; empty // means the operator (root) — unquotaed, unfiltered. type Manager interface { - ClaimWarm(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string) (*types.Sandbox, error) - ClaimProvision(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string) (*types.Sandbox, error) + ClaimWarm(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) + ClaimProvision(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) + ClaimProvisionPromoted(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) Release(ctx context.Context, id string, cred pool.Cred) error Hibernate(ctx context.Context, id string, cred pool.Cred) error Wake(ctx context.Context, id string, cred pool.Cred) error @@ -69,7 +73,9 @@ type Manager interface { Checkpoint(ctx context.Context, id string, cred pool.Cred, name, tenant string) (types.Checkpoint, error) Counters() pool.Counters TenantClaims() map[string]int - Sandboxes() []pool.SandboxSummary + VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, error) + Volumes(tenant string, holders map[string]int) []types.VolumeInfo + Sandboxes(tenant string) []pool.SandboxSummary Sandbox(id string) (pool.SandboxSummary, bool) Stats(ctx context.Context, id string) (pool.SandboxStats, bool) Audit(ctx context.Context, id string, line []byte) @@ -82,6 +88,7 @@ type Manager interface { DeleteCheckpoint(ctx context.Context, ckptID, tenant string, scope pool.DeleteScope) error ClaimDeadline(id, token string) (time.Time, error) HasGolden(ctx context.Context, key types.PoolKey) bool + HasPromotedTemplate(ctx context.Context, key types.PoolKey) bool AgentSocket(id, token string) (string, error) WakeAgentSocket(ctx context.Context, id, token string) (string, error) SetPools(ctx context.Context, pools []config.PoolSpec) error @@ -99,7 +106,11 @@ type Dialer interface { // nil on a single-node deployment (no mesh). type Placer interface { Candidates(keyHash string) []string + VolumeCandidates(keyHash string, names []string) []string TemplateOwners(keyHash string) []string + VolumeOwners(names []string) []string + TemplateVolumeOwners(keyHash string, names []string) []string + VolumeHolders() map[string]int PeerAddrs() []string ConfigMismatches() int } @@ -177,11 +188,12 @@ func New(apiToken string, tenants []config.TenantSpec, advertise string, mgr Man } // Handler builds the route table. Resource-creating verbs and tenant-scoped -// listings/deletes take the api token or a tenant token; operator surfaces -// (index, pools, info, metrics) stay root-only. +// listings/deletes take the api token or a tenant token; pools, info, metrics, +// and per-sandbox operator reads stay root-only. func (s *Server) Handler() http.Handler { mux := http.NewServeMux() mux.HandleFunc("POST /v1/claim", s.requireToken(s.handleClaim)) + mux.HandleFunc("GET /v1/volumes", s.requireToken(s.handleVolumes)) mux.HandleFunc("POST /v1/sandboxes/{id}/release", s.handleRelease) mux.HandleFunc("POST /v1/sandboxes/{id}/hibernate", s.handleSandboxVerb("hibernate", s.mgr.Hibernate)) mux.HandleFunc("POST /v1/sandboxes/{id}/wake", s.handleSandboxVerb("wake", s.mgr.Wake)) @@ -209,7 +221,7 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("GET /v1/sandboxes/{id}/owner", s.handleOwner) mux.HandleFunc("GET /v1/info", s.requireRoot(s.handleInfo)) mux.HandleFunc("GET /v1/peers", s.requireToken(s.handlePeers)) - mux.HandleFunc("GET /v1/sandboxes", s.requireRoot(s.handleSandboxes)) + mux.HandleFunc("GET /v1/sandboxes", s.requireToken(s.handleSandboxes)) mux.HandleFunc("GET /metrics", s.requireRoot(s.handleMetrics)) mux.HandleFunc("GET /healthz", s.handleHealthz) return mux @@ -222,18 +234,22 @@ func (s *Server) handleClaim(w http.ResponseWriter, r *http.Request) { } key := req.Key() hash := key.Hash() + tenant := tenantFrom(r.Context()) + if len(req.Volumes) > 0 { + s.handleVolumeClaim(w, r, req, key, hash, tenant) + return + } // Warm hit here is ownership transfer only. On a warm miss with a mesh, a // peer that reports a warm sandbox gets the claim via redirect (data plane // must be direct, so redirect beats proxy); only if no peer has one does // this node provision (golden clone or cold boot). - tenant := tenantFrom(r.Context()) - sb, err := s.mgr.ClaimWarm(r.Context(), key, req.TTL(), tenant, req.ClaimRef) + sb, err := s.mgr.ClaimWarm(r.Context(), key, req.TTL(), tenant, req.ClaimRef, nil) if errors.Is(err, pool.ErrNoWarm) { if s.redirectClaim(r.Context(), w, req, key, hash) { return } - sb, err = s.mgr.ClaimProvision(r.Context(), key, req.TTL(), tenant, req.ClaimRef) + sb, err = s.mgr.ClaimProvision(r.Context(), key, req.TTL(), tenant, req.ClaimRef, nil) } // A full node bounces the claim to a warm peer before answering 429 — // quota is per node, and a peer with capacity is a better answer. @@ -246,6 +262,91 @@ func (s *Server) handleClaim(w http.ResponseWriter, r *http.Request) { }) } +func (s *Server) handleVolumeClaim(w http.ResponseWriter, r *http.Request, req types.ClaimRequest, key types.PoolKey, hash, tenant string) { + volumes, err := types.ValidateVolumes(req.Volumes) + if err == nil && key.Engine != types.EngineCH { + err = errors.New("volumes require engine ch") + } + if err != nil { + writeErr(w, http.StatusBadRequest, fmt.Errorf("%w: %v", pool.ErrBadVolume, err).Error()) + return + } + req.Volumes = volumes + redirected, err := s.redirectVolumeClaim(r.Context(), w, &req, key, hash, tenant) + if err != nil { + writeResult(w, r, "claim", hash, "provisioning failed", err, func() {}) + return + } + if redirected { + return + } + var sb *types.Sandbox + if req.RequirePromoted { + sb, err = s.mgr.ClaimProvisionPromoted(r.Context(), key, req.TTL(), tenant, req.ClaimRef, req.Volumes) + } else { + sb, err = s.mgr.ClaimWarm(r.Context(), key, req.TTL(), tenant, req.ClaimRef, req.Volumes) + if errors.Is(err, pool.ErrNoWarm) { + if s.placer != nil && !req.NoRedirect && writeRedirect(w, s.placer.VolumeCandidates(hash, types.VolumeNames(req.Volumes))) { + return + } + sb, err = s.mgr.ClaimProvision(r.Context(), key, req.TTL(), tenant, req.ClaimRef, req.Volumes) + } + } + writeResult(w, r, "claim", hash, "provisioning failed", err, func() { + writeJSON(w, http.StatusOK, s.claimResponse(sb)) + }) +} + +func (s *Server) handleVolumes(w http.ResponseWriter, r *http.Request) { + var holders map[string]int + if s.placer != nil { + holders = s.placer.VolumeHolders() + } + writeJSON(w, http.StatusOK, types.VolumeListResponse{ + Volumes: s.mgr.Volumes(tenantFrom(r.Context()), holders), + }) +} + +func (s *Server) redirectVolumeClaim(ctx context.Context, w http.ResponseWriter, req *types.ClaimRequest, key types.PoolKey, hash, tenant string) (bool, error) { + names := types.VolumeNames(req.Volumes) + localVolumes, err := s.mgr.VolumePlacement(key, tenant, names) + if err != nil { + return false, err + } + + localTemplate := s.mgr.HasPromotedTemplate(ctx, key) + var templateOwners []string + if s.placer != nil { + templateOwners = s.placer.TemplateOwners(hash) + } + promoted := req.RequirePromoted || localTemplate || len(templateOwners) > 0 + req.RequirePromoted = promoted + if localVolumes && (!promoted || localTemplate) { + return false, nil + } + if s.placer == nil || req.NoRedirect { + return false, pool.ErrVolumeUnavailable + } + + var owners []string + if promoted { + // A shared template store lets a volume holder resolve the template + // even before that node has advertised the newly published hash. The + // no_redirect target re-checks both resources before provisioning. + owners = s.placer.TemplateVolumeOwners(hash, names) + } else { + owners = s.placer.VolumeCandidates(hash, names) + } + if len(owners) == 0 { + owners = s.placer.VolumeOwners(names) + } + if len(owners) == 0 { + return false, pool.ErrVolumeUnavailable + } + writeJSON(w, http.StatusOK, types.ClaimResponse{Redirect: owners, RequirePromoted: promoted}) + return true, nil +} + // redirectClaim redirects a warm-miss to a better peer — a warm holder, or the // template owner when we lack a golden (so we don't cold-boot a nonexistent // image ref). A no_redirect request must resolve locally, never bounce again, @@ -641,5 +742,6 @@ func (s *Server) claimResponse(sb *types.Sandbox) types.ClaimResponse { return types.ClaimResponse{ ID: sb.ID, Token: sb.Token, Deadline: sb.Deadline, OwnerAddr: s.advertise, FromCheckpoint: sb.FromCheckpoint, TemplateDigest: sb.TemplateDigest, + Volumes: sb.Volumes, } } diff --git a/sandboxd/server/server_test.go b/sandboxd/server/server_test.go index 17066eb2..920fc287 100644 --- a/sandboxd/server/server_test.go +++ b/sandboxd/server/server_test.go @@ -1,6 +1,7 @@ package server import ( + "bytes" "context" "encoding/json" "errors" @@ -11,6 +12,7 @@ import ( "net/http/httptest" "os" "path/filepath" + "slices" "strings" "testing" "time" @@ -72,6 +74,7 @@ func TestClaimErrorMapping(t *testing.T) { }{ {"bad json", `{oops`, nil, http.StatusBadRequest}, {"bad key", `{"template":"rt:24.04","net":"lan"}`, fmt.Errorf("%w: unknown net", pool.ErrBadKey), http.StatusBadRequest}, + {"bad volume", `{"template":"rt:24.04","volumes":[{"name":"data"}]}`, fmt.Errorf("%w: unknown volume", pool.ErrBadVolume), http.StatusBadRequest}, {"no egress", `{"template":"rt:24.04","net":"egress"}`, pool.ErrNoEgress, http.StatusConflict}, {"engine failure", `{"template":"rt:24.04"}`, errors.New("cocoon vm run: boom"), http.StatusInternalServerError}, } @@ -95,6 +98,24 @@ func TestClaimErrorMapping(t *testing.T) { } } +func TestHibernateVolumeCaptureMapsConflict(t *testing.T) { + mgr := &fakeManager{hibernate: func(string, string) error { return pool.ErrVolumeCapture }} + ts := newTestServer(t, "", mgr, nil) + req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/hibernate", nil) + if err != nil { + t.Fatalf("request: %v", err) + } + req.Header.Set("Authorization", "Bearer tok") + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("hibernate: %v", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusConflict { + t.Errorf("status %d, want 409", resp.StatusCode) + } +} + func TestAPITokenGuard(t *testing.T) { ts := newTestServer(t, "sekret", &fakeManager{}, nil) @@ -108,6 +129,10 @@ func TestAPITokenGuard(t *testing.T) { {"claim no token", "/v1/claim", http.MethodPost, "", http.StatusUnauthorized}, {"claim wrong token", "/v1/claim", http.MethodPost, "Bearer nope", http.StatusUnauthorized}, {"claim right token", "/v1/claim", http.MethodPost, "Bearer sekret", http.StatusOK}, + {"volumes no token", "/v1/volumes", http.MethodGet, "", http.StatusUnauthorized}, + {"volumes right token", "/v1/volumes", http.MethodGet, "Bearer sekret", http.StatusOK}, + {"sandboxes no token", "/v1/sandboxes", http.MethodGet, "", http.StatusUnauthorized}, + {"sandboxes right token", "/v1/sandboxes", http.MethodGet, "Bearer sekret", http.StatusOK}, {"info no token", "/v1/info", http.MethodGet, "", http.StatusUnauthorized}, {"info right token", "/v1/info", http.MethodGet, "Bearer sekret", http.StatusOK}, {"put pools no token", "/v1/pools", http.MethodPut, "", http.StatusUnauthorized}, @@ -142,10 +167,74 @@ func TestAPITokenGuard(t *testing.T) { } } -// TestTenantAuthMatrix drives the three token kinds across the two endpoint -// classes: resource-creating verbs take root or tenant tokens (the tenant -// scope reaches the manager), operator surfaces answer 403 to a tenant — -// authenticated but not authorized — and 401 to anything unknown. +func TestVolumeCatalogReturnsScopedFleetProjection(t *testing.T) { + mgr := &fakeManager{volumeCatalog: func(tenant string, holders map[string]int) []types.VolumeInfo { + if tenant != "acme" { + t.Errorf("tenant = %q, want acme", tenant) + } + if holders["imagenet"] != 3 { + t.Errorf("imagenet holders = %d, want 3", holders["imagenet"]) + } + return []types.VolumeInfo{{ + Name: "imagenet", DefaultMount: "/volumes/imagenet", SizeBytes: 42, Available: true, Nodes: 3, + }} + }} + placer := &fakePlacer{volumeHolders: map[string]int{"imagenet": 3}} + srv := New("root", []config.TenantSpec{{Name: "acme", Token: "acme-tok"}}, "node:7777", mgr, &fakeDialer{}, placer, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + t.Cleanup(ts.Close) + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, ts.URL+"/v1/volumes", nil) + if err != nil { + t.Fatalf("request: %v", err) + } + req.Header.Set("Authorization", "Bearer acme-tok") + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("volumes: %v", err) + } + defer resp.Body.Close() + var got types.VolumeListResponse + if err := json.NewDecoder(resp.Body).Decode(&got); err != nil { + t.Fatalf("decode: %v", err) + } + want := []types.VolumeInfo{{ + Name: "imagenet", DefaultMount: "/volumes/imagenet", SizeBytes: 42, Available: true, Nodes: 3, + }} + if !slices.Equal(got.Volumes, want) { + t.Errorf("volumes = %+v, want %+v", got.Volumes, want) + } +} + +func TestSandboxIndexReturnsScopedAppliedVolumes(t *testing.T) { + mgr := &fakeManager{sandboxIndex: func(tenant string) []pool.SandboxSummary { + if tenant != "acme" { + t.Errorf("tenant = %q, want acme", tenant) + } + return []pool.SandboxSummary{{ + ID: "sb_1", Volumes: []types.Volume{{Name: "imagenet", Mount: "/datasets/imagenet"}}, + }} + }} + ts := newTenantTestServer(t, "root", []config.TenantSpec{{Name: "acme", Token: "acme-tok"}}, mgr, nil) + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, ts.URL+"/v1/sandboxes", nil) + if err != nil { + t.Fatalf("request: %v", err) + } + req.Header.Set("Authorization", "Bearer acme-tok") + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("sandboxes: %v", err) + } + defer resp.Body.Close() + var got SandboxListResponse + if err := json.NewDecoder(resp.Body).Decode(&got); err != nil { + t.Fatalf("decode: %v", err) + } + want := []types.Volume{{Name: "imagenet", Mount: "/datasets/imagenet"}} + if len(got.Sandboxes) != 1 || got.Sandboxes[0].ID != "sb_1" || !slices.Equal(got.Sandboxes[0].Volumes, want) { + t.Errorf("sandboxes = %+v, want sb_1 with %+v", got.Sandboxes, want) + } +} + func TestTenantAuthMatrix(t *testing.T) { mgr := &fakeManager{tenantClaims: map[string]int{"acme": 2}} tenants := []config.TenantSpec{{Name: "acme", Token: "acme-tok"}, {Name: "beta", Token: "beta-tok"}} @@ -186,8 +275,11 @@ func TestTenantAuthMatrix(t *testing.T) { {"missing token", http.MethodPost, "/v1/claim", "", http.StatusUnauthorized, ""}, {"tenant lists own checkpoints", http.MethodGet, "/v1/checkpoints", "acme-tok", http.StatusOK, "acme"}, {"root lists all checkpoints", http.MethodGet, "/v1/checkpoints", "sekret", http.StatusOK, ""}, + {"tenant lists visible volumes", http.MethodGet, "/v1/volumes", "acme-tok", http.StatusOK, "acme"}, + {"root lists all volumes", http.MethodGet, "/v1/volumes", "sekret", http.StatusOK, ""}, + {"tenant lists own sandboxes", http.MethodGet, "/v1/sandboxes", "acme-tok", http.StatusOK, "acme"}, + {"root lists all sandboxes", http.MethodGet, "/v1/sandboxes", "sekret", http.StatusOK, ""}, {"tenant forbidden on info", http.MethodGet, "/v1/info", "acme-tok", http.StatusForbidden, ""}, - {"tenant forbidden on index", http.MethodGet, "/v1/sandboxes", "acme-tok", http.StatusForbidden, ""}, {"tenant forbidden on metrics", http.MethodGet, "/metrics", "acme-tok", http.StatusForbidden, ""}, {"tenant forbidden on pools", http.MethodPut, "/v1/pools", "acme-tok", http.StatusForbidden, ""}, {"tenant forbidden on checkpoint blob", http.MethodGet, "/v1/checkpoints/ck_00000000000000aa/blob", "acme-tok", http.StatusForbidden, ""}, @@ -201,8 +293,8 @@ func TestTenantAuthMatrix(t *testing.T) { if resp.StatusCode != tt.want { t.Errorf("status %d, want %d", resp.StatusCode, tt.want) } - reachedManager := resp.StatusCode == http.StatusOK && - (tt.path == "/v1/claim" || tt.path == "/v1/checkpoints") + reachedManager := resp.StatusCode == http.StatusOK && slices.Contains( + []string{"/v1/claim", "/v1/checkpoints", "/v1/volumes", "/v1/sandboxes"}, tt.path) if reachedManager && mgr.gotTenant != tt.wantTenant { t.Errorf("manager saw tenant %q, want %q", mgr.gotTenant, tt.wantTenant) } @@ -681,7 +773,10 @@ func TestCheckpointFlow(t *testing.T) { ts := newTestServer(t, "api", mgr, &fakeDialer{}) post := func(path, body string) *http.Response { - req, _ := http.NewRequest(http.MethodPost, ts.URL+path, strings.NewReader(body)) + req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, ts.URL+path, strings.NewReader(body)) + if err != nil { + t.Fatalf("request: %v", err) + } req.Header.Set("Authorization", "Bearer api") resp, err := http.DefaultClient.Do(req) if err != nil { @@ -713,7 +808,10 @@ func TestCheckpointFlow(t *testing.T) { t.Errorf("unknown checkpoint claim: status %d, want 404", resp3.StatusCode) } - req, _ := http.NewRequest(http.MethodDelete, ts.URL+"/v1/checkpoints/"+cr.Checkpoint.ID, nil) + req, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+"/v1/checkpoints/"+cr.Checkpoint.ID, nil) + if err != nil { + t.Fatalf("request: %v", err) + } req.Header.Set("Authorization", "Bearer api") resp4, err := http.DefaultClient.Do(req) if err != nil { @@ -734,7 +832,10 @@ func TestDeleteCheckpointNoForwardQueryParam(t *testing.T) { ts := newTestServer(t, "api", mgr, &fakeDialer{}) del := func(path string) int { - req, _ := http.NewRequest(http.MethodDelete, ts.URL+path, nil) + req, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+path, nil) + if err != nil { + t.Fatalf("request: %v", err) + } req.Header.Set("Authorization", "Bearer api") resp, err := http.DefaultClient.Do(req) if err != nil { @@ -769,7 +870,10 @@ func TestDeleteCheckpointForgetsProbeCache(t *testing.T) { prober.forgotten = nil mgr := &fakeManager{deleteCheckpoint: func(string) error { return nil }} ts := newPlacerTestServer(t, "api", mgr, prober) - req, _ := http.NewRequest(http.MethodDelete, ts.URL+"/v1/checkpoints/ck_0011223344556677", nil) + req, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+"/v1/checkpoints/ck_0011223344556677", nil) + if err != nil { + t.Fatalf("request: %v", err) + } req.Header.Set("Authorization", "Bearer api") resp, err := http.DefaultClient.Do(req) if err != nil { @@ -785,7 +889,10 @@ func TestDeleteCheckpointForgetsProbeCache(t *testing.T) { prober.forgotten = nil mgr := &fakeManager{deleteCheckpoint: func(string) error { return pool.ErrUnknownCheckpoint }} ts := newPlacerTestServer(t, "api", mgr, prober) - req, _ := http.NewRequest(http.MethodDelete, ts.URL+"/v1/checkpoints/ck_0011223344556677?no_forward=1", nil) + req, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+"/v1/checkpoints/ck_0011223344556677?no_forward=1", nil) + if err != nil { + t.Fatalf("request: %v", err) + } req.Header.Set("Authorization", "Bearer api") resp, err := http.DefaultClient.Do(req) if err != nil { @@ -883,7 +990,10 @@ func TestDeleteTemplateRedirectsToOwner(t *testing.T) { ts := httptest.NewServer(srv.Handler()) t.Cleanup(func() { ts.Close(); srv.CloseRelays() }) - req, _ := http.NewRequest(http.MethodDelete, ts.URL+"/v1/templates?template=tpl", nil) + req, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+"/v1/templates?template=tpl", nil) + if err != nil { + t.Fatalf("request: %v", err) + } resp, err := http.DefaultClient.Do(req) if err != nil { t.Fatalf("delete: %v", err) @@ -901,7 +1011,10 @@ func TestDeleteTemplateRedirectsToOwner(t *testing.T) { } // no_redirect answers for this node alone, even with owners in gossip. - req2, _ := http.NewRequest(http.MethodDelete, ts.URL+"/v1/templates?template=tpl&no_redirect=1", nil) + req2, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+"/v1/templates?template=tpl&no_redirect=1", nil) + if err != nil { + t.Fatalf("request: %v", err) + } resp2, err := http.DefaultClient.Do(req2) if err != nil { t.Fatalf("delete: %v", err) @@ -914,7 +1027,10 @@ func TestDeleteTemplateRedirectsToOwner(t *testing.T) { srvNoOwner := New("", nil, "node-a:7777", mgr, &fakeDialer{}, &fakePlacer{}, nil, nil, nil) ts2 := httptest.NewServer(srvNoOwner.Handler()) t.Cleanup(func() { ts2.Close(); srvNoOwner.CloseRelays() }) - req3, _ := http.NewRequest(http.MethodDelete, ts2.URL+"/v1/templates?template=tpl", nil) + req3, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts2.URL+"/v1/templates?template=tpl", nil) + if err != nil { + t.Fatalf("request: %v", err) + } resp3, err := http.DefaultClient.Do(req3) if err != nil { t.Fatalf("delete: %v", err) @@ -950,6 +1066,278 @@ func TestClaimProvisionsWhenNoCandidate(t *testing.T) { } } +func TestVolumeClaimUsesWarmBeforeRedirectOrProvision(t *testing.T) { + wantVolumes := []types.Volume{{Name: "imagenet", Mount: "/volumes/imagenet"}} + for _, tt := range []struct { + name string + warmHit bool + volumeCandidates []string + wantID string + wantRedirect []string + wantProvisionCalls int + wantCandidateCalls int + }{ + {name: "local warm hit", warmHit: true, wantID: "sb_warm"}, + { + name: "warm miss redirects to volume-aware warm candidate", + volumeCandidates: []string{"warm-volume:7777"}, wantRedirect: []string{"warm-volume:7777"}, + wantCandidateCalls: 1, + }, + { + name: "warm miss without candidate provisions locally", wantID: "sb_provisioned", + wantProvisionCalls: 1, wantCandidateCalls: 1, + }, + } { + t.Run(tt.name, func(t *testing.T) { + mgr := &fakeManager{claim: func(context.Context, types.PoolKey, time.Duration) (*types.Sandbox, error) { + return &types.Sandbox{ID: "sb_provisioned", Token: "tok", Volumes: slices.Clone(wantVolumes)}, nil + }} + if tt.warmHit { + mgr.warmClaim = func(context.Context, types.PoolKey, time.Duration) (*types.Sandbox, error) { + return &types.Sandbox{ID: "sb_warm", Token: "tok", Volumes: slices.Clone(wantVolumes)}, nil + } + } + placer := &fakePlacer{ + addrs: []string{"wrong-general-candidate:7777"}, volumeCandidates: tt.volumeCandidates, + } + srv := New("", nil, "node-a:7777", mgr, &fakeDialer{}, placer, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + t.Cleanup(func() { ts.Close(); srv.CloseRelays() }) + + resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(`{"template":"rt:24.04","volumes":[{"name":"imagenet"}]}`)) + if err != nil { + t.Fatalf("claim: %v", err) + } + defer resp.Body.Close() + var got types.ClaimResponse + if err := json.NewDecoder(resp.Body).Decode(&got); err != nil { + t.Fatalf("decode: %v", err) + } + if resp.StatusCode != http.StatusOK || got.ID != tt.wantID || !slices.Equal(got.Redirect, tt.wantRedirect) { + t.Errorf("status=%d response=%+v", resp.StatusCode, got) + } + if tt.wantID != "" && !slices.Equal(got.Volumes, wantVolumes) { + t.Errorf("volumes=%+v, want %+v", got.Volumes, wantVolumes) + } + if mgr.warmCalls != 1 || mgr.provisionCalls != tt.wantProvisionCalls || + !slices.Equal(mgr.gotWarmVolumes, wantVolumes) { + t.Errorf("warm=%d warm volumes=%+v provision=%d", mgr.warmCalls, mgr.gotWarmVolumes, mgr.provisionCalls) + } + badCandidateNames := tt.wantCandidateCalls > 0 && + !slices.Equal(placer.volumeCandidateNames, []string{"imagenet"}) + if placer.candidateCalls != 0 || placer.volumeCandidateCalls != tt.wantCandidateCalls || badCandidateNames { + t.Errorf("candidates=%d volume candidates=%d names=%v", + placer.candidateCalls, placer.volumeCandidateCalls, placer.volumeCandidateNames) + } + }) + } +} + +func TestVolumeClaimUsesVolumeAndTemplateIntersections(t *testing.T) { + for _, tt := range []struct { + name string + localVolumes bool + localPromoted bool + hasGolden bool + templateOwners []string + volumeOwners []string + templateVolumeOwner []string + wantRedirect string + wantStatus int + wantVolumeCalls int + wantTemplateCalls int + wantWarmCalls int + wantCandidateCalls int + noRedirect bool + requirePromoted bool + wantPromoted bool + }{ + { + name: "ordinary local volume warm-misses then provisions", localVolumes: true, + wantStatus: http.StatusOK, wantWarmCalls: 1, wantCandidateCalls: 1, + }, + { + name: "configured golden with remote volume uses volume owner", hasGolden: true, + volumeOwners: []string{"volume:7777"}, wantRedirect: "volume:7777", + wantStatus: http.StatusOK, wantVolumeCalls: 1, wantCandidateCalls: 1, + }, + { + name: "unknown fleet volume fails without provisioning", + wantStatus: http.StatusBadRequest, wantVolumeCalls: 1, wantCandidateCalls: 1, + }, + { + name: "redirect target resolves locally without another hop", + volumeOwners: []string{"volume:7777"}, wantStatus: http.StatusBadRequest, noRedirect: true, + }, + { + name: "remote promoted template uses true intersection", localVolumes: true, + templateOwners: []string{"template:7777"}, templateVolumeOwner: []string{"both:7777"}, + wantRedirect: "both:7777", wantStatus: http.StatusOK, wantTemplateCalls: 1, wantPromoted: true, + }, + { + name: "redirect target provisions the required promoted template", localVolumes: true, + localPromoted: true, noRedirect: true, requirePromoted: true, + wantStatus: http.StatusOK, wantPromoted: true, + }, + { + name: "local promoted template with remote volume uses intersection", + localPromoted: true, volumeOwners: []string{"volume-only:7777"}, + templateVolumeOwner: []string{"both:7777"}, wantRedirect: "both:7777", + wantStatus: http.StatusOK, wantTemplateCalls: 1, wantPromoted: true, + }, + { + name: "shared-store template falls back to a volume holder", + localPromoted: true, volumeOwners: []string{"volume-only:7777"}, + wantRedirect: "volume-only:7777", wantStatus: http.StatusOK, + wantVolumeCalls: 1, wantTemplateCalls: 1, wantPromoted: true, + }, + { + name: "redirect target refuses missing promoted template before provisioning", + localVolumes: true, noRedirect: true, requirePromoted: true, + wantStatus: http.StatusBadRequest, wantPromoted: true, + }, + { + name: "promoted template refuses when no volume holder exists", + localPromoted: true, wantStatus: http.StatusBadRequest, + wantVolumeCalls: 1, wantTemplateCalls: 1, + }, + } { + t.Run(tt.name, func(t *testing.T) { + mgr := &fakeManager{ + hasGolden: tt.hasGolden, hasPromoted: tt.localPromoted, + volumePlacement: func(types.PoolKey, string, []string) (bool, error) { return tt.localVolumes, nil }, + } + placer := &fakePlacer{ + addrs: []string{"warm-peer:7777"}, owners: tt.templateOwners, + volumeOwners: tt.volumeOwners, templateVolumeOwners: tt.templateVolumeOwner, + } + srv := New("", nil, "node-a:7777", mgr, &fakeDialer{}, placer, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + t.Cleanup(func() { ts.Close(); srv.CloseRelays() }) + + request := types.ClaimRequest{ + Template: "tpl", Volumes: []types.Volume{{Name: "imagenet"}}, + NoRedirect: tt.noRedirect, RequirePromoted: tt.requirePromoted, + } + body, err := json.Marshal(request) + if err != nil { + t.Fatalf("encode request: %v", err) + } + resp, err := http.Post(ts.URL+"/v1/claim", "application/json", bytes.NewReader(body)) + if err != nil { + t.Fatalf("claim: %v", err) + } + defer resp.Body.Close() + if resp.StatusCode != tt.wantStatus { + t.Fatalf("status=%d, want %d", resp.StatusCode, tt.wantStatus) + } + if tt.wantStatus == http.StatusOK { + assertVolumeClaimResponse(t, resp.Body, tt.wantRedirect, tt.wantPromoted) + } + wantProvision := 0 + if tt.wantStatus == http.StatusOK && tt.wantRedirect == "" { + wantProvision = 1 + } + if mgr.provisionCalls != wantProvision { + t.Errorf("provision calls=%d, want %d", mgr.provisionCalls, wantProvision) + } + if mgr.gotRequirePromoted != (wantProvision == 1 && tt.wantPromoted) { + t.Errorf("promoted provision=%v", mgr.gotRequirePromoted) + } + if mgr.warmCalls != tt.wantWarmCalls || placer.candidateCalls != 0 || + placer.volumeCandidateCalls != tt.wantCandidateCalls || + placer.volumeOwnerCalls != tt.wantVolumeCalls || placer.templateVolumeCalls != tt.wantTemplateCalls { + t.Errorf("warm=%d candidates=%d volume candidates=%d volume owners=%d template-volume owners=%d", + mgr.warmCalls, placer.candidateCalls, placer.volumeCandidateCalls, + placer.volumeOwnerCalls, placer.templateVolumeCalls) + } + }) + } +} + +func TestVolumeClaimValidatesKeyBeforePlacement(t *testing.T) { + mgr := &fakeManager{ + volumePlacement: func(types.PoolKey, string, []string) (bool, error) { + return false, pool.ErrNoEgress + }, + } + placer := &fakePlacer{volumeOwners: []string{"volume:7777"}} + srv := New("", nil, "node-a:7777", mgr, &fakeDialer{}, placer, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + t.Cleanup(func() { ts.Close(); srv.CloseRelays() }) + + resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader( + `{"template":"rt:24.04","net":"egress","volumes":[{"name":"imagenet"}]}`)) + if err != nil { + t.Fatalf("claim: %v", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusConflict { + t.Errorf("status=%d, want 409", resp.StatusCode) + } + if mgr.provisionCalls != 0 || placer.volumeOwnerCalls != 0 || placer.templateVolumeCalls != 0 { + t.Errorf("provision=%d volume owners=%d template-volume owners=%d", + mgr.provisionCalls, placer.volumeOwnerCalls, placer.templateVolumeCalls) + } +} + +func TestVolumeClaimQuotaDoesNotRedirect(t *testing.T) { + mgr := &fakeManager{warmClaim: func(context.Context, types.PoolKey, time.Duration) (*types.Sandbox, error) { + return nil, pool.ErrQuota + }} + placer := &fakePlacer{ + addrs: []string{"wrong-general-candidate:7777"}, volumeCandidates: []string{"wrong-volume-candidate:7777"}, + } + srv := New("", nil, "node-a:7777", mgr, &fakeDialer{}, placer, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + t.Cleanup(func() { ts.Close(); srv.CloseRelays() }) + + resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(`{"template":"rt:24.04","volumes":[{"name":"imagenet"}]}`)) + if err != nil { + t.Fatalf("claim: %v", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusTooManyRequests { + t.Errorf("status=%d, want 429", resp.StatusCode) + } + if mgr.warmCalls != 1 || mgr.provisionCalls != 0 || + placer.candidateCalls != 0 || placer.volumeCandidateCalls != 0 { + t.Errorf("warm=%d provision=%d candidates=%d volume candidates=%d", + mgr.warmCalls, mgr.provisionCalls, placer.candidateCalls, placer.volumeCandidateCalls) + } +} + +func TestVolumeClaimRejectsShapeBeforePlacement(t *testing.T) { + for _, body := range []string{ + `{"template":"rt:24.04","volumes":["data"]}`, + `{"template":"rt:24.04","volumes":[{"name":"data"},{"name":"data"}]}`, + `{"template":"rt:24.04","volumes":[{"name":"cocoon-data"}]}`, + `{"template":"rt:24.04","engine":"fc","volumes":[{"name":"data"}]}`, + `{"template":"rt:24.04","volumes":[{"name":"data","mount":"relative"}]}`, + `{"template":"rt:24.04","volumes":[{"name":"data","mount":"/datasets"},{"name":"other","mount":"/datasets/nested"}]}`, + `{"template":"rt:24.04","volumes":[{"name":"a"},{"name":"b"},{"name":"c"},{"name":"d"},{"name":"e"},{"name":"f"},{"name":"g"},{"name":"h"},{"name":"i"}]}`, + } { + mgr := &fakeManager{} + placer := &fakePlacer{addrs: []string{"warm-peer:7777"}, owners: []string{"owner:7777"}} + srv := New("", nil, "node-a:7777", mgr, &fakeDialer{}, placer, nil, nil, nil) + ts := httptest.NewServer(srv.Handler()) + + resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(body)) + if err != nil { + t.Fatalf("claim: %v", err) + } + resp.Body.Close() + ts.Close() + srv.CloseRelays() + if resp.StatusCode != http.StatusBadRequest { + t.Errorf("body=%s status=%d, want 400", body, resp.StatusCode) + } + if mgr.provisionCalls != 0 || placer.candidateCalls != 0 { + t.Errorf("body=%s provision=%d candidates=%d", body, mgr.provisionCalls, placer.candidateCalls) + } + } +} + func TestOwnerEndpoint(t *testing.T) { // AgentSocket succeeds → this node owns it → 200 with owner addr. mgr := &fakeManager{socket: func(id, token string) (string, error) { @@ -1362,6 +1750,26 @@ func TestCheckpointClaimProbesRealPeerAndRedirects(t *testing.T) { } } +func assertVolumeClaimResponse(t *testing.T, body io.Reader, wantRedirect string, wantRequirePromoted bool) { + t.Helper() + var got types.ClaimResponse + if err := json.NewDecoder(body).Decode(&got); err != nil { + t.Fatalf("decode: %v", err) + } + if wantRedirect != "" { + if !slices.Equal(got.Redirect, []string{wantRedirect}) { + t.Errorf("redirect=%v, want [%s]", got.Redirect, wantRedirect) + } + if got.RequirePromoted != wantRequirePromoted { + t.Errorf("require_promoted=%v, want %v", got.RequirePromoted, wantRequirePromoted) + } + return + } + if got.ID == "" || len(got.Redirect) != 0 { + t.Errorf("response=%+v, want local claim", got) + } +} + func newTestServer(t *testing.T, apiToken string, mgr Manager, dialer Dialer) *httptest.Server { t.Helper() return newTenantTestServer(t, apiToken, nil, mgr, dialer) @@ -1403,14 +1811,14 @@ func credToken(cred pool.Cred) string { return cred.Token } -// fakeManager implements Manager with overridable behavior. ClaimWarm always -// misses, so the server's warm-miss → redirect → provision path is exercised; -// the claim hook stands in for the provision result. Tenant-scoped methods -// record the tenant they were handed in gotTenant. +// fakeManager implements Manager with overridable behavior. ClaimWarm misses +// unless warmClaim is set; claim stands in for the provision result. +// Tenant-scoped methods record the tenant they were handed in gotTenant. type fakeManager struct { ckptDir string hasCheckpoint map[string]bool claim func(ctx context.Context, key types.PoolKey, ttl time.Duration) (*types.Sandbox, error) + warmClaim func(ctx context.Context, key types.PoolKey, ttl time.Duration) (*types.Sandbox, error) release func(id, token string) error releaseOp func(id string) error socket func(id, token string) (string, error) @@ -1422,6 +1830,7 @@ type fakeManager struct { deleteGolden func(key types.PoolKey) error hasGolden bool + hasPromoted bool audited func(id string, line []byte) checkpoint func(id, token, name string) (types.Checkpoint, error) @@ -1434,26 +1843,40 @@ type fakeManager struct { infoPools []pool.PoolInfo claimDeadline func(id, token string) (time.Time, error) - gotTenant string - gotClaimRef string - gotNoForward bool - tenantClaims map[string]int - draining bool -} - -func (f *fakeManager) ClaimWarm(_ context.Context, _ types.PoolKey, _ time.Duration, tenant, claimRef string) (*types.Sandbox, error) { + gotTenant string + gotClaimRef string + gotVolumes []types.Volume + gotWarmVolumes []types.Volume + gotRequirePromoted bool + warmCalls int + provisionCalls int + gotNoForward bool + tenantClaims map[string]int + volumePlacement func(key types.PoolKey, tenant string, names []string) (bool, error) + placementCalls int + placementNames []string + volumeCatalog func(tenant string, holders map[string]int) []types.VolumeInfo + sandboxIndex func(tenant string) []pool.SandboxSummary + draining bool +} + +func (f *fakeManager) ClaimWarm(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) { + f.warmCalls++ f.gotTenant = tenant f.gotClaimRef = claimRef - return nil, pool.ErrNoWarm + f.gotWarmVolumes = slices.Clone(volumes) + if f.warmClaim == nil { + return nil, pool.ErrNoWarm + } + return f.warmClaim(ctx, key, ttl) } -func (f *fakeManager) ClaimProvision(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string) (*types.Sandbox, error) { - f.gotTenant = tenant - f.gotClaimRef = claimRef - if f.claim == nil { - return &types.Sandbox{ID: "sb_1", Token: "tok"}, nil - } - return f.claim(ctx, key, ttl) +func (f *fakeManager) ClaimProvision(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) { + return fakeClaimProvision(f, ctx, key, ttl, tenant, claimRef, volumes, false) +} + +func (f *fakeManager) ClaimProvisionPromoted(ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume) (*types.Sandbox, error) { + return fakeClaimProvision(f, ctx, key, ttl, tenant, claimRef, volumes, true) } // Release dispatches to the operator or per-sandbox hook, mirroring the @@ -1519,6 +1942,10 @@ func (f *fakeManager) HasGolden(context.Context, types.PoolKey) bool { return f.hasGolden } +func (f *fakeManager) HasPromotedTemplate(context.Context, types.PoolKey) bool { + return f.hasPromoted +} + func (f *fakeManager) ClaimDeadline(id, token string) (time.Time, error) { if f.claimDeadline != nil { return f.claimDeadline(id, token) @@ -1536,7 +1963,31 @@ func (f *fakeManager) Counters() pool.Counters { return pool.Counters{} } func (f *fakeManager) TenantClaims() map[string]int { return f.tenantClaims } -func (f *fakeManager) Sandboxes() []pool.SandboxSummary { return nil } +func (f *fakeManager) VolumePlacement(key types.PoolKey, tenant string, names []string) (bool, error) { + f.placementCalls++ + f.gotTenant = tenant + f.placementNames = slices.Clone(names) + if f.volumePlacement == nil { + return true, nil + } + return f.volumePlacement(key, tenant, names) +} + +func (f *fakeManager) Volumes(tenant string, holders map[string]int) []types.VolumeInfo { + f.gotTenant = tenant + if f.volumeCatalog == nil { + return nil + } + return f.volumeCatalog(tenant, holders) +} + +func (f *fakeManager) Sandboxes(tenant string) []pool.SandboxSummary { + f.gotTenant = tenant + if f.sandboxIndex == nil { + return nil + } + return f.sandboxIndex(tenant) +} func (f *fakeManager) Sandbox(string) (pool.SandboxSummary, bool) { return pool.SandboxSummary{}, false @@ -1615,6 +2066,21 @@ func (f *fakeManager) Drain(context.Context) { f.draining = true } func (f *fakeManager) Uncordon(context.Context) { f.draining = false } +// FetchCheckpoint serves the peer-transfer read; the fake reports every record +// missing unless a test supplies a directory. +func (f *fakeManager) FetchCheckpoint(_ context.Context, _ string) (string, []byte, func(), error) { + if f.ckptDir == "" { + return "", nil, nil, pool.ErrUnknownCheckpoint + } + return f.ckptDir, []byte(`{"id":"ck_00000000000000aa"}`), func() {}, nil +} + +// HasCheckpoint answers the probe endpoint straight from the fake's map; +// unset ids report false. +func (f *fakeManager) HasCheckpoint(_ context.Context, ckptID string) bool { + return f.hasCheckpoint[ckptID] +} + type fakeDialer struct { dial func(ctx context.Context, sock string) (net.Conn, error) } @@ -1628,14 +2094,42 @@ func (f *fakeDialer) DialSilkd(ctx context.Context, sock string) (net.Conn, erro } type fakePlacer struct { - addrs []string - owners []string + addrs []string + owners []string + volumeCandidates []string + volumeOwners []string + templateVolumeOwners []string + volumeHolders map[string]int + candidateCalls int + volumeCandidateCalls int + volumeCandidateNames []string + volumeOwnerCalls int + templateVolumeCalls int +} + +func (f *fakePlacer) Candidates(string) []string { + f.candidateCalls++ + return f.addrs } -func (f *fakePlacer) Candidates(string) []string { return f.addrs } +func (f *fakePlacer) VolumeCandidates(_ string, names []string) []string { + f.volumeCandidateCalls++ + f.volumeCandidateNames = slices.Clone(names) + return f.volumeCandidates +} func (f *fakePlacer) TemplateOwners(string) []string { return f.owners } -func (f *fakePlacer) PeerAddrs() []string { return f.addrs } -func (f *fakePlacer) ConfigMismatches() int { return 0 } +func (f *fakePlacer) VolumeOwners([]string) []string { + f.volumeOwnerCalls++ + return f.volumeOwners +} + +func (f *fakePlacer) TemplateVolumeOwners(string, []string) []string { + f.templateVolumeCalls++ + return f.templateVolumeOwners +} +func (f *fakePlacer) VolumeHolders() map[string]int { return f.volumeHolders } +func (f *fakePlacer) PeerAddrs() []string { return f.addrs } +func (f *fakePlacer) ConfigMismatches() int { return 0 } // fakeProber stubs CheckpointProber with a fixed answer, standing in for a // live HEAD fan-out. forgotten records every id Forget was called with, so @@ -1648,21 +2142,6 @@ type fakeProber struct { func (f *fakeProber) Owners(context.Context, string) []string { return f.owners } func (f *fakeProber) Forget(id string) { f.forgotten = append(f.forgotten, id) } -// FetchCheckpoint serves the peer-transfer read; the fake reports every record -// missing unless a test supplies a directory. -func (f *fakeManager) FetchCheckpoint(_ context.Context, _ string) (string, []byte, func(), error) { - if f.ckptDir == "" { - return "", nil, nil, pool.ErrUnknownCheckpoint - } - return f.ckptDir, []byte(`{"id":"ck_00000000000000aa"}`), func() {}, nil -} - -// HasCheckpoint answers the probe endpoint straight from the fake's map; -// unset ids report false. -func (f *fakeManager) HasCheckpoint(_ context.Context, ckptID string) bool { - return f.hasCheckpoint[ckptID] -} - // newPlacerTestServer builds a server with a checkpoint prober, which the // shared helper deliberately leaves nil (most tests are single-node); the // mesh placer stays nil throughout — these tests exercise checkpoint claims @@ -1688,3 +2167,15 @@ func postJSON(t *testing.T, url, token, body string) *http.Response { } return resp } + +func fakeClaimProvision(f *fakeManager, ctx context.Context, key types.PoolKey, ttl time.Duration, tenant, claimRef string, volumes []types.Volume, requirePromoted bool) (*types.Sandbox, error) { + f.provisionCalls++ + f.gotTenant = tenant + f.gotClaimRef = claimRef + f.gotVolumes = slices.Clone(volumes) + f.gotRequirePromoted = requirePromoted + if f.claim == nil { + return &types.Sandbox{ID: "sb_1", Token: "tok", Volumes: slices.Clone(volumes)}, nil + } + return f.claim(ctx, key, ttl) +} diff --git a/sandboxd/store/dir/dir.go b/sandboxd/store/dir/dir.go index 48836c2e..4bbb3a51 100644 --- a/sandboxd/store/dir/dir.go +++ b/sandboxd/store/dir/dir.go @@ -22,6 +22,7 @@ import ( "golang.org/x/sync/errgroup" "github.com/cocoonstack/sandbox/sandboxd/store" + "github.com/cocoonstack/sandbox/sandboxd/utils" ) const ( @@ -168,19 +169,10 @@ func (d *Store) Delete(_ context.Context, id string) error { // SweepStaging clears crashed staging and delegates generation retention. func (d *Store) SweepStaging() error { - entries, err := os.ReadDir(d.root) - if err != nil { + // ReadDir + suffix, not Glob: the root path may hold glob metacharacters. + if err := utils.RemoveDirEntries(d.root, func(name string) bool { return strings.HasSuffix(name, ".tmp") }); err != nil { return err } - // ReadDir + suffix, not Glob: the root path may hold glob - // metacharacters. - for _, e := range entries { - if strings.HasSuffix(e.Name(), ".tmp") { - if err := os.RemoveAll(filepath.Join(d.root, e.Name())); err != nil { - return err - } - } - } return d.SweepGenerations() } diff --git a/sandboxd/store/peer/peer.go b/sandboxd/store/peer/peer.go index 37988245..2db1a134 100644 --- a/sandboxd/store/peer/peer.go +++ b/sandboxd/store/peer/peer.go @@ -5,11 +5,10 @@ import ( "context" "errors" "fmt" - "os" - "path/filepath" "time" "github.com/cocoonstack/sandbox/sandboxd/store" + "github.com/cocoonstack/sandbox/sandboxd/utils" ) // healBudget bounds one Pull across every owner tried, not each. @@ -61,7 +60,8 @@ func (h *Healer) pullFrom(ctx context.Context, id, staging string, addrs []strin perOwner := budget / time.Duration(len(addrs)) var errs []error for _, addr := range addrs { - if err := clearDir(staging); err != nil { + // One peer's rejected or partial transfer must not linger into the next's. + if err := utils.RemoveDirEntries(staging, nil); err != nil { return fmt.Errorf("reset staging: %w", err) } attemptCtx, cancel := context.WithTimeout(ctx, perOwner) @@ -87,18 +87,3 @@ func (h *Healer) pullFrom(ctx context.Context, id, staging string, addrs []strin } return store.ErrNotFound // every owner answered not-found: stale gossip } - -// clearDir empties dir between owner attempts, so one peer's rejected or -// partial transfer cannot linger into the next's. -func clearDir(dir string) error { - entries, err := os.ReadDir(dir) - if err != nil { - return err - } - for _, e := range entries { - if err := os.RemoveAll(filepath.Join(dir, e.Name())); err != nil { - return err - } - } - return nil -} diff --git a/sandboxd/store/peer/probe.go b/sandboxd/store/peer/probe.go index 1fc9b419..ba0554a0 100644 --- a/sandboxd/store/peer/probe.go +++ b/sandboxd/store/peer/probe.go @@ -7,6 +7,7 @@ import ( "crypto/sha256" "encoding/base64" "encoding/binary" + "maps" "net/http" "sync" "time" @@ -192,11 +193,7 @@ func (p *HTTPProber) cachePut(id string, owners []string, start uint64) { p.cache = make(map[string]redirectCacheEntry) } now := time.Now() - for k, e := range p.cache { - if now.After(e.expires) { - delete(p.cache, k) - } - } + maps.DeleteFunc(p.cache, func(_ string, e redirectCacheEntry) bool { return now.After(e.expires) }) p.cache[id] = redirectCacheEntry{owners: owners, expires: now.Add(cmp.Or(p.cacheTTL, redirectCacheTTL))} } diff --git a/sandboxd/store/s3/s3.go b/sandboxd/store/s3/s3.go index b6dbfacd..6985886e 100644 --- a/sandboxd/store/s3/s3.go +++ b/sandboxd/store/s3/s3.go @@ -26,6 +26,7 @@ import ( "golang.org/x/sync/singleflight" "github.com/cocoonstack/sandbox/sandboxd/store" + "github.com/cocoonstack/sandbox/sandboxd/utils" ) const ( @@ -206,18 +207,7 @@ func (s *Store) Delete(ctx context.Context, id string) error { // it runs at startup, when no clone can be mid-flight. A crash between // upload and meta.json leaves orphan objects invisible to Metas; an S3 // lifecycle rule on the bucket reclaims those (documented in deploy). -func (s *Store) SweepStaging() error { - entries, err := os.ReadDir(s.staging) - if err != nil { - return err - } - for _, e := range entries { - if err := os.RemoveAll(filepath.Join(s.staging, e.Name())); err != nil { - return err - } - } - return nil -} +func (s *Store) SweepStaging() error { return utils.RemoveDirEntries(s.staging, nil) } // SweepGenerations is a no-op: Delete reclaims committed S3 generations, and // bucket lifecycle policy handles invisible upload orphans. diff --git a/sandboxd/types/api.go b/sandboxd/types/api.go index 43ae7202..bd3bf11d 100644 --- a/sandboxd/types/api.go +++ b/sandboxd/types/api.go @@ -23,8 +23,12 @@ type ClaimRequest struct { Net NetShape `json:"net,omitempty"` Size Size `json:"size,omitempty"` Engine Engine `json:"engine,omitempty"` + Volumes []Volume `json:"volumes,omitempty"` TTLField NoRedirect bool `json:"no_redirect,omitempty"` + // RequirePromoted is carried from a promoted-volume redirect to make the + // target refuse a cold-image fallback when template gossip is stale. + RequirePromoted bool `json:"require_promoted,omitempty"` // ClaimRef is an opaque caller reference (the aggregated apiserver passes // the k8s "/") recorded on the claim so the read path can // map a listed sandbox back to the name it was claimed under. @@ -50,9 +54,27 @@ type ClaimResponse struct { // FromCheckpoint names the checkpoint a branched claim was born from, // so clients can reconstruct the checkpoint tree. - FromCheckpoint string `json:"from_checkpoint,omitempty"` + FromCheckpoint string `json:"from_checkpoint,omitempty"` + Volumes []Volume `json:"volumes,omitempty"` Redirect []string `json:"redirect,omitempty"` + // RequirePromoted tells a redirecting client to preserve that requirement + // on its no_redirect retry. It is omitted from successful claims. + RequirePromoted bool `json:"require_promoted,omitempty"` +} + +// VolumeInfo is the caller-visible, host-path-free catalog projection. +type VolumeInfo struct { + Name string `json:"name"` + DefaultMount string `json:"default_mount"` + SizeBytes int64 `json:"size_bytes"` + Available bool `json:"available"` + Nodes int `json:"nodes"` +} + +// VolumeListResponse is the wire reply of GET /v1/volumes. +type VolumeListResponse struct { + Volumes []VolumeInfo `json:"volumes"` } // ForkRequest is the wire body of POST /v1/sandboxes/{id}/fork. The diff --git a/sandboxd/types/types.go b/sandboxd/types/types.go index 0216d01e..c7183eaa 100644 --- a/sandboxd/types/types.go +++ b/sandboxd/types/types.go @@ -5,8 +5,11 @@ import ( "cmp" "crypto/sha256" "encoding/hex" + "errors" "fmt" + "path/filepath" "regexp" + "strings" "sync" "sync/atomic" "time" @@ -27,6 +30,12 @@ const ( EngineCH Engine = "ch" EngineFC Engine = "fc" + + MaxClaimVolumes = 8 + + DirectIOOn = "on" + DirectIOOff = "off" + DirectIOAuto = "auto" ) var ( @@ -34,6 +43,8 @@ var ( // which ride in journal fields and metric labels: one conservative // charset, also accepted by cocoon's snapshot naming. NameRe = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._:/-]{0,62}$`) + // VolumeNameRe is the virtio disk serial grammar shared with cocoon. + VolumeNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,19}$`) sizeSpecs = map[Size]SizeSpec{ SizeSmall: {CPU: 1, Memory: "512M", MemoryBytes: 512 << 20}, @@ -41,6 +52,11 @@ var ( SizeLarge: {CPU: 4, Memory: "4G", MemoryBytes: 4 << 30}, SizeXLarge: {CPU: 4, Memory: "8G", MemoryBytes: 8 << 30}, } + + guestOSMountRoots = []string{ + "/bin", "/boot", "/dev", "/etc", "/lib", "/lib64", "/proc", + "/run", "/sbin", "/sys", "/usr", "/var", + } ) // NetShape selects whether the Cloud Hypervisor guest has a NIC. @@ -163,6 +179,8 @@ type Sandbox struct { // the operator index so a listed sandbox maps back to its claim name. // Empty for warm-pool, fork, and checkpoint-branch claims. ClaimRef string `json:"claim_ref,omitempty"` + // Volumes records the read-only volumes successfully applied to this claim. + Volumes []Volume `json:"volumes,omitempty"` VsockSocket string `json:"vsock_socket,omitempty"` // TAP is the egress-lane NIC's host tap, captured at provision; empty on @@ -253,3 +271,96 @@ type VMNetConfig struct { type VMConfig struct { Name string `json:"name"` } + +// Volume is one requested or applied read-only dataset mount. Mount is empty +// only before request validation; persisted and response entries are effective. +type Volume struct { + Name string `json:"name"` + Mount string `json:"mount,omitempty"` +} + +// ValidVolumeName reports whether name is a legal cocoon data-disk serial. +func ValidVolumeName(name string) bool { + return VolumeNameRe.MatchString(name) && !strings.HasPrefix(name, "cocoon-") +} + +// DefaultVolumeMount returns the guest mount used when a request omits one. +func DefaultVolumeMount(name string) string { + return "/volumes/" + name +} + +// ValidDirectIO reports whether mode is a legal volume direct-I/O setting. +func ValidDirectIO(mode string) bool { + return mode == DirectIOOn || mode == DirectIOOff || mode == DirectIOAuto +} + +// VolumeNames projects the entries' names in order; nil for none. +func VolumeNames(volumes []Volume) []string { + if len(volumes) == 0 { + return nil + } + names := make([]string, len(volumes)) + for i, volume := range volumes { + names[i] = volume.Name + } + return names +} + +// ValidateVolumes validates a request and returns detached entries with every +// default mount filled. The input is not modified. +func ValidateVolumes(volumes []Volume) ([]Volume, error) { + if len(volumes) > MaxClaimVolumes { + return nil, fmt.Errorf("volumes must contain at most %d entries, got %d", MaxClaimVolumes, len(volumes)) + } + applied := make([]Volume, len(volumes)) + names := make(map[string]struct{}, len(volumes)) + for i, volume := range volumes { + if !ValidVolumeName(volume.Name) { + return nil, fmt.Errorf("volumes[%d] name %q must match %s and not start with cocoon-", i, volume.Name, VolumeNameRe) + } + if _, ok := names[volume.Name]; ok { + return nil, fmt.Errorf("volumes[%d] duplicates name %q", i, volume.Name) + } + names[volume.Name] = struct{}{} + mount := volume.Mount + if mount == "" { + mount = DefaultVolumeMount(volume.Name) + } + if err := validateVolumeMount(mount); err != nil { + return nil, fmt.Errorf("volumes[%d] mount %q: %w", i, mount, err) + } + for j := range i { + other := applied[j].Mount + switch { + case mount == other: + return nil, fmt.Errorf("volumes[%d] mount %q duplicates volumes[%d]", i, mount, j) + case pathWithin(other, mount), pathWithin(mount, other): + return nil, fmt.Errorf("volumes[%d] mount %q nests with volumes[%d] mount %q", i, mount, j, other) + } + } + applied[i] = Volume{Name: volume.Name, Mount: mount} + } + return applied, nil +} + +func validateVolumeMount(mount string) error { + if !filepath.IsAbs(mount) { + return errors.New("must be absolute") + } + if filepath.Clean(mount) != mount { + return errors.New("must be clean") + } + if mount == "/" { + return errors.New("must be outside the guest OS tree") + } + for _, root := range guestOSMountRoots { + if mount == root || pathWithin(root, mount) { + return errors.New("must be outside the guest OS tree") + } + } + return nil +} + +func pathWithin(parent, child string) bool { + return strings.HasPrefix(child, parent+"/") +} diff --git a/sandboxd/types/volume_test.go b/sandboxd/types/volume_test.go new file mode 100644 index 00000000..581be644 --- /dev/null +++ b/sandboxd/types/volume_test.go @@ -0,0 +1,101 @@ +package types + +import ( + "slices" + "testing" +) + +func TestValidVolumeName(t *testing.T) { + for _, tt := range []struct { + name string + valid bool + }{ + {"dataset", true}, + {"a_b-0", true}, + {"abcdefghijklmnopqrst", true}, + {"", false}, + {"Dataset", false}, + {"1dataset", false}, + {"data/set", false}, + {"abcdefghijklmnopqrstu", false}, + {"cocoon-data", false}, + } { + t.Run(tt.name, func(t *testing.T) { + if got := ValidVolumeName(tt.name); got != tt.valid { + t.Errorf("ValidVolumeName(%q) = %v, want %v", tt.name, got, tt.valid) + } + }) + } +} + +func TestValidateVolumes(t *testing.T) { + volumes := []Volume{{Name: "dataset"}, {Name: "weights-1", Mount: "/models"}} + wantInput := slices.Clone(volumes) + got, err := ValidateVolumes(volumes) + if err != nil { + t.Fatalf("ValidateVolumes: %v", err) + } + want := []Volume{ + {Name: "dataset", Mount: "/volumes/dataset"}, + {Name: "weights-1", Mount: "/models"}, + } + if !slices.Equal(got, want) { + t.Errorf("volumes %v, want %v", got, want) + } + if !slices.Equal(volumes, wantInput) { + t.Errorf("input mutated to %v, want %v", volumes, wantInput) + } + if got, err := ValidateVolumes([]Volume{ + {Name: "a"}, + {Name: "b"}, + {Name: "c"}, + {Name: "d"}, + {Name: "e"}, + {Name: "f"}, + {Name: "g"}, + {Name: "h"}, + }); err != nil || len(got) != MaxClaimVolumes { + t.Errorf("ValidateVolumes at limit: got=%v err=%v", got, err) + } + + for _, tt := range []struct { + name string + volumes []Volume + }{ + {"too-many", []Volume{{Name: "a"}, {Name: "b"}, {Name: "c"}, {Name: "d"}, {Name: "e"}, {Name: "f"}, {Name: "g"}, {Name: "h"}, {Name: "i"}}}, + {"invalid-name", []Volume{{Name: "bad/name"}}}, + {"duplicate-name", []Volume{{Name: "dataset"}, {Name: "dataset", Mount: "/other"}}}, + {"relative-mount", []Volume{{Name: "dataset", Mount: "data"}}}, + {"unclean-mount", []Volume{{Name: "dataset", Mount: "/data/../dataset"}}}, + {"doubled-separator", []Volume{{Name: "dataset", Mount: "/data//dataset"}}}, + {"duplicate-mount", []Volume{{Name: "a", Mount: "/data"}, {Name: "b", Mount: "/data"}}}, + {"nested-mount", []Volume{{Name: "a", Mount: "/data"}, {Name: "b", Mount: "/data/child"}}}, + {"parent-after-child", []Volume{{Name: "a", Mount: "/data/child"}, {Name: "b", Mount: "/data"}}}, + } { + t.Run(tt.name, func(t *testing.T) { + if _, err := ValidateVolumes(tt.volumes); err == nil { + t.Fatal("ValidateVolumes succeeded") + } + }) + } +} + +func TestValidateVolumesRejectsGuestOSMounts(t *testing.T) { + for _, root := range append([]string{"/"}, guestOSMountRoots...) { + t.Run(root, func(t *testing.T) { + if _, err := ValidateVolumes([]Volume{{Name: "dataset", Mount: root}}); err == nil { + t.Fatalf("accepted OS mount %q", root) + } + if root != "/" { + if _, err := ValidateVolumes([]Volume{{Name: "dataset", Mount: root + "/child"}}); err == nil { + t.Fatalf("accepted mount under OS root %q", root) + } + } + }) + } + for _, mount := range []string{"/data", "/home/dataset", "/opt-data", "/usrdata"} { + if _, err := ValidateVolumes([]Volume{{Name: "dataset", Mount: mount}}); err != nil { + t.Errorf("rejected allowed mount %q: %v", mount, err) + } + } +} diff --git a/sandboxd/utils/utils.go b/sandboxd/utils/utils.go index 4594215f..a6ab900b 100644 --- a/sandboxd/utils/utils.go +++ b/sandboxd/utils/utils.go @@ -7,11 +7,19 @@ import ( "errors" "fmt" "io" + "net" "os" "path/filepath" "strings" ) +// CloseWrite signals EOF to the peer without tearing down the read direction. +func CloseWrite(conn net.Conn) { + if cw, ok := conn.(interface{ CloseWrite() error }); ok { + _ = cw.CloseWrite() + } +} + // WriteFileSync durably replaces path with data (temp + fsync + rename + dir // fsync) so the rename survives a crash. The temp name is per-call unique, so // concurrent writers to the same path can't race a shared temp into a lost rename. @@ -58,6 +66,23 @@ func WriteFileSync(path string, data []byte, perm os.FileMode) error { return nil } +// RemoveDirEntries removes dir's entries for which match returns true; nil matches all. +func RemoveDirEntries(dir string, match func(name string) bool) error { + entries, err := os.ReadDir(dir) + if err != nil { + return err + } + for _, e := range entries { + if match != nil && !match(e.Name()) { + continue + } + if err := os.RemoveAll(filepath.Join(dir, e.Name())); err != nil { + return err + } + } + return nil +} + // DecodeStrictJSON decodes one JSON value into v, refusing unknown fields, // duplicated keys, and trailing data — for hand-edited operator input, where // a typo must fail instead of silently changing what was configured. diff --git a/scripts/sandboxd-e2e.sh b/scripts/sandboxd-e2e.sh index fcfc0326..465b43e2 100755 --- a/scripts/sandboxd-e2e.sh +++ b/scripts/sandboxd-e2e.sh @@ -8,6 +8,27 @@ ADDR=${ADDR:-127.0.0.1:7777} TOKEN=${TOKEN:-e2e} WARM=${WARM:-2} REPO=$(cd "$(dirname "$0")/.." && pwd) +# VOLUME_IMAGE opts into the CH volume proof; its filesystem must contain the non-empty VOLUME_PROBE file. +VOLUME_IMAGE=${VOLUME_IMAGE:-} +VOLUME_NAME=${VOLUME_NAME:-e2e-data} +VOLUME_PROBE=${VOLUME_PROBE:-volume-e2e.txt} +VOLUME_DIRECTIO=${VOLUME_DIRECTIO:-off} + +VOLUME_CHECKSUM="" +if [[ -n $VOLUME_IMAGE ]]; then + [[ $VOLUME_IMAGE == /* ]] || { echo "VOLUME_IMAGE must be absolute"; exit 1; } + [[ -f $VOLUME_IMAGE && -r $VOLUME_IMAGE ]] || { echo "VOLUME_IMAGE must be a readable file"; exit 1; } + [[ $VOLUME_NAME =~ ^[a-z][a-z0-9_-]{0,19}$ && $VOLUME_NAME != cocoon-* ]] || { + echo "VOLUME_NAME must match ^[a-z][a-z0-9_-]{0,19}$ and not start with cocoon-" + exit 1 + } + [[ $VOLUME_DIRECTIO == on || $VOLUME_DIRECTIO == off || $VOLUME_DIRECTIO == auto ]] || { + echo "VOLUME_DIRECTIO must be on, off, or auto" + exit 1 + } + (( WARM >= 2 )) || { echo "volume e2e needs WARM>=2 for the concurrent warm-volume assertion"; exit 1; } + VOLUME_CHECKSUM=$(sha256sum -- "$VOLUME_IMAGE" | awk '{print $1}') +fi DATA=$(mktemp -d /tmp/sandboxd-e2e.XXXXXX) DAEMON_PID="" @@ -34,6 +55,10 @@ cleanup() { trap cleanup EXIT api() { curl -sf -H "Authorization: Bearer $TOKEN" "http://$ADDR/v1/$1"; } +metrics() { curl -sf -H "Authorization: Bearer $TOKEN" "http://$ADDR/metrics"; } +warm_claims() { + metrics | awk '$1 == "sandboxd_claims_total{tier=\"warm\"}" {print $2}' +} start_daemon() { "$DATA/sandboxd" -config "$DATA/config.json" >>"$DATA/daemon.log" 2>&1 & @@ -50,10 +75,17 @@ echo "== build" # Prebuilt binaries let the script run on nodes without a Go toolchain. if [[ -n ${SANDBOXD_BIN:-} && -n ${DEMO_BIN:-} && -n ${SMOKE_BIN:-} ]]; then cp "$SANDBOXD_BIN" "$DATA/sandboxd" && cp "$DEMO_BIN" "$DATA/demo" && cp "$SMOKE_BIN" "$DATA/smoke" + if [[ -n $VOLUME_IMAGE ]]; then + [[ -n ${VOLUME_SMOKE_BIN:-} ]] || { echo "VOLUME_SMOKE_BIN is required with prebuilt binaries"; exit 1; } + cp "$VOLUME_SMOKE_BIN" "$DATA/volumesmoke" + fi else (cd "$REPO/sandboxd" && GOWORK=off go build -o "$DATA/sandboxd" .) (cd "$REPO/e2e" && GOWORK=off go build -o "$DATA/demo" ./cmd/demo) (cd "$REPO/e2e" && GOWORK=off go build -o "$DATA/smoke" ./cmd/smoke) + if [[ -n $VOLUME_IMAGE ]]; then + (cd "$REPO/e2e" && GOWORK=off go build -o "$DATA/volumesmoke" ./cmd/volumesmoke) + fi fi # BRIDGE (optional) attaches the node to an egress bridge, adds a small @@ -75,7 +107,14 @@ if [[ -n ${S3_ENDPOINT:-} ]]; then STORE_LINE="\"checkpoint_store\": {\"kind\": \"s3\", \"s3\": {\"bucket\": \"${S3_BUCKET:-sbx-checkpoints}\", \"prefix\": \"e2e/\", \"endpoint\": \"$S3_ENDPOINT\", \"region\": \"us-east-1\", \"force_path_style\": true}}," fi -echo "== start (pool: $TEMPLATE none/small warm=$WARM${BRIDGE:+, egress via $BRIDGE}${S3_ENDPOINT:+, s3 store at $S3_ENDPOINT})" +VOLUME_LINE="" +if [[ -n $VOLUME_IMAGE ]]; then + VOLUME_CATALOG=$(jq -cn --arg name "$VOLUME_NAME" --arg path "$VOLUME_IMAGE" --arg directio "$VOLUME_DIRECTIO" \ + '[{name: $name, path: $path, directio: $directio}]') + VOLUME_LINE="\"volumes\": $VOLUME_CATALOG," +fi + +echo "== start (pool: $TEMPLATE none/small warm=$WARM${BRIDGE:+, egress via $BRIDGE}${S3_ENDPOINT:+, s3 store at $S3_ENDPOINT}${VOLUME_IMAGE:+, volume $VOLUME_NAME})" cat >"$DATA/config.json" <"$DATA/config.json" < $warm_after" + "$DATA/demo" -addr "$ADDR" -token "$TOKEN" -template "$TEMPLATE" -n 2 +else + "$DATA/demo" -addr "$ADDR" -token "$TOKEN" -template "$TEMPLATE" -n 3 +fi echo "== v2 smoke: files/session/find/replace/watch/git/pty through the relay" # LSP_TEMPLATE (optional, a python-flavor ref) adds the LSP broker step. "$DATA/smoke" -addr "$ADDR" -token "$TOKEN" -template "$TEMPLATE" ${BRIDGE:+-egress} ${LSP_TEMPLATE:+-lsp-template "$LSP_TEMPLATE"} +if [[ -n $VOLUME_IMAGE ]]; then + echo "== volumes: two warm read-only claims, shared bytes, and EROFS" + for i in $(seq 1 30); do + if api info | jq -e 'all(.pools[]; .warm >= .target)' >/dev/null 2>&1; then + break + fi + [[ $i == 30 ]] && { echo "pool did not refill before volume smoke"; api info | jq . || true; exit 1; } + sleep 1 + done + warm_before=$(warm_claims) + "$DATA/volumesmoke" -addr "$ADDR" -token "$TOKEN" -template "$TEMPLATE" \ + -volume "$VOLUME_NAME" -probe "$VOLUME_PROBE" + warm_after=$(warm_claims) + [[ $warm_after -ge $((warm_before + 2)) ]] || { + echo "volume claims did not consume the two ready warm VMs: before=$warm_before after=$warm_after" + exit 1 + } + echo "volume warm claim counter: $warm_before -> $warm_after" + after_checksum=$(sha256sum -- "$VOLUME_IMAGE" | awk '{print $1}') + [[ $after_checksum == "$VOLUME_CHECKSUM" ]] || { + echo "volume backing image changed: before=$VOLUME_CHECKSUM after=$after_checksum" + exit 1 + } + echo "volume backing checksum unchanged: $after_checksum" +fi + echo "== reap: leaked 5s-ttl claim is destroyed by the owner" "$DATA/demo" -addr "$ADDR" -token "$TOKEN" -template "$TEMPLATE" -n 1 -ttl 5 -leak sleep 12 diff --git a/sdk/go/checkpoint.go b/sdk/go/checkpoint.go index bca4c039..1751674f 100644 --- a/sdk/go/checkpoint.go +++ b/sdk/go/checkpoint.go @@ -3,6 +3,7 @@ package sandbox import ( "bytes" "context" + "errors" "net/http" "time" ) @@ -31,10 +32,13 @@ func (ck *Checkpoint) New(ctx context.Context, opts ...Option) (*Sandbox, error) for _, opt := range opts { opt(&claim) } + if len(claim.Volumes) > 0 { + return nil, errors.New("checkpoint claims do not accept WithVolumes") + } if err := claim.rejectPinnedAxes(); err != nil { return nil, err } - addr, cr, err := claimFollow(ck.addr, "claim checkpoint", func(noRedirect bool) ([]byte, error) { + addr, cr, err := claimFollow(ck.addr, "claim checkpoint", func(noRedirect, _ bool) ([]byte, error) { return encodeBody("checkpoint claim", checkpointClaimRequest{TTLSeconds: claim.TTLSeconds, NoRedirect: noRedirect}) }, func(a string, body []byte) (claimResponse, error) { return ck.claimAt(ctx, a, body) diff --git a/sdk/go/client.go b/sdk/go/client.go index bdf2dd96..104c091e 100644 --- a/sdk/go/client.go +++ b/sdk/go/client.go @@ -52,8 +52,8 @@ func (c *Client) New(ctx context.Context, template string, opts ...Option) (*San for _, opt := range opts { opt(&claim) } - addr, cr, err := claimFollow(c.addr, "claim", func(noRedirect bool) ([]byte, error) { - claim.NoRedirect = noRedirect + addr, cr, err := claimFollow(c.addr, "claim", func(noRedirect, requirePromoted bool) ([]byte, error) { + claim.NoRedirect, claim.RequirePromoted = noRedirect, requirePromoted return encodeBody("claim", claim) }, func(a string, body []byte) (claimResponse, error) { return c.claimAt(ctx, a, body) @@ -64,6 +64,15 @@ func (c *Client) New(ctx context.Context, template string, opts ...Option) (*San return c.handleFrom(addr, cr), nil } +// Volumes lists the caller-visible fleet catalog; availability is local. +func (c *Client) Volumes(ctx context.Context) ([]VolumeInfo, error) { + resp, err := doJSON[volumeListResponse](ctx, c, http.MethodGet, c.addr, "/v1/volumes", nil, c.apiToken, "list volumes") + if err != nil { + return nil, err + } + return resp.Volumes, nil +} + // Lookup relocates a sandbox handle whose owner address was lost, given its // id and token: it asks the entry node, then scatters across the cluster's // peers concurrently, and returns a handle bound to whichever node confirms @@ -132,7 +141,8 @@ func (c *Client) ownerAt(ctx context.Context, addr, id, token string) (string, e // node that answered when a single-node deployment omits owner_addr. func (c *Client) handleFrom(dialed string, cr claimResponse) *Sandbox { return &Sandbox{ - ID: cr.ID, Deadline: cr.Deadline, FromCheckpoint: cr.FromCheckpoint, TemplateDigest: cr.TemplateDigest, + ID: cr.ID, Deadline: cr.Deadline, Volumes: cr.Volumes, + FromCheckpoint: cr.FromCheckpoint, TemplateDigest: cr.TemplateDigest, c: c, token: cr.Token, owner: cmp.Or(cr.OwnerAddr, dialed), } } @@ -289,8 +299,8 @@ func retryTransient(err error) bool { // claimFollow runs the claim protocol from origin: claim there, and on a // redirect re-encode with no_redirect and follow via redirectFallback. Only // the fallback error carries the verb — first-contact errors return raw. -func claimFollow(origin, verb string, encode func(noRedirect bool) ([]byte, error), claimAt func(addr string, body []byte) (claimResponse, error)) (string, claimResponse, error) { - body, err := encode(false) +func claimFollow(origin, verb string, encode func(noRedirect, requirePromoted bool) ([]byte, error), claimAt func(addr string, body []byte) (claimResponse, error)) (string, claimResponse, error) { + body, err := encode(false, false) if err != nil { return "", claimResponse{}, err } @@ -301,7 +311,7 @@ func claimFollow(origin, verb string, encode func(noRedirect bool) ([]byte, erro if len(cr.Redirect) == 0 { return origin, cr, nil } - if body, err = encode(true); err != nil { + if body, err = encode(true, cr.RequirePromoted); err != nil { return "", claimResponse{}, err } addr, target, err := redirectFallback(origin, cr.Redirect, func(a string) (claimResponse, error) { @@ -413,11 +423,13 @@ func apiError(verb string, resp *http.Response) error { // claimRequest mirrors sandboxd's wire type; duplicated so the SDK stays // dependency-free — the e2e module guards against drift. type claimRequest struct { - Template string `json:"template"` - Net string `json:"net,omitempty"` - Size string `json:"size,omitempty"` - TTLSeconds int `json:"ttl_seconds,omitempty"` - NoRedirect bool `json:"no_redirect,omitempty"` + Template string `json:"template"` + Net string `json:"net,omitempty"` + Size string `json:"size,omitempty"` + Volumes []Volume `json:"volumes,omitempty"` + TTLSeconds int `json:"ttl_seconds,omitempty"` + NoRedirect bool `json:"no_redirect,omitempty"` + RequirePromoted bool `json:"require_promoted,omitempty"` } // rejectPinnedAxes fails a snapshot claim (checkpoint, template) that passed @@ -430,13 +442,19 @@ func (r claimRequest) rejectPinnedAxes() error { } type claimResponse struct { - ID string `json:"id"` - Token string `json:"token"` - Deadline time.Time `json:"deadline"` - OwnerAddr string `json:"owner_addr,omitempty"` - FromCheckpoint string `json:"from_checkpoint,omitempty"` - TemplateDigest string `json:"template_digest,omitempty"` - Redirect []string `json:"redirect,omitempty"` + ID string `json:"id"` + Token string `json:"token"` + Deadline time.Time `json:"deadline"` + OwnerAddr string `json:"owner_addr,omitempty"` + FromCheckpoint string `json:"from_checkpoint,omitempty"` + TemplateDigest string `json:"template_digest,omitempty"` + Volumes []Volume `json:"volumes,omitempty"` + Redirect []string `json:"redirect,omitempty"` + RequirePromoted bool `json:"require_promoted,omitempty"` +} + +type volumeListResponse struct { + Volumes []VolumeInfo `json:"volumes"` } type forkRequest struct { diff --git a/sdk/go/client_test.go b/sdk/go/client_test.go index 5f2a648e..9ad60447 100644 --- a/sdk/go/client_test.go +++ b/sdk/go/client_test.go @@ -4,6 +4,7 @@ import ( "encoding/json" "net/http" "net/http/httptest" + "reflect" "strings" "testing" "time" @@ -58,7 +59,7 @@ func TestNewSendsClaim(t *testing.T) { t.Fatalf("New: %v", err) } want := claimRequest{Template: "python:3.12", Net: "egress", Size: "medium", TTLSeconds: 90} - if gotBody != want { + if !reflect.DeepEqual(gotBody, want) { t.Errorf("body %+v, want %+v", gotBody, want) } if gotAuth != "Bearer sekret" { diff --git a/sdk/go/files.go b/sdk/go/files.go index 9e013258..ba23eb00 100644 --- a/sdk/go/files.go +++ b/sdk/go/files.go @@ -3,17 +3,11 @@ package sandbox import ( "bytes" "context" - "fmt" - "io" "slices" "github.com/cocoonstack/sandbox/protocol/wire" - "github.com/cocoonstack/sandbox/sdk/go/silkd" ) -// fsChunk matches silkd's BULK_CHUNK. -const fsChunk = 256 * 1024 - // WriteFile writes data to path in the sandbox, atomically (silkd renames a // temp file into place). mode, when non-nil, sets the file's permission bits. func (s *Sandbox) WriteFile(ctx context.Context, path string, data []byte, mode *uint32) error { @@ -73,160 +67,3 @@ func (s *Sandbox) Remove(ctx context.Context, path string, recursive bool) error func (s *Sandbox) Rename(ctx context.Context, from, to string) error { return s.doneRPC(ctx, &wire.FsRename{From: from, To: to}) } - -// doneRPC sends a request that answers with Done or an error frame. -func (s *Sandbox) doneRPC(ctx context.Context, req wire.Request) error { - conn, done, err := s.call(ctx, req) - if err != nil { - return err - } - defer done() - return terminalErr(ctx, conn) -} - -// uploadRPC sends req, streams r as Data frames, and expects a terminal Done. -func (s *Sandbox) uploadRPC(ctx context.Context, req wire.Request, r io.Reader) error { - conn, done, err := s.call(ctx, req) - if err != nil { - return err - } - defer done() - if err := uploadStream(conn, r); err != nil { - return err - } - return terminalErr(ctx, conn) -} - -// downloadRPC sends req and drains its Data stream into sink until Done. -func (s *Sandbox) downloadRPC(ctx context.Context, req wire.Request, sink func([]byte) error) error { - conn, done, err := s.call(ctx, req) - if err != nil { - return err - } - defer done() - return drainData(ctx, conn, sink) -} - -// oneShotRPC sends req and returns its single typed reply frame. -func oneShotRPC[T any](ctx context.Context, s *Sandbox, req wire.Request) (*T, error) { - conn, done, err := s.call(ctx, req) - if err != nil { - return nil, err - } - defer done() - return expect[T](ctx, conn) -} - -// collectRPC sends req and gathers every streamed frame of type T until Done. -func collectRPC[T any](ctx context.Context, s *Sandbox, req wire.Request) ([]T, error) { - conn, done, err := s.call(ctx, req) - if err != nil { - return nil, err - } - defer done() - var out []T - for { - resp, err := recv(ctx, conn) - if err != nil { - return nil, err - } - // Via any: a direct resp.(*T) assertion is rejected at compile time - // (*T is not known to implement Response). - if v, ok := any(resp).(*T); ok { - out = append(out, *v) - continue - } - switch r := resp.(type) { - case *wire.Done: - return out, nil - case *wire.ErrorResp: - return nil, r - default: - return nil, unexpected(resp) - } - } -} - -// uploadStream chunks r into Data frames terminated by DataEnd; shared by the -// FsWrite payload and the FsPush tar stream. -func uploadStream(conn *silkd.Conn, r io.Reader) error { - buf := make([]byte, fsChunk) - for { - n, readErr := r.Read(buf) - if n > 0 { - if err := conn.Send(&wire.Data{Data: buf[:n]}); err != nil { - return err - } - } - if readErr == io.EOF { - return conn.Send(wire.DataEnd{}) - } - if readErr != nil { - return readErr - } - } -} - -// drainData consumes Data frames into sink until Done; an error frame or an -// unexpected frame is a Go error. -func drainData(ctx context.Context, conn *silkd.Conn, sink func([]byte) error) error { - for { - resp, err := recv(ctx, conn) - if err != nil { - return err - } - switch r := resp.(type) { - case *wire.DataResp: - if err := sink(r.Data); err != nil { - return err - } - case *wire.Done: - return nil - case *wire.ErrorResp: - return r - default: - return unexpected(resp) - } - } -} - -// terminalErr reads one frame and requires it to be Done (else the error). -func terminalErr(ctx context.Context, conn *silkd.Conn) error { - _, err := expect[wire.Done](ctx, conn) - return err -} - -// expect reads one frame and requires it to be a *T, mapping an error frame -// to a Go error. -func expect[T any](ctx context.Context, conn *silkd.Conn) (*T, error) { - resp, err := recv(ctx, conn) - if err != nil { - return nil, err - } - if v, ok := any(resp).(*T); ok { - return v, nil - } - if e, ok := resp.(*wire.ErrorResp); ok { - return nil, e - } - return nil, unexpected(resp) -} - -// recv reads one frame, translating a canceled ctx and an early EOF. -func recv(ctx context.Context, conn *silkd.Conn) (wire.Response, error) { - resp, err := conn.Recv() - if err != nil { - if ctxErr := ctx.Err(); ctxErr != nil { - return nil, ctxErr - } - if err == io.EOF { - return nil, fmt.Errorf("connection closed before a terminal frame") - } - return nil, err - } - return resp, nil -} - -func unexpected(resp wire.Response) error { - return fmt.Errorf("unexpected frame %q", resp.RespType()) -} diff --git a/sdk/go/options.go b/sdk/go/options.go index db3a0f7d..faeb49fa 100644 --- a/sdk/go/options.go +++ b/sdk/go/options.go @@ -1,6 +1,9 @@ package sandbox -import "time" +import ( + "slices" + "time" +) const ( // NetNone is the hardened default: no NIC at all, vsock-only I/O. @@ -21,6 +24,21 @@ type NetShape string // node's warm pools. type Size string +// Volume requests one catalog entry at an optional guest mount path. +type Volume struct { + Name string `json:"name"` + Mount string `json:"mount,omitempty"` +} + +// VolumeInfo describes one visible fleet catalog entry. +type VolumeInfo struct { + Name string `json:"name"` + DefaultMount string `json:"default_mount"` + SizeBytes int64 `json:"size_bytes"` + Available bool `json:"available"` + Nodes int `json:"nodes"` +} + // Option configures a New claim. type Option func(*claimRequest) @@ -34,6 +52,12 @@ func WithSize(s Size) Option { return func(r *claimRequest) { r.Size = string(s) } } +// WithVolumes requests read-only catalog volumes for a claim. +func WithVolumes(volumes ...Volume) Option { + volumes = slices.Clone(volumes) + return func(r *claimRequest) { r.Volumes = volumes } +} + // WithTimeout bounds the sandbox's lifetime: the owning node reaps it after // d (rounded up to seconds) even if the client vanishes. func WithTimeout(d time.Duration) Option { diff --git a/sdk/go/sandbox.go b/sdk/go/sandbox.go index c683a1c1..fcd23c67 100644 --- a/sdk/go/sandbox.go +++ b/sdk/go/sandbox.go @@ -55,6 +55,8 @@ func (e *ExitError) Error() string { type Sandbox struct { ID string Deadline time.Time + // Volumes lists the read-only volumes finalized on this claim. + Volumes []Volume // TemplateDigest is the content identity of the promoted-template export // this sandbox was cloned from; empty for any other source. TemplateDigest string @@ -168,8 +170,9 @@ func (s *Sandbox) Fork(ctx context.Context, count int, ttl time.Duration) ([]*Sa // clone from it, provision-on-demand. Re-promoting to the same name replaces // the template. Like Fork, a hibernated sandbox is promoted from its memory // image without waking. Templates are node-local — the returned handle is -// bound to the owning node, and its New/Delete always reach it (name-based -// Client calls only see the connected node's templates). +// bound to the owning node. Delete and New without volumes reach that node; +// New with volumes may follow one placement redirect (name-based Client calls +// only see the connected node's templates). func (s *Sandbox) Promote(ctx context.Context, template string) (*Template, error) { body, err := encodeBody("promote", promoteRequest{Token: s.token, Template: template}) if err != nil { diff --git a/sdk/go/silkd/silkdtest/fake.go b/sdk/go/silkd/silkdtest/fake.go index aaa99cf6..15c10965 100644 --- a/sdk/go/silkd/silkdtest/fake.go +++ b/sdk/go/silkd/silkdtest/fake.go @@ -41,11 +41,6 @@ func NewFake(root string) *Fake { return &Fake{Root: root, sessions: map[string]bool{}, branches: []string{"main"}, current: "main"} } -// Serve accepts connections until l closes, one RPC per connection. -func (f *Fake) Serve(l net.Listener) { - acceptLoop(l, f.ServeConn) -} - // ServeConn speaks one RPC on an already-open connection (e.g. after an HTTP // hijack) and closes it. func (f *Fake) ServeConn(conn net.Conn) { @@ -247,7 +242,7 @@ func (f *Fake) fsPull(conn net.Conn, path string) { return } _ = tw.Close() - send(conn, wire.DataResp{Data: buf.Bytes()}) + send(conn, &wire.DataResp{Data: buf.Bytes()}) send(conn, wire.Done{}) } @@ -457,7 +452,6 @@ func errFrame(conn net.Conn, kind, msg string) { send(conn, &wire.ErrorResp{Kind: kind, Message: msg}) } -// drainUpload reads Data frames until DataEnd, concatenating their payloads. func drainUpload(r *bufio.Reader) ([]byte, error) { var out []byte for { diff --git a/sdk/go/silkd/silkdtest/silkdtest.go b/sdk/go/silkd/silkdtest/silkdtest.go index b0e0df92..7132ceb2 100644 --- a/sdk/go/silkd/silkdtest/silkdtest.go +++ b/sdk/go/silkd/silkdtest/silkdtest.go @@ -36,27 +36,19 @@ func ListenHybrid(sockPath string, port int) (io.Closer, error) { if err != nil { return nil, err } - go func() { - for { - conn, err := l.Accept() - if err != nil { - return - } - go func(c net.Conn) { - r := bufio.NewReader(c) - line, err := r.ReadString('\n') - if err != nil || strings.TrimSpace(line) != fmt.Sprintf("CONNECT %d", port) { - _ = c.Close() - return - } - if _, err := fmt.Fprintf(c, "OK %d\n", port); err != nil { - _ = c.Close() - return - } - handle(c, r) - }(conn) + go acceptLoop(l, func(c net.Conn) { + r := bufio.NewReader(c) + line, err := r.ReadString('\n') + if err != nil || strings.TrimSpace(line) != fmt.Sprintf("CONNECT %d", port) { + _ = c.Close() + return + } + if _, err := fmt.Fprintf(c, "OK %d\n", port); err != nil { + _ = c.Close() + return } - }() + handle(c, r) + }) return l, nil } diff --git a/sdk/go/template.go b/sdk/go/template.go index f475f195..0d5f106d 100644 --- a/sdk/go/template.go +++ b/sdk/go/template.go @@ -5,9 +5,9 @@ import ( "net/url" ) -// Template is a promoted template bound to the node that holds it. Its New -// and Delete dial the owner directly — no gossip involved — so unlike the -// name-based Client calls they are usable the instant Promote returns. +// Template is a promoted template bound to the node that holds it. Delete and +// New without volumes dial the owner directly, so they are usable the instant +// Promote returns; a volume claim may follow one placement redirect. type Template struct { Name string ContentDigest string @@ -18,9 +18,9 @@ type Template struct { size string } -// New claims a sandbox cloned from the template, on the template's node. -// Options may set the TTL; the key axes (network lane, size) are the -// template's own and cannot be overridden. +// New claims a sandbox cloned from the template. Options may set the TTL or +// request volumes; the key axes (network lane, size) are the template's own +// and cannot be overridden. func (t *Template) New(ctx context.Context, opts ...Option) (*Sandbox, error) { claim := claimRequest{Template: t.Name} for _, opt := range opts { @@ -29,18 +29,29 @@ func (t *Template) New(ctx context.Context, opts ...Option) (*Sandbox, error) { if err := claim.rejectPinnedAxes(); err != nil { return nil, err } - // The template only exists under its exact key, and this node holds it: - // a redirect elsewhere could never find it. - claim.Net, claim.Size, claim.NoRedirect = t.net, t.size, true - body, err := encodeBody("claim", claim) - if err != nil { - return nil, err + claim.Net, claim.Size = t.net, t.size + if len(claim.Volumes) == 0 { + claim.NoRedirect = true + body, err := encodeBody("claim", claim) + if err != nil { + return nil, err + } + cr, err := t.c.claimAt(ctx, t.addr, body) + if err != nil { + return nil, err + } + return t.c.handleFrom(t.addr, cr), nil } - cr, err := t.c.claimAt(ctx, t.addr, body) + addr, cr, err := claimFollow(t.addr, "claim", func(noRedirect, requirePromoted bool) ([]byte, error) { + claim.NoRedirect, claim.RequirePromoted = noRedirect, requirePromoted + return encodeBody("claim", claim) + }, func(addr string, body []byte) (claimResponse, error) { + return t.c.claimAt(ctx, addr, body) + }) if err != nil { return nil, err } - return t.c.handleFrom(t.addr, cr), nil + return t.c.handleFrom(addr, cr), nil } // Delete removes the template from its node. The handle is owner-bound, so diff --git a/sdk/go/utils.go b/sdk/go/utils.go new file mode 100644 index 00000000..1fbd32d8 --- /dev/null +++ b/sdk/go/utils.go @@ -0,0 +1,170 @@ +package sandbox + +import ( + "context" + "fmt" + "io" + + "github.com/cocoonstack/sandbox/protocol/wire" + "github.com/cocoonstack/sandbox/sdk/go/silkd" +) + +// fsChunk matches silkd's BULK_CHUNK. +const fsChunk = 256 * 1024 + +// doneRPC sends a request that answers with Done or an error frame. +func (s *Sandbox) doneRPC(ctx context.Context, req wire.Request) error { + conn, done, err := s.call(ctx, req) + if err != nil { + return err + } + defer done() + return terminalErr(ctx, conn) +} + +// uploadRPC sends req, streams r as Data frames, and expects a terminal Done. +func (s *Sandbox) uploadRPC(ctx context.Context, req wire.Request, r io.Reader) error { + conn, done, err := s.call(ctx, req) + if err != nil { + return err + } + defer done() + if err := uploadStream(conn, r); err != nil { + return err + } + return terminalErr(ctx, conn) +} + +// downloadRPC sends req and drains its Data stream into sink until Done. +func (s *Sandbox) downloadRPC(ctx context.Context, req wire.Request, sink func([]byte) error) error { + conn, done, err := s.call(ctx, req) + if err != nil { + return err + } + defer done() + return drainData(ctx, conn, sink) +} + +// oneShotRPC sends req and returns its single typed reply frame. +func oneShotRPC[T any](ctx context.Context, s *Sandbox, req wire.Request) (*T, error) { + conn, done, err := s.call(ctx, req) + if err != nil { + return nil, err + } + defer done() + return expect[T](ctx, conn) +} + +// collectRPC sends req and gathers every streamed frame of type T until Done. +func collectRPC[T any](ctx context.Context, s *Sandbox, req wire.Request) ([]T, error) { + conn, done, err := s.call(ctx, req) + if err != nil { + return nil, err + } + defer done() + var out []T + for { + resp, err := recv(ctx, conn) + if err != nil { + return nil, err + } + // Via any: a direct resp.(*T) assertion is rejected at compile time + // (*T is not known to implement Response). + if v, ok := any(resp).(*T); ok { + out = append(out, *v) + continue + } + switch r := resp.(type) { + case *wire.Done: + return out, nil + case *wire.ErrorResp: + return nil, r + default: + return nil, unexpected(resp) + } + } +} + +// uploadStream chunks r into Data frames terminated by DataEnd; shared by the +// FsWrite payload and the FsPush tar stream. +func uploadStream(conn *silkd.Conn, r io.Reader) error { + buf := make([]byte, fsChunk) + for { + n, readErr := r.Read(buf) + if n > 0 { + if err := conn.Send(&wire.Data{Data: buf[:n]}); err != nil { + return err + } + } + if readErr == io.EOF { + return conn.Send(wire.DataEnd{}) + } + if readErr != nil { + return readErr + } + } +} + +// drainData consumes Data frames into sink until Done; an error frame or an +// unexpected frame is a Go error. +func drainData(ctx context.Context, conn *silkd.Conn, sink func([]byte) error) error { + for { + resp, err := recv(ctx, conn) + if err != nil { + return err + } + switch r := resp.(type) { + case *wire.DataResp: + if err := sink(r.Data); err != nil { + return err + } + case *wire.Done: + return nil + case *wire.ErrorResp: + return r + default: + return unexpected(resp) + } + } +} + +// terminalErr reads one frame and requires it to be Done (else the error). +func terminalErr(ctx context.Context, conn *silkd.Conn) error { + _, err := expect[wire.Done](ctx, conn) + return err +} + +// expect reads one frame and requires it to be a *T, mapping an error frame +// to a Go error. +func expect[T any](ctx context.Context, conn *silkd.Conn) (*T, error) { + resp, err := recv(ctx, conn) + if err != nil { + return nil, err + } + if v, ok := any(resp).(*T); ok { + return v, nil + } + if e, ok := resp.(*wire.ErrorResp); ok { + return nil, e + } + return nil, unexpected(resp) +} + +// recv reads one frame, translating a canceled ctx and an early EOF. +func recv(ctx context.Context, conn *silkd.Conn) (wire.Response, error) { + resp, err := conn.Recv() + if err != nil { + if ctxErr := ctx.Err(); ctxErr != nil { + return nil, ctxErr + } + if err == io.EOF { + return nil, fmt.Errorf("connection closed before a terminal frame") + } + return nil, err + } + return resp, nil +} + +func unexpected(resp wire.Response) error { + return fmt.Errorf("unexpected frame %q", resp.RespType()) +} diff --git a/sdk/go/volumes_test.go b/sdk/go/volumes_test.go new file mode 100644 index 00000000..7d3f8dfc --- /dev/null +++ b/sdk/go/volumes_test.go @@ -0,0 +1,172 @@ +package sandbox + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "slices" + "strings" + "testing" +) + +func TestClientNewSendsVolumes(t *testing.T) { + var got struct { + Template string `json:"template"` + Volumes []Volume `json:"volumes"` + } + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := json.NewDecoder(r.Body).Decode(&got); err != nil { + t.Errorf("decode body: %v", err) + } + _ = json.NewEncoder(w).Encode(claimResponse{ + ID: "sb_1", Token: "tok", Volumes: []Volume{{Name: "weights-llama", Mount: "/models"}}, + }) + })) + t.Cleanup(ts.Close) + + volumes := []Volume{{Name: "imagenet"}, {Name: "weights-llama", Mount: "/models"}} + withVolumes := WithVolumes(volumes...) + volumes[0].Name = "changed" + sb, err := testClient(t, ts).New(t.Context(), "rt:24.04", withVolumes) + if err != nil { + t.Fatalf("New: %v", err) + } + if got.Template != "rt:24.04" { + t.Errorf("template %q, want rt:24.04", got.Template) + } + if want := []Volume{{Name: "imagenet"}, {Name: "weights-llama", Mount: "/models"}}; !slices.Equal(got.Volumes, want) { + t.Errorf("volumes %v, want %v", got.Volumes, want) + } + if want := []Volume{{Name: "weights-llama", Mount: "/models"}}; !slices.Equal(sb.Volumes, want) { + t.Errorf("response volumes %v, want %v", sb.Volumes, want) + } +} + +func TestTemplateNewSendsVolumes(t *testing.T) { + var got struct { + Template string `json:"template"` + Net string `json:"net"` + Size string `json:"size"` + Volumes []Volume `json:"volumes"` + NoRedirect bool `json:"no_redirect"` + } + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := json.NewDecoder(r.Body).Decode(&got); err != nil { + t.Errorf("decode body: %v", err) + } + _ = json.NewEncoder(w).Encode(claimResponse{ + ID: "sb_2", Token: "tok", Volumes: []Volume{{Name: "imagenet", Mount: "/datasets/imagenet"}}, + }) + })) + t.Cleanup(ts.Close) + + c := testClient(t, ts) + tpl := &Template{Name: "task:v1", c: c, addr: c.addr, net: "none", size: "small"} + wantVolumes := []Volume{{Name: "imagenet", Mount: "/datasets/imagenet"}} + sb, err := tpl.New(t.Context(), WithVolumes(wantVolumes...)) + if err != nil { + t.Fatalf("New: %v", err) + } + if got.Template != "task:v1" || got.Net != "none" || got.Size != "small" || got.NoRedirect { + t.Errorf("claim %+v, want redirectable task:v1/none/small", got) + } + if !slices.Equal(got.Volumes, wantVolumes) { + t.Errorf("volumes %v, want %v", got.Volumes, wantVolumes) + } + if !slices.Equal(sb.Volumes, wantVolumes) { + t.Errorf("response volumes %v, want %v", sb.Volumes, wantVolumes) + } +} + +func TestTemplateNewVolumeClaimFollowsRedirect(t *testing.T) { + want := []Volume{{Name: "imagenet", Mount: "/datasets/imagenet"}} + var got claimRequest + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := json.NewDecoder(r.Body).Decode(&got); err != nil { + t.Errorf("decode body: %v", err) + } + _ = json.NewEncoder(w).Encode(claimResponse{ID: "sb_3", Token: "tok", Volumes: want}) + })) + t.Cleanup(target.Close) + entry := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(claimResponse{ + Redirect: []string{strings.TrimPrefix(target.URL, "http://")}, RequirePromoted: true, + }) + })) + t.Cleanup(entry.Close) + + c := testClient(t, entry) + tpl := &Template{Name: "task:v1", c: c, addr: c.addr, net: "none", size: "small"} + sb, err := tpl.New(t.Context(), WithVolumes(want...)) + if err != nil { + t.Fatalf("New: %v", err) + } + if !got.NoRedirect || !got.RequirePromoted || !slices.Equal(got.Volumes, want) { + t.Errorf("redirected claim = %+v, want promoted no_redirect with %v", got, want) + } + if !slices.Equal(sb.Volumes, want) { + t.Errorf("response volumes = %+v, want %+v", sb.Volumes, want) + } +} + +func TestClientVolumes(t *testing.T) { + want := []VolumeInfo{{Name: "imagenet", DefaultMount: "/volumes/imagenet", SizeBytes: 42, Available: true, Nodes: 3}} + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet || r.URL.Path != "/v1/volumes" { + t.Errorf("request = %s %s", r.Method, r.URL.Path) + } + if got := r.Header.Get("Authorization"); got != "Bearer sekret" { + t.Errorf("authorization = %q", got) + } + _ = json.NewEncoder(w).Encode(volumeListResponse{Volumes: want}) + })) + t.Cleanup(ts.Close) + + got, err := testClient(t, ts, WithAPIToken("sekret")).Volumes(t.Context()) + if err != nil { + t.Fatalf("Volumes: %v", err) + } + if !slices.Equal(got, want) { + t.Errorf("volumes = %+v, want %+v", got, want) + } +} + +func TestVolumeClaimRedirectPreservesEntries(t *testing.T) { + want := []Volume{{Name: "imagenet", Mount: "/datasets/imagenet"}} + var got claimRequest + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := json.NewDecoder(r.Body).Decode(&got); err != nil { + t.Errorf("decode body: %v", err) + } + _ = json.NewEncoder(w).Encode(claimResponse{ID: "sb_3", Token: "tok", Volumes: want}) + })) + t.Cleanup(target.Close) + entry := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(claimResponse{ + Redirect: []string{strings.TrimPrefix(target.URL, "http://")}, + }) + })) + t.Cleanup(entry.Close) + + sb, err := testClient(t, entry).New(t.Context(), "rt:24.04", WithVolumes(want...)) + if err != nil { + t.Fatalf("New: %v", err) + } + if !got.NoRedirect || !slices.Equal(got.Volumes, want) { + t.Errorf("redirected claim = %+v, want no_redirect with %v", got, want) + } + if !slices.Equal(sb.Volumes, want) { + t.Errorf("response volumes = %+v, want %+v", sb.Volumes, want) + } +} + +func TestCheckpointNewRejectsVolumesLocally(t *testing.T) { + ck := &Checkpoint{} + sb, err := ck.New(t.Context(), WithVolumes(Volume{Name: "imagenet"})) + if err == nil || !strings.Contains(err.Error(), "do not accept WithVolumes") { + t.Errorf("err %v, want local WithVolumes rejection", err) + } + if sb != nil { + t.Errorf("sandbox %+v, want nil", sb) + } +} diff --git a/sdk/python/cocoonsandbox/client.py b/sdk/python/cocoonsandbox/client.py index 2c3701f2..683b97dd 100644 --- a/sdk/python/cocoonsandbox/client.py +++ b/sdk/python/cocoonsandbox/client.py @@ -11,6 +11,7 @@ import urllib.error import urllib.parse import urllib.request +from collections.abc import Mapping from .checkpoint import Checkpoint from .errors import APIError @@ -27,23 +28,14 @@ def __init__(self, addr: str, api_token: str = "", timeout: float = 120.0): self.api_token = api_token self.timeout = timeout - def new(self, template: str, net: str = "", size: str = "", ttl_seconds: int = 0) -> Sandbox: + def new(self, template: str, net: str = "", size: str = "", ttl_seconds: int = 0, + volumes: list[str | Mapping[str, str]] | None = None) -> Sandbox: """Claims a sandbox; a warm hit is milliseconds. On a cluster a warm miss may redirect to a peer, followed transparently; if every candidate fails transiently, the claim falls back to the origin once so it provisions or heals locally.""" - claim = _claim_body(template, net, size, ttl_seconds) - reply = self._post_json(self.addr, "/v1/claim", claim, "claim") - redirect = reply.get("redirect") or [] - if not redirect: - return self._handle_from(self.addr, reply) - claim["no_redirect"] = True - - def post(peer): - return self._post_json(peer, "/v1/claim", claim, "claim") - - addr, reply = _redirect_fallback(self.addr, redirect, post, "claim") - return self._handle_from(addr, reply) + claim = _claim_body(template, net, size, ttl_seconds, volumes) + return self._claim_from(self.addr, claim) def delete_template(self, template: str, net: str = "", size: str = "") -> None: """Removes a promoted template by name; on a cluster the delete @@ -85,10 +77,30 @@ def checkpoints(self) -> list[Checkpoint]: reply = self._request(self.addr, "GET", "/v1/checkpoints", None, "list checkpoints") return [Checkpoint(self, self.addr, rec) for rec in reply.get("checkpoints") or []] + def volumes(self) -> list[dict]: + """Lists the caller-visible fleet catalog; availability is local.""" + reply = self._request(self.addr, "GET", "/v1/volumes", None, "list volumes") + return [dict(volume) for volume in reply.get("volumes") or []] + def info(self) -> dict: """The node's pool/claim counters, as served by GET /v1/info.""" return self._request(self.addr, "GET", "/v1/info", None, "info") + def _claim_from(self, addr: str, claim: dict) -> Sandbox: + reply = self._post_json(addr, "/v1/claim", claim, "claim") + redirect = reply.get("redirect") or [] + if not redirect: + return self._handle_from(addr, reply) + claim["no_redirect"] = True + if reply.get("require_promoted"): + claim["require_promoted"] = True + + def post(peer): + return self._post_json(peer, "/v1/claim", claim, "claim") + + owner, reply = _redirect_fallback(addr, redirect, post, "claim") + return self._handle_from(owner, reply) + def _peers(self) -> list: # /v1/peers is tenant-accessible (cluster topology); /v1/info is # operator-only, so a tenant lookup cannot read peers from it. A @@ -110,6 +122,7 @@ def _handle_from(self, dialed: str, reply: dict) -> Sandbox: deadline=reply.get("deadline", ""), from_checkpoint=reply.get("from_checkpoint", ""), template_digest=reply.get("template_digest", ""), + volumes=reply.get("volumes") or [], ) def _post_json(self, addr: str, path: str, body: dict, verb: str) -> dict: @@ -149,7 +162,8 @@ def _request(self, addr: str, method: str, path: str, body, verb: str, bearer: s raise APIError(verb, 0, "malformed JSON in response") from exc -def _claim_body(template: str, net: str, size: str, ttl_seconds: int) -> dict: +def _claim_body(template: str, net: str, size: str, ttl_seconds: int, + volumes: list[str | Mapping[str, str]] | None = None) -> dict: claim = {"template": template} if net: claim["net"] = net @@ -157,9 +171,21 @@ def _claim_body(template: str, net: str, size: str, ttl_seconds: int) -> dict: claim["size"] = size if ttl_seconds: claim["ttl_seconds"] = ttl_seconds + if volumes: + claim["volumes"] = [_volume_body(volume) for volume in volumes] return claim +def _volume_body(volume: str | Mapping[str, str]) -> dict: + if isinstance(volume, str): + return {"name": volume} + if not isinstance(volume, Mapping): + raise TypeError("volume must be a name string or mapping") + if set(volume) - {"name", "mount"}: + raise TypeError("volume mapping accepts only name and mount") + return dict(volume) + + def _template_query(template: str, net: str, size: str) -> dict: query = {"template": template} if net: diff --git a/sdk/python/cocoonsandbox/conn.py b/sdk/python/cocoonsandbox/conn.py index 573268a9..e1db2dec 100644 --- a/sdk/python/cocoonsandbox/conn.py +++ b/sdk/python/cocoonsandbox/conn.py @@ -6,24 +6,31 @@ import contextlib import socket from collections.abc import Iterator +from typing import TypeVar from .errors import APIError, ProtocolError, SilkdError from .frames import MAX_FRAME, decode_response, encode_request +_CloseableT = TypeVar("_CloseableT", bound="_Closeable") -class Conn: - """A live frame stream to one sandbox's silkd, via the owner node.""" - def __init__(self, sock: socket.socket, reader): - self._sock = sock - self._reader = reader +class _Closeable: + """Context-manager mixin for handles whose exit is just close().""" - def __enter__(self) -> Conn: + def __enter__(self: _CloseableT) -> _CloseableT: return self def __exit__(self, *exc) -> None: self.close() + +class Conn(_Closeable): + """A live frame stream to one sandbox's silkd, via the owner node.""" + + def __init__(self, sock: socket.socket, reader): + self._sock = sock + self._reader = reader + def send(self, op: str, **fields) -> None: self._sock.sendall(encode_request(op, **fields)) diff --git a/sdk/python/cocoonsandbox/sandbox.py b/sdk/python/cocoonsandbox/sandbox.py index a76f5808..979bd8c2 100644 --- a/sdk/python/cocoonsandbox/sandbox.py +++ b/sdk/python/cocoonsandbox/sandbox.py @@ -12,7 +12,7 @@ from typing import TYPE_CHECKING from .checkpoint import Checkpoint -from .conn import Conn, dial_agent +from .conn import Conn, _Closeable, dial_agent from .errors import APIError, ExitError, ProtocolError from .frames import BULK_CHUNK, FS_CHUNK from .template import Template @@ -25,7 +25,8 @@ class Sandbox: """One claimed microVM.""" def __init__(self, client: Client, id: str, token: str, owner: str, - deadline: str = "", from_checkpoint: str = "", template_digest: str = ""): + deadline: str = "", from_checkpoint: str = "", template_digest: str = "", + volumes: list[dict] | None = None): self._client = client self.id = id self.token = token @@ -33,6 +34,7 @@ def __init__(self, client: Client, id: str, token: str, owner: str, self.deadline = deadline self.from_checkpoint = from_checkpoint self.template_digest = template_digest + self.volumes = [dict(volume) for volume in volumes or []] def __enter__(self) -> Sandbox: return self @@ -377,18 +379,12 @@ def close(self) -> None: self._sandbox._done_rpc("session_rm", id=self.id) -class Watcher: +class Watcher(_Closeable): """A live filesystem event stream; iterate for {kind, path} events.""" def __init__(self, conn: Conn): self._conn = conn - def __enter__(self) -> Watcher: - return self - - def __exit__(self, *exc) -> None: - self.close() - def __iter__(self) -> Iterator[dict]: # Connection-bound: a close, drop, or undecodable frame ends iteration; # a real server error frame (SilkdError) propagates. @@ -404,7 +400,7 @@ def close(self) -> None: self._conn.close() -class Pty: +class Pty(_Closeable): """An interactive shell under a guest pty; read/write are raw bytes.""" def __init__(self, sandbox: Sandbox, conn: Conn, pid: int): @@ -412,12 +408,6 @@ def __init__(self, sandbox: Sandbox, conn: Conn, pid: int): self._conn = conn self.pid = pid - def __enter__(self) -> Pty: - return self - - def __exit__(self, *exc) -> None: - self.close() - def read(self) -> bytes: """The next output chunk; b'' once the shell exits.""" frame = self._conn.recv() @@ -455,19 +445,13 @@ def stop(self) -> None: self._sandbox._done_rpc("lsp_stop", server_id=self.server_id) -class PortConn: +class PortConn(_Closeable): """A byte stream to a guest port, relayed over the silkd connection.""" def __init__(self, conn: Conn): self._conn = conn self._eof = False - def __enter__(self) -> PortConn: - return self - - def __exit__(self, *exc) -> None: - self.close() - def send(self, data: bytes) -> None: _send_chunks(self._conn, data, chunk=BULK_CHUNK) diff --git a/sdk/python/cocoonsandbox/template.py b/sdk/python/cocoonsandbox/template.py index 5aff4c6d..3158aef9 100644 --- a/sdk/python/cocoonsandbox/template.py +++ b/sdk/python/cocoonsandbox/template.py @@ -1,10 +1,9 @@ -"""Template: a promoted sandbox state, claimable by name. The handle is -bound to the node that holds it, so it works the instant promote returns — -name-based Client calls route via gossip and lag a promote by about a tick.""" +"""Promoted template handles and owner-aware claims.""" from __future__ import annotations import urllib.parse +from collections.abc import Mapping from typing import TYPE_CHECKING if TYPE_CHECKING: @@ -23,14 +22,15 @@ def __init__(self, client: Client, addr: str, name: str, net: str, size: str, co self.size = size self.content_digest = content_digest - def new(self, ttl_seconds: int = 0) -> Sandbox: - """Claims a sandbox cloned from the template, on the template's - node; the key axes are the template's own.""" + def new(self, ttl_seconds: int = 0, volumes: list[str | Mapping[str, str]] | None = None) -> Sandbox: + """Claims the template, following placement when volumes require it.""" # Local import: a top-level one would close the client → sandbox → # template cycle. from .client import _claim_body - claim = _claim_body(self.name, self.net, self.size, ttl_seconds) + claim = _claim_body(self.name, self.net, self.size, ttl_seconds, volumes) + if volumes: + return self._client._claim_from(self._addr, claim) claim["no_redirect"] = True reply = self._client._post_json(self._addr, "/v1/claim", claim, "claim") return self._client._handle_from(self._addr, reply) diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 0bf34fe8..71de5bcc 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -1,5 +1,4 @@ -"""Control-plane behavior against an in-process fake node: claim happy path, -redirect follow with no_redirect, API error mapping, checkpoint handles.""" +"""Control-plane claims, redirects, volume discovery, errors, and checkpoints.""" import json import threading @@ -7,7 +6,7 @@ import pytest -from cocoonsandbox import APIError, Client +from cocoonsandbox import APIError, Client, Template class FakeNode(BaseHTTPRequestHandler): @@ -62,6 +61,91 @@ def test_claim_happy_path(node): assert sb.template_digest == "sha256:task" +def test_claim_sends_volumes(node): + seen = [] + + def claim(body, path): + seen.append(body) + return 200, {"id": "sb_1", "token": "tok", "volumes": [ + {"name": "imagenet", "mount": "/volumes/imagenet"}, + {"name": "weights-llama", "mount": "/models"}, + ]} + + FakeNode.routes[("POST", "/v1/claim")] = claim + sb = Client(node).new("rt:24.04", volumes=[ + "imagenet", {"name": "weights-llama", "mount": "/models"}]) + assert seen == [{"template": "rt:24.04", "volumes": [ + {"name": "imagenet"}, + {"name": "weights-llama", "mount": "/models"}, + ]}] + assert sb.volumes == [ + {"name": "imagenet", "mount": "/volumes/imagenet"}, + {"name": "weights-llama", "mount": "/models"}, + ] + + +def test_claim_rejects_legacy_volume_tuple(node): + with pytest.raises(TypeError, match="name string or mapping"): + Client(node).new("rt:24.04", volumes=[("imagenet", "/datasets/imagenet")]) + + +def test_claim_rejects_volume_mode(node): + with pytest.raises(TypeError, match="only name and mount"): + Client(node).new("rt:24.04", volumes=[{"name": "imagenet", "mode": "rw"}]) + + +def test_template_claim_sends_volumes(node): + seen = [] + + def claim(body, path): + seen.append(body) + return 200, {"id": "sb_2", "token": "tok", "volumes": [ + {"name": "imagenet", "mount": "/datasets/imagenet"}, + ]} + + FakeNode.routes[("POST", "/v1/claim")] = claim + sb = Template(Client(node), node, "task:v1", "none", "small").new( + volumes=[{"name": "imagenet", "mount": "/datasets/imagenet"}]) + assert seen == [{ + "template": "task:v1", + "net": "none", + "size": "small", + "volumes": [{"name": "imagenet", "mount": "/datasets/imagenet"}], + }] + assert sb.volumes == [{"name": "imagenet", "mount": "/datasets/imagenet"}] + + +def test_template_volume_claim_follows_redirect(node): + seen = [] + + def claim(body, path): + seen.append(body) + if len(seen) == 1: + return 200, {"redirect": [node], "require_promoted": True} + return 200, {"id": "sb_2", "token": "tok", "volumes": body["volumes"]} + + FakeNode.routes[("POST", "/v1/claim")] = claim + sb = Template(Client(node), node, "task:v1", "none", "small").new( + volumes=[{"name": "imagenet", "mount": "/datasets/imagenet"}]) + assert "no_redirect" not in seen[0] + assert seen[1]["no_redirect"] is True + assert seen[1]["require_promoted"] is True + assert seen[1]["volumes"] == [{"name": "imagenet", "mount": "/datasets/imagenet"}] + assert sb.volumes == seen[1]["volumes"] + + +def test_volume_catalog(node): + want = [{ + "name": "imagenet", + "default_mount": "/volumes/imagenet", + "size_bytes": 42, + "available": True, + "nodes": 3, + }] + FakeNode.routes[("GET", "/v1/volumes")] = lambda body, path: (200, {"volumes": want}) + assert Client(node).volumes() == want + + def test_promote_returns_content_digest(node): FakeNode.routes[("POST", "/v1/claim")] = lambda body, path: ( 200, {"id": "sb_1", "token": "tok", "owner_addr": node}) @@ -86,10 +170,14 @@ def claim(body, path): return 200, {"id": "sb_2", "token": "tok"} FakeNode.routes[("POST", "/v1/claim")] = claim - sb = Client(node).new("rt:24.04") + volumes = [{"name": "imagenet", "mount": "/datasets/imagenet"}] + sb = Client(node).new("rt:24.04", volumes=volumes) assert sb.id == "sb_2" assert "no_redirect" not in seen[0] assert seen[1]["no_redirect"] is True + assert seen[0]["volumes"] == seen[1]["volumes"] == [ + {"name": "imagenet", "mount": "/datasets/imagenet"}, + ] def test_api_error_carries_server_message(node): diff --git a/sdk/python/tests/test_hardening.py b/sdk/python/tests/test_hardening.py index eb7a898b..985cd048 100644 --- a/sdk/python/tests/test_hardening.py +++ b/sdk/python/tests/test_hardening.py @@ -6,10 +6,8 @@ import pytest -from cocoonsandbox import APIError, Client +from cocoonsandbox import APIError, Client, ProtocolError, SilkdError, Watcher from cocoonsandbox.conn import Conn, dial_agent -from cocoonsandbox.errors import ProtocolError, SilkdError -from cocoonsandbox.sandbox import Watcher def test_dial_agent_rejects_control_chars_in_identity(): diff --git a/sdk/python/tests/test_wire_binding.py b/sdk/python/tests/test_wire_binding.py index f08ca9d5..24c2cc39 100644 --- a/sdk/python/tests/test_wire_binding.py +++ b/sdk/python/tests/test_wire_binding.py @@ -12,7 +12,7 @@ import pytest -from cocoonsandbox import Client +from cocoonsandbox import Client, Lsp, Pty, Sandbox, Session from cocoonsandbox.conn import Conn from cocoonsandbox.frames import PROTO_VERSION @@ -94,28 +94,20 @@ def _pty_stub(sb, pid): - from cocoonsandbox.sandbox import Pty - return Pty(sb, None, pid) def _session_stub(sb, id): - from cocoonsandbox.sandbox import Session - return Session(sb, id) def _lsp_stub(sb, server_id): - from cocoonsandbox.sandbox import Lsp - return Lsp(sb, server_id) def _fake_sandbox(monkeypatch, replies): """A Sandbox whose _dial yields a real Conn over a socketpair; a guest thread records inbound frames and answers the scripted replies.""" - from cocoonsandbox.sandbox import Sandbox - sent = [] client_sock, guest_sock = socket.socketpair() diff --git a/silkd/src/main.rs b/silkd/src/main.rs index e89d75ae..125320d2 100644 --- a/silkd/src/main.rs +++ b/silkd/src/main.rs @@ -1,10 +1,7 @@ //! silkd: the in-guest sandbox daemon. Listens on a hybrid-vsock port for //! newline-JSON RPC frames from the host (relayed by sandboxd) and runs //! commands with context, tracks processes, moves files, and holds sessions. -//! -//! Verbs: exec, info, ps, kill, attach, logs, session.{create,list,rm}, -//! fs.{write,read,list,stat,mkdir,rm,rename,push,pull,find,replace,watch}, -//! pty.{open,resize}, git.{clone,status,add,commit,push,pull,branch}. +//! `proto::Request` is the authoritative verb list. use std::sync::Arc; diff --git a/silkd/src/watch.rs b/silkd/src/watch.rs index f9fcc65d..353126d3 100644 --- a/silkd/src/watch.rs +++ b/silkd/src/watch.rs @@ -52,8 +52,8 @@ where frame = rx.recv() => match frame { Some(frame) => { let terminal = matches!(frame, Response::Error { .. }); - proto::write_frame(w, &frame).await?; - if terminal { + // A failed write is the disconnect the EOF arm can lose the select to. + if proto::write_frame(w, &frame).await.is_err() || terminal { return Ok(()); } }