Skip to content

pool: pre-attached read-only dataset volumes on warm pools #75

Description

@CMGS

The volume attach+mount path costs ~40ms at claim time (measured settled on
bare metal: ro-volume claim 37.6ms vs volume-less 0.5ms). For datasets that
are hot on a node, that cost can move to refill time: the warm pool carries
VMs with the dataset already attached and mounted, and a matching claim pays
nothing — the same shape that made volume-less warm claims 0.3ms.

Scoped to READ-ONLY volumes only. A writable image is single-writer, so a
pool of pre-attached writable VMs is a contradiction (one VM could hold it;
the rest would fail refill against the hypervisor lock).

Design questions to settle

  1. Placement shape. A pool-spec volumes list, so the pool key gains a
    volumes axis (separate warm pools per volume set), vs a post-key filter on
    one pool. The axis is the honest shape — a volume-carrying VM and a bare
    VM are not interchangeable (see 2) — but it multiplies pools; config
    should probably declare it per-pool rather than per-combination.
  2. Isolation invariant: a pre-attached VM must never satisfy a claim that
    did not name the volume.
    A volume-less claim handed a VM with a mounted
    dataset reads data it never requested — with tenant ACLs on the catalog
    this is an access-control hole, not just surprise. The claim→pool match
    must be exact on the volume set, both directions.
  3. Tenancy. The catalog ACL gates claims per tenant, but a pool is shared
    config. A pre-attached pool whose volume has a tenant ACL either inherits
    the restriction (pool only claimable by those tenants — new coupling) or
    must be rejected at config load. Fail-closed at load looks right.
  4. Refill semantics. Attach+mount joins the refill pipeline (after probe,
    before ready). Refill failure on the volume step = destroy and retry with
    the existing backoff — but a missing/deleted image makes the pool
    permanently unfillable; needs the watermark/error surfacing story.
  5. Catalog interplay. Entry removed or re-pathed while a pool pre-attaches
    it: config reload must drain or refuse. Dirty marker on a ro pre-attach:
    refill blocks on needs-recovery like a claim would.
  6. Mesh. Warm candidates already intersect volume names (VolumeCandidates)
    for catalog-held names; pre-attached pools advertise the stronger fact.
    Probably reuses NodeState.Volumes unchanged (the name is available either
    way) with the warm count doing the rest.
  7. Capture/lifecycle. Everything shipped in Writable catalog volumes with a guest filesystem shutdown lifecycle #72 stays: pre-attached claims
    refuse capture, the idle sweep skips them, release quiesce is a no-op for
    ro. Hibernate of a pre-attached WARM (unclaimed) VM: refused today by
    cocoon (hot-attached disk) — pool hibernation of such pools must be
    disabled or detach-first; needs a decision.

Relation

  • Writable catalog volumes with a guest filesystem shutdown lifecycle #72 shipped claim-time attach for both modes; this issue moves the ro cost
    to refill for declared-hot datasets.
  • A parallel-attach optimization (separate PR) reduces the multi-volume
    claim-time cost independently; both can coexist.
  • The lazy/async opt-in (separate PR) serves the claim-then-think pattern;
    pre-attach serves claim-then-read.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions