Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -851,7 +851,9 @@ schema's source of truth.
Human and JSON status include derived warm counts globally and per platform.
`ready` devices contribute to those counts; `reclaiming` and `quarantined`
devices remain visible as busy running capacity and never contribute to warm
inventory. A `quarantined` device is one whose release-time purge failed, or
inventory. A `quarantined` device is one whose release-time purge failed, one
created under `lease.identity` `fresh` whose lease-end delete failed (it is
retried as a delete, never returned to the pool), or one
whose `provisioning`/`reclaiming` transition stalled past its driver-derived
threshold (see `simlock doctor` below): it stays visible in `status` and
`list --devices` with that state while `QuarantineCoordinator` retries it in
Expand Down
8 changes: 7 additions & 1 deletion docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,14 @@ a warning. Inspect the effective, merged configuration at any time with
| `capacity.strategy` | Which policy decides how many devices may exist and run at once: `resource` or `fixed`. The options under `capacity.config` are that strategy's own -- see [Capacity strategies](#capacity-strategies). | `resource` |
| `idle.shutdownAfterMs` | How long an unused device sits idle before Simlock shuts it down (tier 1, reclaims RAM). | `10 minutes` |
| `idle.deleteAfterMs` | How long a shut-down device sits idle before Simlock deletes it (tier 2, reclaims disk). | `1 hour` |
| `warmPool.quarantine.maxRetries` | Failed purge retries allowed on a quarantined device (after the triggering failure) before Simlock gives up and destroys it. | `3` |
| `warmPool.quarantine.maxRetries` | Failed purge retries allowed on a quarantined device (after the triggering failure) before Simlock gives up and destroys it. A device created under `lease.identity` `fresh` retries its delete instead of a purge. | `3` |
| `warmPool.quarantine.retryBackoffMs` | Delay before the first quarantine purge retry. | `30 seconds` |
| `warmPool.quarantine.retryBackoffMultiplier` | Growth factor applied to the backoff after each failed retry. | `2` |
| `warmPool.quarantine.maxRetryBackoffMs` | Cap on the quarantine retry backoff. | `5 minutes` |
| `lease.defaultTtlMs` | TTL applied to a lease whose `lease.request` carried no `ttlMs` — **that request only**. It is *not* the renew fallback: a renew given no explicit TTL re-applies the lease's own stored width, so a lease granted for longer keeps it. A lease not renewed before its deadline expires and its device is reclaimed. | `15 minutes` |
| `lease.maxTtlMs` | Largest TTL a request or a renew may ask for. A larger `ttlMs` is rejected with `BAD_REQUEST` rather than silently clamped, so a caller is never left believing it has more time than it does. | `4 hours` |
| `lease.identity.ios` | Whether iOS leases reuse simulators. `reusable` erases a released simulator and returns it to the pool. `fresh` creates a simulator for each lease and deletes it when the lease ends — by release, by expiry, or because the device was lost — so no simulator serves two leases. If the delete fails, the simulator is quarantined, never leased again, and the delete is retried. **Boot cost:** under `fresh`, no simulator stays warm after a release, so every lease pays a full boot (about 30 seconds on an Apple silicon Mac). Each device keeps the setting it was created with, even if you change this value later. | `reusable` |
| `lease.identity.android` | The same setting for Android emulators. `fresh` is accepted, but it is designed for iOS. | `reusable` |
| `gateway.url` | **Worker side.** Base URL of the gateway this worker joins, e.g. `wss://gw.example:4700`; the worker dials `<url>/v1/uplink` and upgrades it to a WebSocket. Joining grants that gateway the `admin` role on this daemon, and `gateway.token` rides on the upgrade request as a bearer credential, so use `wss://` — or plain `ws://` only over loopback or inside your own tunnel. `http://` and `https://` are rejected at load. Unset means "do not join a fleet": the default, and the only thing that changes about a joined worker. | unset |
| `gateway.token` | **Worker side.** The join token (`simlock token create --role worker`, minted on the gateway) this worker presents when it opens its uplink. Required whenever `gateway.url` is set. | unset |
| `gateway.label` | **Worker side.** Display name for this worker in `simlock worker list`, `status`, the console, and on the lease's `worker` block. Display-only: nothing routes on it and it need not be unique. | the worker's own id |
Expand Down Expand Up @@ -83,6 +85,10 @@ positive numbers.
**`mode: "gateway"` with `http.enabled: false` is rejected at load**, naming the key: a
gateway is the fleet's contact point over HTTP, so one nothing can reach has
no safe reading.
`lease.identity.ios` and `lease.identity.android` must each be `reusable` or
`fresh`. Any other value is rejected at load, and the error names the key.
No lease request can change this setting.

`lease.defaultTtlMs` and `lease.maxTtlMs` must be positive numbers, and
`lease.defaultTtlMs` must be `<=` `lease.maxTtlMs`. A config that violates
either rule is **rejected at load and the daemon does not start**, naming the
Expand Down
14 changes: 7 additions & 7 deletions docs/EVENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,14 @@ the gateway's own lease index before any worker is ever contacted.
|---|---|---|---|---|
| `device.provisioned` | device id, spec, driver, duration | driver `provision` committed to registry | Registry | implemented |
| `device.ready` | device id, boot duration | readiness probe passed | Registry | implemented |
| `device.reclaimed` | device id, strategy (erase/snapshot/wipe), duration | fresh-state reclaim finished | Registry | implemented |
| `device.purge-failed` | device id, lease id, attempted strategy, duration, stable error summary | release-time purge failed; the device enters `quarantined` (see below) rather than rejoining the pool | WarmPoolCoordinator | implemented |
| `device.quarantined` | device id, max retries, next retry deadline | a device committed to `quarantined` — present in the registry, still counted as running, not eligible for a grant. Fires immediately after `device.purge-failed` for a release-time purge failure, or on its own for a stalled-transition timeout (see `device.stalled-transition-detected`) | QuarantineCoordinator | implemented |
| `device.quarantine-recovered` | device id, attempts, reclaim strategy | a quarantined device's retried purge succeeded; it returned to `ready`/`shutdown` and rejoined the warm pool | QuarantineCoordinator | implemented |
| `device.quarantine-abandoned` | device id, attempts | a quarantined device exhausted its configured retry budget (`warmPool.quarantine.maxRetries`) and was destroyed | QuarantineCoordinator | implemented |
| `device.quarantine-stranded` | device id, attempts, stable error summary | a quarantined device exhausted its retry budget and the destroy that should have retired it also failed; it stays `quarantined` with no further retry until an operator intervenes | QuarantineCoordinator | implemented |
| `device.reclaimed` | device id, strategy (erase/snapshot/wipe), duration | fresh-state reclaim finished. Never emitted for a device created under `lease.identity` `fresh`: nothing is reclaimed, the device is deleted instead | Registry | implemented |
| `device.purge-failed` | device id, lease id, attempted strategy (erase/snapshot/wipe/delete), duration, stable error summary | release-time purge failed, or (strategy `delete`) the shutdown or delete that ends a `fresh` device's lease failed; the device enters `quarantined` (see below) rather than rejoining the pool. The strategy list can grow: a consumer must tolerate a strategy it does not know | WarmPoolCoordinator | implemented |
| `device.quarantined` | device id, max retries, next retry deadline | a device committed to `quarantined` — present in the registry, still counted as running, not eligible for a grant. Fires immediately after `device.purge-failed` for a release-time purge or delete failure, or on its own for a stalled-transition timeout (see `device.stalled-transition-detected`) | QuarantineCoordinator | implemented |
| `device.quarantine-recovered` | device id, attempts, reclaim strategy | a quarantined device's retried purge succeeded; it returned to `ready`/`shutdown` and rejoined the warm pool. Never emitted for a `fresh` device: its retry is a delete, and a successful one emits `device.deleted` | QuarantineCoordinator | implemented |
| `device.quarantine-abandoned` | device id, attempts | a quarantined device exhausted its configured retry budget (`warmPool.quarantine.maxRetries`) — purge retries, or delete retries for a `fresh` device — and was destroyed | QuarantineCoordinator | implemented |
| `device.quarantine-stranded` | device id, attempts, stable error summary | a quarantined device exhausted its retry budget (purge or delete retries) and the destroy that should have retired it also failed; it stays `quarantined` with no further retry until an operator intervenes | QuarantineCoordinator | implemented |
| `device.shutdown` | device id, initiator (rule/command) | device stopped, still on disk | Registry; WarmPoolCoordinator for interrupted reclaim recovery | implemented |
| `device.deleted` | device id, initiator | device removed from disk and registry | Registry | implemented |
| `device.deleted` | device id, initiator (`lease-end` when a `fresh` device's lease ended and its delete completed, including a delete retried from quarantine) | device removed from disk and registry | Registry | implemented |
| `device.foreign-state-detected` | device id, platform, expected (running/stopped), observed (running/stopped) | doctor reconcile found a managed device's observed boot state disagreeing with the committed registry state | Doctor | implemented |
| `device.foreign-provenance-detected` | device id, platform, detail (erased/mark-mismatch/durable-mark-missing) | doctor reconcile found a managed device's provenance marks no longer proving Simlock owns it | Doctor | implemented |
| `device.stalled-transition-detected` | device id, platform, state (provisioning/reclaiming), age, threshold | doctor reconcile found a `provisioning`/`reclaiming` device whose time in that state exceeds a driver-derived threshold — the driver call meant to resolve the transition never did | Doctor | implemented |
Expand Down
18 changes: 13 additions & 5 deletions docs/internal/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -879,12 +879,19 @@ One shared lifecycle for both platforms; drivers map onto it, never extend it:

```
provisioning → ready → leased → reclaiming → ready/shutdown → deleted
↓ ↓
└──────────→ quarantined ←─────┘
↓ ↓
└──────────→ quarantined ←─────┴───────────────
ready/shutdown/deleted
```

A device created under `lease.identity: fresh` serves one lease. Its lease end
skips the purge: `reclaiming → shutdown` (driver shutdown), then
`shutdown → deleted` (driver destroy). `mayBeGranted` in `domain.ts` keeps a
spent fresh device out of every grant path, including in the window between
those two commits. A failed delete enters quarantine from `shutdown`, and
quarantine retries the delete, never a reclaim.

All transitions go through the core. `simlock status` reads identically for
iOS and Android because of this.

Expand All @@ -905,9 +912,10 @@ helpers select targets by exact state (`state === "ready"`), never by
excluding known-bad states. Anything that needs "in the registry, counts
against capacity, not grantable" is expressed by adding its own entry into
`quarantined`, not by inventing a second state: the release-time purge
failure (`reclaiming → quarantined`) and the stalled-transition timeout
(`provisioning → quarantined`, both owned by `QuarantineCoordinator`) are its
two entries. The latter fires from `simlock doctor`'s `stalled-transition`
failure (`reclaiming → quarantined`), a fresh device's failed delete
(`shutdown → quarantined`), and the stalled-transition timeout
(`provisioning → quarantined`, all owned by `QuarantineCoordinator`) are its
three entries. The latter fires from `simlock doctor`'s `stalled-transition`
finding — a `provisioning`/`reclaiming` device whose time in that state has
outrun a driver-derived threshold, meaning the driver call meant to resolve
it never did and the registry's view has diverged from the driver's. Safer
Expand Down
14 changes: 7 additions & 7 deletions docs/internal/EVENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,14 @@ worker is ever contacted.
|---|---|---|---|---|
| `device.provisioned` | device id, spec, driver, duration | driver `provision` committed to registry | Registry | implemented |
| `device.ready` | device id, boot duration | readiness probe passed | Registry | implemented |
| `device.reclaimed` | device id, strategy (erase/snapshot/wipe), duration | fresh-state reclaim finished | Registry | implemented |
| `device.purge-failed` | device id, lease id, attempted strategy, duration, stable error summary | release-time purge failed; the device enters `quarantined` (see below) rather than rejoining the pool | WarmPoolCoordinator | implemented |
| `device.quarantined` | device id, max retries, next retry deadline | a device committed to `quarantined` — present in the registry, still counted as running, not eligible for a grant. Fires immediately after `device.purge-failed` for a release-time purge failure, or on its own for a stalled-transition timeout (see `device.stalled-transition-detected`) | QuarantineCoordinator | implemented |
| `device.quarantine-recovered` | device id, attempts, reclaim strategy | a quarantined device's retried purge succeeded; it returned to `ready`/`shutdown` and rejoined the warm pool | QuarantineCoordinator | implemented |
| `device.quarantine-abandoned` | device id, attempts | a quarantined device exhausted its configured retry budget (`warmPool.quarantine.maxRetries`) and was destroyed | QuarantineCoordinator | implemented |
| `device.quarantine-stranded` | device id, attempts, stable error summary | a quarantined device exhausted its retry budget and the destroy that should have retired it also failed; it stays `quarantined` with no further retry until an operator intervenes | QuarantineCoordinator | implemented |
| `device.reclaimed` | device id, strategy (erase/snapshot/wipe), duration | fresh-state reclaim finished. Never emitted for a device created under `lease.identity` `fresh` (#75): nothing is reclaimed, the device is deleted instead | Registry | implemented |
| `device.purge-failed` | device id, lease id, attempted strategy (erase/snapshot/wipe/delete), duration, stable error summary | release-time purge failed, or (strategy `delete`, #75) the shutdown or delete that ends a `fresh` device's lease failed; the device enters `quarantined` (see below) rather than rejoining the pool. `delete` widens a published vocabulary (events rule 6 allows additive changes): a consumer must tolerate a strategy it does not know | WarmPoolCoordinator (strategy `delete`: WarmPoolCoordinator's spent-device path) | implemented |
| `device.quarantined` | device id, max retries, next retry deadline | a device committed to `quarantined` — present in the registry, still counted as running, not eligible for a grant. Fires immediately after `device.purge-failed` for a release-time purge or delete failure (`reclaiming`/`shutdown → quarantined`), or on its own for a stalled-transition timeout (see `device.stalled-transition-detected`) | QuarantineCoordinator | implemented |
| `device.quarantine-recovered` | device id, attempts, reclaim strategy | a quarantined device's retried purge succeeded; it returned to `ready`/`shutdown` and rejoined the warm pool. Never emitted for a spent `fresh` device (`mayBeGranted` false): its retry is a delete, and a successful one emits `device.deleted` with initiator `lease-end` | QuarantineCoordinator | implemented |
| `device.quarantine-abandoned` | device id, attempts | a quarantined device exhausted its configured retry budget (`warmPool.quarantine.maxRetries`) — purge retries, or delete retries for a spent `fresh` device — and was destroyed | QuarantineCoordinator | implemented |
| `device.quarantine-stranded` | device id, attempts, stable error summary | a quarantined device exhausted its retry budget (purge or delete retries) and the destroy that should have retired it also failed; it stays `quarantined` with no further retry until an operator intervenes | QuarantineCoordinator | implemented |
| `device.shutdown` | device id, initiator (rule/command) | device stopped, still on disk | Registry; WarmPoolCoordinator for interrupted reclaim recovery | implemented |
| `device.deleted` | device id, initiator | device removed from disk and registry | Registry | implemented |
| `device.deleted` | device id, initiator (`lease-end` when a `fresh` device's lease ended and its delete completed — from WarmPoolCoordinator, at startup convergence, or retried from quarantine; #75) | device removed from disk and registry | Registry | implemented |
| `device.foreign-state-detected` | device id, platform, expected (running/stopped), observed (running/stopped) | doctor reconcile found a managed device's observed boot state disagreeing with the committed registry state | Doctor | implemented |
| `device.foreign-provenance-detected` | device id, platform, detail (erased/mark-mismatch/durable-mark-missing) | doctor reconcile found a managed device's provenance marks no longer proving Simlock owns it | Doctor | implemented |
| `device.stalled-transition-detected` | device id, platform, state (provisioning/reclaiming), age, threshold | doctor reconcile found a `provisioning`/`reclaiming` device whose time in that state exceeds a driver-derived threshold (`stalledTransition.thresholdMultiplier` over `Driver.estimate`, floored at `stalledTransition.minimumThresholdMs`) — the driver call meant to resolve the transition never did | Doctor | implemented |
Expand Down
Loading