Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -121,9 +121,11 @@ jobs:
echo "authority: $AUTHORITY ($AUTHORITY_NA not applicable)"
echo "templates: $TEMPLATES ($TEMPLATES_NA not applicable)"
test "$AUTHORITY" = "verified 11/11"
test "$AUTHORITY_NA" = "0"
# G13 is N/A on SQLite, the action's default store: SQLite has no clock of its own
# to diverge from (SPEC-v0.7 §8.9).
test "$AUTHORITY_NA" = "1"
test "$TEMPLATES" = "verified 6/6"
test "$TEMPLATES_NA" = "5"
test "$TEMPLATES_NA" = "6"
test -s verify-badge.json
test -s verify-report.json
test -s verify-report.xml
Expand Down
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,28 @@ any change to one appears here.
3.11, 3.12, 3.13 and 3.14, and the package classifiers name all four. The floor is unchanged:
`requires-python` stays `>=3.11`, and mypy and ruff still check against 3.11. No library code
changed; the one test fix is below.
- **Clock-skew detection** (SPEC-v0.7 §3, item 1). `PostgresStateStore` measures its server's
clock against the application's at open, and again when an expired lease is declared
`AMBIGUOUS` (at most once per `DEFAULT_LEASE`), in one round trip whose half is the
measurement's bound, so latency alone is never reported as skew. It keeps its latest
measurement as the optional, read-only `clock_skew` attribute, a `ctrlrun.state.ClockSkew`;
`Control` reads it at the start of every `execute` and `resume` and after an `AmbiguousEffect`,
and appends one new event type, `CLOCK_SKEW_DETECTED`, for a measurement past
`clock_skew_threshold` (default one second, at most `DEFAULT_LEASE`, and no value switches it
off). **It observes and reports, and changes no decision**: every lease is still evaluated
against the application clock exactly as at 0.6.1, no reservation outcome changes, and a
measurement that fails is logged and changes nothing. Verify gains G13, graded against a
Postgres `--store-url` and `N/A` on SQLite, and the catalogue moves to
`ctrlrun.guarantees/v3`; the store conformance suite gains a `clock` case, `not_applicable`
on SQLite and the in-memory store because neither has a clock of its own.

### Fixed

- **`PostgresStateStore.events()` read a missing `action_id` back as the string `"None"`.** An
event about no action (the three `DELEGATION_*` types, and now `CLOCK_SKEW_DETECTED` reported at
open) named a proposal called "None" on Postgres alone; SQLite and the in-memory store returned
`None`. It now returns `None` on all three.

- **The migration tests' release fixtures could not build a venv on some interpreters.**
`venv.create` copies the interpreter by default, and a copied binary from a shared-libpython
build (uv's CPython 3.14 on macOS) aborted inside `ensurepip`, so all five release fixtures
Expand Down
4 changes: 4 additions & 0 deletions docs/SPEC-v0.6.md
Original file line number Diff line number Diff line change
Expand Up @@ -1818,6 +1818,10 @@ A fixture that fails nothing is a failure; a fixture whose named suite passed is
exception of `reservation`'s cross-process case for `InMemoryStateStore`, which is
`not_applicable` with §2.4's reason. No other N/A is accepted, from either backend.

*Amended by `SPEC-v0.7.md` §8 T214 (its §9.6, item 7):* the `clock` suite's `skew-measured` case
is also `not_applicable` on both, with T214's reason, because neither has a clock of its own to
measure. That is the one further N/A accepted.

#### T142 — The report refuses a degenerate run
Every case `not_applicable` → `report.ok` is `False`. `run(backend, only=…)` naming a case that is
not in the registry **raises**, rather than silently running everything or nothing.
Expand Down
103 changes: 103 additions & 0 deletions docs/SPEC-v0.7.md
Original file line number Diff line number Diff line change
Expand Up @@ -2450,6 +2450,109 @@ decided, not afterwards.

### 12.1 Item 1: clock skew

**Item 1 observes and reports.** No lease is evaluated differently, no reservation outcome changes and no
store write changes: the measurement is one `SELECT clock_timestamp()` and one attribute assignment, and T213
compares every decision and every record afterwards with both `plan_reservation` and a SQLite store driven
through the same steps.

**Where the `E3` re-measurement lives.** In `_ambiguate`, after the kept `AMBIGUOUS` write commits and before
`_plan` raises the refusal, on that write's own fresh connection, which the commit has just left outside any
transaction. So `_plan`, the reservation transaction and the lost-commit paths are untouched (item 3a changes
those), and a measurement that fails cannot abort the transaction the refusal belongs to. A kept write that
itself fails raises as at 0.6.1 and measures nothing, because it is no longer §3.5's moment.

**The rate limit counts attempts, on the application clock.** A re-measurement is due unless the last one was
at or before `now` and less than `DEFAULT_LEASE` ago. The attempt counts, not the success, so a failing query is
not retried on every refusal either; a clock that moved backwards makes one due, since that is itself worth a
reading. The application clock and not a monotonic one, because it is the clock the lease that just expired was
judged by, and it is the only one a test can move (T215).

**A round trip the application clock measured as negative** (an injected or stepped clock) is taken by its size:
`bound = |t1 - t0| / 2`, `midpoint = min(t0, t1) + bound`. The doubt is the same whichever way the reads came.

**`ClockSkew` checks its fields at construction**: `timedelta` for the three durations, a non-negative `bound`, a
positive `threshold`, an aware `measured_at` and a `trigger` in the closed pair. A third-party store that builds a
malformed one then fails where it built it, rather than inside the `Control` that would report it. `Control` still
treats any exception from reading or rendering the value as §3.6's "read raised", because a subclass can override
`exceeded`. The pair is private (`_CLOCK_SKEW_TRIGGERS`), so no public name is added beyond §9.2's.

**"Not the measurement it last reported" is equality, not identity.** A store whose property builds a fresh
`ClockSkew` on every read with the same fields is then reported once, not once per action. Two measurements that
differ in any field are two reports, which is what T217's third step asserts.

**"Once per store per kind" is two kinds**: the value is not a `ClockSkew` (whatever its type), and the read or
its rendering raised. They are keyed on the store object, weakly, so a process that builds a `Control` per request
around one store still logs each kind once; a store that cannot be weakly referenced falls back to the reading
`Control`'s own set.

**`getattr(store, "clock_skew", None)` treats a property that raises `AttributeError` as absent**, which is
§3.6's literal read and is kept. The conformance case is where that store's author finds out: it asks for the
attribute with `inspect.getattr_static` first, so a present property whose read raises `AttributeError` fails the
case by name rather than earning the `not_applicable` reserved for an absent one. A forwarding wrapper whose
`__getattr__` reaches a real attribute counts as exposing it.

**The pull is the first statement of `execute` and of `resume`**, before argument checks and before
`take_continuation`, so an at-open report precedes the first `ACTION_PROPOSED` (G13's observable). The pull after
an `AmbiguousEffect` is the first statement of `_secure`'s handler, before reconciliation and before the refusal's
own event, and observe mode's reservation refusal pulls too, because observe mode reserves and so meets `E3`.
`evaluate`, `delegate` and `revoke` do not pull: §3.6 names `execute` and `resume`, and none of the three meets a
lease.

**`clock_timestamp()` against `now()` is an equivalent mutant as built**, and the mutation table says so rather
than claiming it closed. Both measurements run outside any transaction (the store's connections are autocommit, and
the `E3` one runs after its commit), so `now()` is the single statement's start and agrees with
`clock_timestamp()` to within the statement. `clock_timestamp()` stays, per §3.4, so that a later caller who
measures inside a transaction does not inherit an error the tests cannot see.

**`data.measured_at` uses the event-data timestamp convention** (`iso_timestamp`, milliseconds, `Z`), as
`lease_expires_at` does. The three numbers are integer microseconds, exact, as §3.6 requires.

**A defect the event exposed, fixed here.** `PostgresStateStore.events()` read a NULL `action_id` back as the
string `"None"`, so the three `DELEGATION_*` events have named a proposal called "None" on Postgres since that
store shipped.
T217's comparison of what a sink was handed with what `events()` returns found it. The fix is on the read path
only; nothing is written differently.

**The conformance case.** Suite `clock`, case `skew-measured`. It aligns by a first measurement against the
host's real clock, as G13 does, and grades a store's retained measurement (`exceeded`), because the suite grades
stores and not `Control`. Four broken-store fixtures keep each check live: a look-alike type, a read that raises, a
detector that never fires and one that always fires.

**G13 needs nothing from the document but the store.** It proposes an action no document names, so the policy
denies it with `unknown_action`; the report under test is taken at the start of `execute`, before any decision, so
a denial reaches it as surely as an allow. On Postgres it is therefore never `N/A`, which is what §8.9's
*Requires* line says. Its catalogue title is *clock divergence is named*, short enough for the report's column.

**What §3.8 predicted, measured.** Every verify scenario on Postgres now appends one `CLOCK_SKEW_DETECTED` per
scratch store, because verify's clocks are anchored to the document. No scenario counted events, and all three
shipped examples still pass every applicable guarantee under `--store-url postgresql://…`. No existing Postgres
test asserted a complete event sequence against an injected clock, so none needed changing. SQLite runs of verify
report G13 `N/A`, which moves the counts T113 and T116 pin by one.

**An injection is sized against the bound, never fixed (review of #136).** G13 and the conformance case both
inject a skew and ask whether it was reported, and both first used a fixed margin past the threshold. The bound is
half the round trip to the store, so on a link whose half exceeds that margin a *conforming* store reports nothing
and the fixed margin calls that silence a defect: verify would have graded the link and blamed the kernel. Both now
widen the injection from the bound the shifted store measured, until a store honest within its bound would have to
report it (`threshold + 2 * bound + alignment`, one definition in `state.py` so the two cannot drift apart), and
both retry a bounded number of times. A report on the clock they meant to align, where the aligning measurement's
own doubt could explain it, is met by aligning again rather than by a FAIL. A link that cannot be outrun is
**verify's internal error, exit 3** (`v0.4 §3.8`: a fact about the machine, never a verdict on the kernel) and, in
the suite, a failure whose reason names the link and says it is not a report the store failed to make. The test for
each injects real latency rather than simulating it. Whether the alignment's own doubt excuses a report is decided
by recomputing the rule from the measurement's fields rather than by reading `exceeded`, so a store whose
`exceeded` always answers true is still caught by the control.

**A report the store cannot store changes nothing (review of #136).** `append_event` can fail, and it sat
unguarded, so a locked database would have raised out of `execute` before the action was decided, and out of the
`AmbiguousEffect` handler in place of the refusal the caller was owed: an observation deciding an outcome, which is
the one thing §3 says it never does. The append is now guarded like the read, logged once per store per kind, and
`_skew_reported` moves only after the store accepted the event, so a report that was lost is made by the next
action that can store it and a sink is handed only an event that was stored.

**`v0.1 §6.2`'s list is not edited in place.** v0.2 and v0.3 added nine event types without touching it, and
§9.6 item 2 records this one where the others are recorded. `v0.6 §8` T141 is amended in place, as §8 T214 asks.

### 12.2 Item 2: the transport classifier

### 12.3a Item 3a: attempt numbers never repeat
Expand Down
7 changes: 4 additions & 3 deletions src/ctrlrun/conformance/store/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,10 @@
**Not applicable is not a pass.** §2.4 allows exactly two N/As, each a property of the backend
rather than of the harness: storage that cannot be opened from another process, and storage that
does not outlive the object holding it. Both describe `InMemoryStateStore`, which says so in its
own docstring, and `falsely-declares-no-url` is what keeps the declaration honest. Any third N/A
is a failure, `report.ok` is `False` for a zero denominator, and there is no flag that folds one
into the count.
own docstring, and `falsely-declares-no-url` is what keeps the declaration honest. SPEC-v0.7
§8 T214 adds exactly one more, also a property of the backend: a store that exposes no clock
measurement, because it reads only the application's clock. Any other N/A is a failure,
`report.ok` is `False` for a zero denominator, and there is no flag that folds one into the count.

Nothing in the kernel imports this package, and `import ctrlrun` does not reach it (T140f).
"""
Expand Down
90 changes: 88 additions & 2 deletions src/ctrlrun/conformance/store/fixtures.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@
import tempfile
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, replace
from datetime import datetime, timedelta
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any

from ...action import Action
from ...effect import DEFAULT_LEASE, EffectRecord, EffectState, Reservation
from ...errors import NotExecuted
from ...receipt import Event
from ...state import DelegationRecord, StateStore
from ...state import ClockSkew, DelegationRecord, StateStore
from .backends import SQLiteBackend, StoreBackend


Expand Down Expand Up @@ -364,6 +364,68 @@ def put_delegation(self, record: DelegationRecord) -> None:
self._inner.put_delegation(record)


# --- clock (SPEC-v0.7 §8 T214) ---------------------------------------------------------------


@dataclass(frozen=True)
class _ClockSkewLookAlike:
"""Every field `ClockSkew` has, and not `ClockSkew`. `Control` ignores it by design."""

skew: timedelta
bound: timedelta
threshold: timedelta
measured_at: datetime
trigger: str

@property
def exceeded(self) -> bool:
return abs(self.skew) > self.threshold + self.bound


class _SkewLookAlike(_Wrapped):
"""Exposes its measurement as a look-alike type rather than `ctrlrun.state.ClockSkew`."""

@property
def clock_skew(self) -> Any:
return _ClockSkewLookAlike(
timedelta(seconds=30), timedelta(0), timedelta(seconds=1), datetime.now(UTC), "open"
)


class _SkewReadRaises(_Wrapped):
"""Exposes the attribute, and reading it raises."""

@property
def clock_skew(self) -> ClockSkew | None:
raise RuntimeError("the measurement could not be read")


def _pinned(skew: timedelta) -> ClockSkew:
return ClockSkew(
skew=skew,
bound=timedelta(0),
threshold=timedelta(seconds=1),
measured_at=datetime.now(UTC),
trigger="open",
)


class _SkewNeverReported(_Wrapped):
"""A detector that never fires: every measurement says the clocks agree."""

@property
def clock_skew(self) -> ClockSkew | None:
return _pinned(timedelta(0))


class _SkewAlwaysReported(_Wrapped):
"""A detector that always fires: every measurement says the clocks are an hour apart."""

@property
def clock_skew(self) -> ClockSkew | None:
return _pinned(timedelta(hours=1))


# --- the declarations ------------------------------------------------------------------------


Expand Down Expand Up @@ -551,6 +613,30 @@ def raises_not_executed(root: Path) -> StoreBackend:
_wrapping("upserts-a-delegation", _UpsertsADelegation),
because="upserted on a duplicate id",
),
Fixture(
"skew-look-alike",
{"clock": "skew-measured"},
_wrapping("skew-look-alike", _SkewLookAlike),
because="not a ctrlrun.state.ClockSkew",
),
Fixture(
"skew-read-raises",
{"clock": "skew-measured"},
_wrapping("skew-read-raises", _SkewReadRaises),
because="reading clock_skew raised",
),
Fixture(
"skew-never-reported",
{"clock": "skew-measured"},
_wrapping("skew-never-reported", _SkewNeverReported),
because="was not reported",
),
Fixture(
"skew-always-reported",
{"clock": "skew-measured"},
_wrapping("skew-always-reported", _SkewAlwaysReported),
because="aligned with the store's was reported",
),
Fixture(
"falsely-declares-no-url",
{"reservation": "e1-cross-process", "durability": "ambiguous-survives"},
Expand Down
Loading
Loading