design: recorders — recorded, non-deterministic, at-least-once transformations

[antiguru]

What

A design doc proposing recorders: a way to house transformations that must record a decision made at processing time and never recompute it — non-deterministic, side-effecting, or time-dependent logic (including UDFs). The guarantee is at-least-once application of the function and at-most-once persistence of its result (committed once via a per-commit CAS).

Write-side complement to the CHANGES table function (#36869, draft); a deliberate revival of the removed Continual Tasks feature (database-issues#9694 / PR #35967), pulled for lack of design consensus and incompleteness, not because the model is unsound.

Docs: doc/developer/design/20260604_recorders.md (conceptual) and 20260604_recorders_implementation.md (feasibility). The model matches the WG "Recorders" design (Notion); these docs are the long-form version.

The calculus

• a TVC — a normal collection; change implicit.
• a dTVC — a changelog carrying its change as data: a time and a diff column (ordinary, user-named).

differentiate (CHANGES) and integrate (INTEGRATE) are a carrier-preserving inverse pair, both in the input's timeline (domain A). The cross-domain, non-deterministic leg — picking the commit time T, sampling frozen values like now() — lives only in RECORD. freeze is sample-and-hold.

The surface (no new collection kind)

-- a regular table that declares its carriers + an associated progress collection.
CREATE TABLE enriched (
  a BIGINT, b BIGINT, c BIGINT, val TEXT,
  change_ts mz_timestamp NOT NULL, change_diff bigint NOT NULL
) WITH (PROGRESS, TIMESTAMP = change_ts, DIFF = change_diff);   -- inits progress to (0,0)

-- the one new standing object; bare query like an MV, INTO like a SINK.
CREATE RECORDER enrich INTO enriched AS
  SELECT e.*, d.val
  FROM CHANGES(events AS OF AT LEAST mz_now() - INTERVAL '1 hour'
               USING TIME change_ts, DIFF change_diff) e
  JOIN dim d ON e.fk = d.key;            -- bare TVC ref ⇒ frozen lookup

-- argument-free: carriers + progress come from the table.
CREATE MATERIALIZED VIEW enriched_now AS SELECT * FROM INTEGRATE(enriched);

-- bounding is ordinary DELETE; integral-preserving = data-domain compaction.

• The store is a regular table that declares its carriers + an associated progress collection via WITH (PROGRESS, …) (initialized to (0,0); empty progress = sealed frontier). No DELTA TABLE collection kind.
• change_diff is data, not the persist multiplicity. CHANGES emits each change as a +1 row with the signed count in the column.
• INTEGRATE(table) is argument-free — carriers and the progress collection come from the table; it accumulates DIFF per row, thresholding at max(0, Σ) internally (safe by construction; repeat_row not exposed). A stateful reduce, memory ∝ live output.
• The progress collection is owned by the table, engine-written / user-read-only (the CREATE SOURCE … EXPOSE PROGRESS AS precedent); INTEGRATE reclocks through it. A RECORD writer only writes a lane. Taking a table (not arbitrary SQL) is what keeps the progress lookup unambiguous.
• Application shaping — dedup, first-seen, upsert, top-k — lives in the RECORD body, not in INTEGRATE; first-seen/upsert bodies are self-referential (Tier 3).
• Bounding is ordinary DELETE; the integral-preserving form is data-domain compaction — a clamp + GROUP BY/SUM reduce, not a bare UPDATE.

Decisions worth reviewers' attention

• Per-table progress collection + per-writer lanes. B is the table's single shard frontier ⇒ exactly one progress collection per table; all RECORDs and INTEGRATEs share it (multiple readers consistent by construction). Multi-writer = per-writer lanes R_i: B → X_i, table A-completeness = meet_i R_i[B_upper] at read time (a merged frontier can't work). Drop the last writer → freeze or seal (open).
• Optimizer barrier (hard invariant): the recorded table is authoritative; never recomputed from the RECORD body's inputs.
• Freeze is typing, not a keyword — bare TVC ref in a RECORD body = frozen; must be diagnosable, never silent (EXPLAIN + plan-time NOTICE). In a body, now() (frozen wall-clock) ≠ mz_now() (domain-A event time).
• Aging domain decided: retention defaults to wall-clock (domain B), event-age (A) opt-in. Where RETAIN HISTORY sits (table vs INTEGRATE consumer vs recorder — leaning INTEGRATE) is open.
• GDPR erasure = DELETE + advancing since; a cascade DELETE alone is not erasure.

SQL-consistency pass

An agent reviewed the surface against Materialize's grammar (file:line) and prior art:

• Carriers via a USING clause / table WITH (…), not => named args (MZ has no named-arg support; => already means map-entry). CHANGES's AS OF parses like SUBSCRIBE (query) AS OF …. Dropped the AS RECORD(...) wrapper; IN DOMAIN → WITH (TIMELINE = …).
• Deliberate divergences: DIFF is a signed multiplicity, not an op-enum (cf. Snowflake METADATA$ACTION, Flink +I/-U, Debezium c,u,d); CHANGES knowingly collides with Snowflake's CHANGES clause.

Status

Draft, for discussion. De-risked by the CHANGES PR (#36869) and the BEGIN CONTINUAL TRANSACTION prototype. Gating dependency: the (unbuilt) OCC timestamped-write substrate — INTEGRATE and bounding DML are off that critical path. Collapsing to a regular table + progress collection removes the speculative DELTA TABLE kind (impl M2) and reshapes engine-owned compaction (M3/M5). Open agenda: co-design the CHANGES carrier spelling with #36869, the RECORDER keyword, commit-timestamp policy, RETAIN HISTORY placement, freeze-vs-seal on drop-all.

Stakeholders: @antiguru + the WG Continual Tasks folks (Aljoscha, Seth, Frank).

https://claude.ai/code/session_015YFH7J7PaEqkBSrAQYaq8H

[antiguru]

Review — Recorders design doc

Strong, unusually well-grounded design. The conceptual framing is the best part; my substantive concerns are concentrated in freeze-by-typing ergonomics, the multi-writer domain-A frontier, and one success-criterion vs. mechanism mismatch on compliance erasure. None are blockers for a discussion draft — they're the agenda.

Strengths

• The calculus reduces surface area honestly. Framing stream-table join, upsert, finalization, dedup, and compliance as compositions of four operations — rather than bespoke features — is the right answer to exactly why CTs bloated and stalled.
• Intellectual honesty about feasibility. H1 states the gating OCC commit substrate is unbuilt ("zero hits in the tree") — confirmed. H2 admits the relaxed evaluation rule removes the fixpoint but not the lagged self-read machinery.
• Code references are accurate. Line numbers check out (group_commit at appends.rs:344, append_table at storage-controller/src/lib.rs:2082); the CT as-of special-casing the doc wants to reuse is genuinely still live in compute-client/src/as_of_selection.rs. The continual_task.rs render module is gone but the doc correctly notes it's recoverable from base commit add050bf8. The salvage list is credible.
• The z-transform duality (D = 1−z⁻¹, I = D⁻¹, freeze = sample-and-hold) is a genuinely clarifying mental model, and makes "INTEGRATE carries no non-determinism, hence no new object kind" obvious.

Substantive concerns / questions

1. Freeze-by-typing is the sharpest usability footgun — strengthen the diagnostics story.
The same SQL — JOIN dim d — means a maintained join in a plain MV but a silently frozen, sampled-once snapshot inside a RECORD body, with freeze as the default and tracking the opt-in. This is spooky-action-at-a-distance: identical join syntax flips semantics based on the enclosing object kind, with no marker at the join site. The plan-time taint rule protects plain MVs from accidental freeze, but nothing protects a RECORD author from accidentally freezing a reference they meant to track — and they get no error. I'd push Open Question 4 further: consider an explicit marker (even if redundant for the type checker) or strong EXPLAIN/notice output naming every frozen reference in a body. Arguably the safe (tracked) reading should be the default and freeze the explicit one.

2. The multi-writer domain-A completion frontier is underspecified.
Several RECORD writers into one delta table is marked Decided, and the table-owned reclock "recovers the precise A→B mapping." But INTEGRATE must drive v's domain-A frontier, which is a meet across all active writers' committed-through A-times — one slow/idle writer holds back the whole table's A-completeness. The doc never states this meet explicitly, nor what happens when a writer is dropped (does the A-frontier jump forward?) or stalls indefinitely. Given multi-writer is decided, the cross-writer A-frontier computation deserves specification, not just an assertion that it "works."

3. The mz_now()/aging domain (A vs B) is flagged open, but it's the most consequential unresolved semantic — resolve it early.
Whether retention mz_now() < mz_timestamp + W is event-age (domain A, stalls when input idles) or wall-clock (domain B, always advances) changes user-visible behavior dramatically and dictates which temporal-filter machinery you reuse (M3/M4). This isn't polish-phase; it gates the bounding design and the user mental model. Worth pulling forward to a Phase 0/1 decision.

4. mz_timestamp-surfaced-as-data instability may quietly undermine the headline dedup/first-seen use case.
The pitfall is honestly documented: … AS first_seen_at inherits the max(mz_timestamp, upper) clamp and can advance as since ticks forward, stable only within RETAIN HISTORY. But "first seen at" is precisely the column users will project and expect to be immutable — tying its stability to a retention knob makes a correctness-flavored property depend on an ops setting. This is more than "document it"; it warrants a louder warning / lint (gestured at in Open Q5), and the dedup example shouldn't be read as endorsing a stable first_seen_at it can't guarantee.

5. Make the "DELTA TABLE is not recomputable from inputs" optimizer barrier a stated hard invariant.
The impl doc (Open Q7) correctly says the optimizer must not treat the delta table as recomputable from the original inputs, but may treat INTEGRATE as recomputable over the delta table. This is correctness-critical — if the optimizer ever "sees through" RECORD to re-derive frozen/non-deterministic values, the determinism-boundary argument collapses. It currently reads as a passing remark; promote it to a named invariant in the conceptual doc's "determinism boundary" section, which is the load-bearing soundness argument.

6. Success-criterion vs. mechanism mismatch on compliance erasure.
Success Criteria says deleting a user "physically erases" their recorded rows. "Bounding growth" then correctly explains true erasure requires advancing since (forfeiting AS OF/replay in that range), and that a consolidating retraction at event-time alone leaves rows visible to earlier AS OF reads — "insufficient for GDPR." These are consistent only if you read the criterion as "consolidating delete + since advancement." As written it overpromises; restate the criterion with the since-advancement caveat so it can't be read as "a cascade DELETE alone is GDPR-compliant."

Minor / editorial

• Heavy duplication between the two docs. The time-domains/reclock explanation appears nearly verbatim in both the conceptual doc ("Time domains and reclocking") and the impl doc (Open Q7) — ~40 echoed lines. The impl doc should reference rather than restate.
• RECORDER vs RECORD writer terminology drifts (example says CREATE RECORDER … INTO, prose says "the RECORD writer", title says "Recorders"). Self-flagged as open, but worth picking one for readability even in draft.
• Tier table: Tier 1 "exactly-once into persist" still rests on the per-commit CAS (stated elsewhere); worth noting in the table.
• The DELETE (SELECT …) FROM enriched cascade example reads as pseudo-syntax — fine given "syntax TBD," but maybe label it inline.

Recommendation

Approve the direction for discussion — the doc is in excellent shape for driving WG consensus. Before it graduates from draft I'd want: (a) the freeze-default safety question resolved, (b) the multi-writer A-frontier specified, (c) the mz_now() domain chosen, (d) the compliance success-criterion reworded to match the mechanism. Items 2–4 are also the most likely to bite the implementation, so resolving them now de-risks Phase 1.

---

Generated by Claude Code