From 51d469d4a9a6b1e6bae4870382cedfe17675bb16 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Thu, 8 Oct 2026 18:08:48 +1100 Subject: [PATCH 1/4] docs(adr): one value model, three encodings in vitaminc, EQL v4 by producer Stack Encrypt can produce one EQL type because its ciphertext, equality and order encodings disagree about which kinds exist, and the mapping from a kind to a Rust type lives in stack-encrypt (Scalar, dynamic::Value) instead of vitaminc. EQL also cannot tell a Stack Encrypt column from a cipherstash-client one, so a query from one producer against the other's column matches nothing, silently. ADR-0002 records the decisions from the value-encodings RFC review: the new kinds and their frozen tags, one canonical form per kind for terms, term derivation moving into vitaminc-prf and a new vitaminc-ore, and EQL v4 as the v3 SQL emitted under a second name with "v": 4 envelopes, so Postgres keeps the two producers apart. The plan-builder plan defined "EQL v4" as a name for a Stack Encrypt payload in the v3 envelope and eql_v3 domains; it now points at the ADR. --- ...-three-encodings-and-eql-v4-by-producer.md | 269 ++++++++++++++++++ docs/plans/2026-10-04-plan-builder.md | 13 +- 2 files changed, 277 insertions(+), 5 deletions(-) create mode 100644 docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md new file mode 100644 index 000000000..b5f858b10 --- /dev/null +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -0,0 +1,269 @@ +--- +status: accepted +date: 2026-10-08 +relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 +--- + +# One value model, three encodings in vitaminc, and EQL v4 separated by producer + +Every value Stack Encrypt handles is encoded three times: as a **ciphertext** +(reversible and self-describing), as an **equality term** (a keyed hash that +must be unambiguous) and as an **order term** (bytes whose order is the +value's order). This ADR decides where each encoding lives, which kinds of +value exist, the canonical form each kind takes, and how EQL keeps columns +written by Stack Encrypt apart from columns written by cipherstash-client. + +## The problem + +In October 2026 Stack Encrypt could produce one of EQL's 51 types, `TextEq`. +The number, date, timestamp and boolean families were blocked on encoding, +not code: + +- **The three encodings disagreed about which types exist.** `i16` had an + equality domain in `vitaminc-prf` and an order encoding, but no ciphertext + tag, so it could not be a kind. Date, timestamp and decimal had order + encodings only. +- **The mapping from a kind to a Rust type lived in Stack Encrypt.** + `dynamic::term::Scalar` mirrored `FfiValue`'s variants one for one to + dispatch to the term crates, and `dynamic::Value` wrapped `FfiValue` only to + add `Clone`. Both restated vitaminc's vocabulary in a downstream crate. +- **Ordering was implemented twice.** `orderable-bytes` defines a canonical, + order-preserving encoding per type. cllw-ore uses it for chrono and decimal + but hand-rolls integers and floats, and differs on `-0.0`. Block ORE was not + derivable at all, so `TextOrdOre` and `TextSearchOre` were refused. +- **The existing writer's encodings are inconsistent.** cipherstash-client + hashes a timestamp's milliseconds but orders by its nanoseconds, and hashes + a decimal with its scale while ordering ignores it. Its block ORE text is + ASCII-only, lowercased, maps every digit to one symbol and truncates to six + blocks. cllw-ore's text decomposes to NFD and strips accents. +- **Two producers wrote the same EQL domains.** A Stack Encrypt query term + never matches a cipherstash-client term for the same value, and the + `eql_v3_*` domains could not tell them apart. A query from one against a + column written by the other returned no rows, silently. + +## Options considered + +**Where the value model lives.** + +1. **Move `vitaminc-aead-value` into Stack Encrypt.** It looks like FFI + plumbing. But it is the value model for vitaminc's own `Cipher` traits: + `aead-napi` and vitaminc's Go binding encrypt with `Aes256Cipher` through + it, without Stack Encrypt. Moving it makes vitaminc Rust-only, makes + vitaminc depend on Stack Encrypt (which depends on six vitaminc crates), + or forks the frozen tag table into two copies. +2. **Keep it in vitaminc, and move term derivation there too.** All three + encodings then live in one repository, and one exhaustive `match` on the + value type per layer makes "a kind exists in every layer or in none" a + compile error rather than a convention. Chosen. + +**How EQL separates the two producers.** The SQL for both is identical; only +the producer of the terms differs. + +1. **Name only.** Stack Encrypt payloads go in the `eql_v3_*` domains, + distinguished by the `stack-encrypt:1:` ciphertext prefix. Nothing in the + database stops a cross-producer comparison. +2. **A producer tag in every payload and term, checked by every operator.** + A runtime check on the hottest SQL paths, across about 24k lines of + hand-written SQL. +3. **One SQL source, emitted under two names.** The build writes the same + source out as `eql_v3` (cipherstash-client terms) and `eql_v4` (Stack + Encrypt terms). A column's domain names its producer, and Postgres refuses + to compare an `eql_v4_*` query term with an `eql_v3_*` column at plan time. + Chosen. + +**How v3 and v4 ship.** Bumping `main` to v4 and patching v3 from a branch +needs a maintenance release path for five lockstep artefacts that does not +exist, and the EQL publish script puts every stable release on `latest`, so a +3.x patch would move `latest` backwards. A second package doubles the +trusted-publishing and release surface. One package carrying both bundles +needs neither. Chosen. + +## Decision + +### Kinds and the ciphertext + +- vitaminc 0.6.0 adds the kinds `Int8`, `UInt8`, `Int16`, `UInt16`, `Int128`, + `UInt128`, `Date`, `Timestamp` and `Decimal`, each with a `ValueKind` name, + a `Value` variant and a ciphertext tag, in one breaking release. Rust `i128` + and `u128` get `Encrypt` and `Decrypt` impls. +- `FfiValue` is renamed `Value`, with `#[deprecated] pub type FfiValue = Value` + for one release. `Value` and `ValueKind` become `#[non_exhaustive]`, so later + kinds are additive. +- `Value` implements `Clone` as a deep copy that rebuilds every leaf into a + fresh `Protected`. A clone is under the same custody as its original. +- The sealed leaf tag table (`tags.rs`) stays frozen and contiguous. The + transport codec's framing tags move from `0x10`–`0x12` to `0xF0`–`0xF2`; + transport carries no compatibility commitment and every user of it updates + together. The new leaves: + + | Tag | Kind | Payload | + |---|---|---| + | `0x0C`–`0x11` | `Int8`, `UInt8`, `Int16`, `UInt16`, `Int128`, `UInt128` | 1, 1, 2, 2, 16, 16 bytes; two's complement for signed; little-endian | + | `0x12` | `Date` | `i32` days counted from 0001-01-01 (`num_days_from_ce`), little-endian | + | `0x13` | `Timestamp` | `i64` Unix seconds then `u32` nanoseconds, little-endian, UTC | + | `0x14` | `Decimal` | rust_decimal's 16-byte `serialize()`, which keeps the scale | + +- `transport` stays a module of `vitaminc-aead-value`. + +### Canonical forms + +The ciphertext keeps the value exactly as given: `1.50` decrypts as `1.50`, +and a timestamp keeps its nanoseconds. Equality and order terms are computed +from one canonical form per kind, and both layers use the same one, so +equality and ordering agree by construction. + +| Kind | Canonical form for terms | +|---|---| +| `Timestamp` | truncated to microseconds, Postgres's precision | +| `Decimal` | scale normalised (`1`, `1.0` and `1.00` are equal). NaN and ±Infinity are refused at encode time; rust_decimal cannot represent them | +| `Float32`, `Float64` | `-0.0` folded into `+0.0`; every NaN replaced by one positive quiet NaN, which sorts above +Infinity. This matches Postgres | +| text, equality | NFC | +| text, order | NFC, then NFD, combining marks removed, Unicode default case folding | + +Text normalisation is pinned. The Unicode version is part of the encoding's +domain label (for example `text-nfc/unicode-16/v1`), the +`unicode-normalization` crate is pinned to it, and strings containing +unassigned code points are refused. Order terms fold accent and case because +code-point order puts `é` after `z`; a fixed, pinned fold approximates the +first level of Unicode collation without depending on ICU, whose sort keys +change between versions. Equality is not folded: a folded order term only +produces ties, while a folded equality term produces false matches. A +case-insensitive equality is its own domain. + +Truncation and alphabet packing are settings of an EQL domain, named in its +label, never part of the shared encoding. + +### Which kinds get which terms + +- **Every scalar kind has a ciphertext.** Containers and null have no terms. +- **Equality:** every scalar kind except `Bool`, including floats over their + canonical bits. +- **Order:** every scalar kind except `Bool`, under all three schemes. +- **`Bool` has a ciphertext only.** A keyed hash or an order term over a + domain of two values hides nothing: it splits the rows into two groups, and + an order term also says which group is `true`. The existing CLLW order terms + for `Bool` are removed. + +These exceptions are an explicit, documented list kept next to vitaminc's +conformance test, which fails for any kind that is missing from a layer +without an entry. + +### Where each encoding lives + +- **Ciphertext:** `vitaminc-aead-value`. +- **Equality:** `vitaminc-prf`, which gains domains for every new kind, floats + and `Decimal`, over the canonical bytes. +- **Order:** a new `vitaminc-ore` crate. + - The plaintext layer is `orderable-bytes`, which stays its own crate in + ore.rs and gains a variable-length encoding for text and bytes. Its + existing fixed-length output does not change. + - A `Scheme` trait, with block ORE (over ore-rs), CLLW ORE and CLLW OPE + (over cllw-ore). A scheme only encrypts the bytes the plaintext layer + produces, so every orderable kind works under every scheme. + - Block ORE uses each kind's natural width. The 8-byte padding is a + cipherstash-client detail kept for its stored terms. + - cllw-ore keeps its typed impls, unchanged, behind a cargo feature that + cipherstash-client enables, and gains a bytes-level entry point that + `vitaminc-ore` calls. +- **Term derivation** takes a `&Value` in `vitaminc-prf` and `vitaminc-ore`, + with one exhaustive match per layer that keeps leaves inside `Protected`. A + pairing a layer does not support returns a typed error from vitaminc. + +### Stack Encrypt + +- `dynamic::Value` and `dynamic::term::Scalar` are deleted. `dynamic::term` + takes a `&vitaminc_aead_value::Value`. `admits` keeps only the rules that + belong to Stack Encrypt, such as `Match` taking text only. +- Order terms go through `vitaminc-ore`, and a block ORE term kind joins CLLW + ORE and OPE. Stack Encrypt no longer depends on cllw-ore directly. +- `TextEq` normalises to NFC and moves to EQL v4. The Stack Encrypt targets on + `eql_v3_*` domains are deleted. The `stack-encrypt:1:` ciphertext prefix + stays: decryption does not pass through Postgres types, and the prefix is + what lets it reject the other producer's payload. + +### Host languages + +- **Go.** `int8`, `int16`, `uint8` and `uint16` map to their own kinds instead + of widening to 32 bits; this lands before the Go SDK ships, so no stored + data uses the old mapping. `Int128` and `Uint128` are SDK value types. + `time.Time` means `Timestamp`, and `encrypt.Date{Year, Month, Day}` is a + date. A field whose EQL target is a date family accepts `time.Time`, + truncated to its UTC calendar day. +- **JavaScript.** A `BigInt` maps to the smallest kind that holds it, up to + 128 bits, and decodes as a `BigInt`. `Date` means `Timestamp`; a date has + its own wrapper. +- **Every binding** runs vitaminc's shared test vectors: a host value, the + kind it must map to, and the exact `[tag] ++ payload` leaf bytes, including + normalisation cases. The leaf is deterministic even though the AEAD is not. + +### EQL v4 + +- **v4 is the v3 SQL with Stack Encrypt terms.** One SQL source is emitted + under two names. The hand-written SQL takes the schema as a build-time + placeholder, as eql-codegen's templates already do. Consistent with + ADR-0001, the data-bearing domains (`eql_v3_*`, `eql_v4_*`) live in + `public` and survive reinstall, and the implementation schemas + (`eql_v3`, `eql_v4` and their `_internal` schemas) stay disposable. The + second name separates producers; it is not a versioned upgrade mechanism of + the kind ADR-0001 rejects. +- **A v4 payload's envelope carries `"v": 4`**, and the `eql_v4_*` domains' + check constraints test it, as the `eql_v3_*` ones test `3`. The version is + one more build-time placeholder. A payload written to the other producer's + domain then fails on insert, rather than only matching nothing when it is + queried. +- **New EQL targets are v4-only.** In v3 they stay refused, with a reason + that points to v4. +- **`@cipherstash/eql` 4.x ships both bundles** from `main`. A SQL fix lands in + both names in one release. +- **`stash eql install --eql-version 3|4|all`** chooses the bundle and defaults + to 3, which is today's behaviour. The default changes to 4, announced in + advance, when the TypeScript stack has moved to Stack Encrypt. +- **During the overlap**, expected to last a quarter or more, cipherstash-client + takes fixes only. New kinds and domains are produced through Stack Encrypt + and v4. An exception is a decision written down in its issue. + +### Review + +Using the canonical order bytes as the PRF input is the simplest way to make +equality and ordering agree, and it is frozen once data is stored under it. +Dan Draper signs it off in writing before the `vitaminc-prf` change lands. + +## Consequences + +- **The release order is fixed.** ore.rs (`orderable-bytes`) and cllw-ore + first, then vitaminc 0.6.0, then the Stack Encrypt breaking release. Stack + Encrypt must reach crates.io before `eql-bindings` uses its new API, since + `cargo publish` builds `eql-bindings` against the registry. The EQL v4 + bundle and the Go SDK changes follow. +- **Adding a kind later is additive**, because `Value` and `ValueKind` are + `#[non_exhaustive]`. It still takes a tag, a PRF domain, an order encoding, + a codec in every binding, and test vectors, or an entry in the exceptions + list. +- **Stack Encrypt's terms change** for `-0.0` and negative-sign NaN floats, + for non-NFC text equality, and for `Bool` order terms, which are removed. + None is deployed, so the breaking release carries them without migration. +- **cipherstash-client's terms do not change.** Its fixed-length + `orderable-bytes` output, cllw-ore's typed impls and its block ORE padding + are all kept, so every stored v3 payload stays queryable. +- **Moving a column from v3 to v4 means re-encrypting it.** The ciphertext and + every term change. How that migration runs belongs to the decision that + moves the TypeScript stack onto Stack Encrypt. +- **The CLI surface changes** (`--eql-version`), so `skills/stash-cli`, + `skills/stash-indexing` and `skills/stash-postgres` change in the same PR. +- **This replaces a definition in a plan.** `docs/plans/2026-10-04-plan-builder.md` + calls "EQL v4" a name for a Stack Encrypt payload in the v3 envelope and + the `eql_v3_*` domains. That plan is updated to this ADR. + +## Deferred + +- **Block ORE for text** waits for ore-rs's chained, variable-length block ORE + to be reviewed and released, then arrives in a minor release of + `vitaminc-ore`. Until then it is an entry in the exceptions list. +- **Block ORE for `Bool`.** Block ORE stored as right ciphertexts only is + fully randomised and semantically secure, so a two-value domain leaks + nothing through it. `Bool` could support that scheme alone, enforced by a + marker trait on schemes. Not built until needed. +- **Locale-aware collation**, as its own domain with the collation version in + its name. +- **ASCII-packed text domains**, until a customer's column sizes make the + case. diff --git a/docs/plans/2026-10-04-plan-builder.md b/docs/plans/2026-10-04-plan-builder.md index 88dd628fb..e70ea3b11 100644 --- a/docs/plans/2026-10-04-plan-builder.md +++ b/docs/plans/2026-10-04-plan-builder.md @@ -662,9 +662,11 @@ In the `Text` family, the three `Ord` suffixes carry equality too. Each type has a query type, with `Query` after its name: `TextEqQuery`. The value of `encrypt_into` is the Go type name. -An EQL value has the EQL v3 envelope: its version field is `3`, and Postgres stores it in an `eql_v3` domain. +An EQL value from the engine is an EQL v4 value: its version field is `4`, and Postgres stores it in an `eql_v4_*` domain. The ciphertext inside it is a Stack Encrypt ciphertext, which starts with `stack-encrypt:1:`. -"EQL v4" in this plan is the name of that form, and not a new envelope. +EQL v4 is the v3 SQL, emitted from the same source under a second name, so that a column written by Stack Encrypt and one written by cipherstash-client are different Postgres types ([ADR-0002](../adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md)). +This replaces this plan's first definition, under which "EQL v4" named a Stack Encrypt payload in the v3 envelope and an `eql_v3` domain; Postgres could not tell the two producers apart, and a query from one against a column written by the other matched nothing. +The `TextEq` described below is still produced into an `eql_v3` domain until that move lands. The engine produces one EQL type today: `TextEq`. Status (2026-10-06, #1062): `TextEq` is producible through the data plan's target form. @@ -1157,9 +1159,10 @@ model rather than a strain: it puts key material in the database. ## EQL v4 types as field targets -Naming: **EQL v4** is the EQL form of a stack-encrypt payload. **EQL v3** is -the existing SQL bundle and its `eql_v3_*` domains, which this section does -not change. Depends on #971 (`TextEq` / `TextEqQuery` through stack-encrypt +Naming: **EQL v4** is the EQL form of a stack-encrypt payload: the v3 SQL +emitted under a second name, with `eql_v4_*` domains and `"v": 4` envelopes +(ADR-0002). **EQL v3** is the same SQL holding cipherstash-client payloads in +its `eql_v3_*` domains, which this section does not change. Depends on #971 (`TextEq` / `TextEqQuery` through stack-encrypt in `eql-bindings`, and `Identifier` as a two-segment `Label`). **The engine returns an EQL type only when a plan names it as a target.** From e9eb1a9329c2ef323595897664ac5789bada97f4 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 11 Oct 2026 17:01:10 +1100 Subject: [PATCH 2/4] docs(plans): put vitaminc at the bottom of the order and equality term stack Starting vitaminc-ore under ADR-0002 surfaced four problems: order encodings, scheme traits and term derivation spread over three repositories; scheme crates that accept raw, hand-encoded bytes; vitaminc-prf reaching orderable-bytes through aead-value's canonical module; and one canonical form shared by equality and order, which rules out leakage-tuned domains such as an exact timestamp for equality with day resolution for order. The plan settles the replacement design: vitaminc-ore owns the order encodings and an OrderScheme trait with a sealed input, and ore-rs and cllw-ore implement it; each term crate applies a semantic baseline and per-domain transforms named in its labels; PRF has no ordering dependency; orderable-bytes is frozen. Exhaustive matching over Value is left open, with both approaches and their shared prerequisite laid out. It includes current and target dependency diagrams and the steps, and ADR-0002 now points to it. Claude-Session: https://claude.ai/code/session_012zHhtTEo968WazExKxA5oT --- ...-three-encodings-and-eql-v4-by-producer.md | 9 + ...10-11-order-and-equality-terms-layering.md | 331 ++++++++++++++++++ 2 files changed, 340 insertions(+) create mode 100644 docs/plans/2026-10-11-order-and-equality-terms-layering.md diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md index b5f858b10..33a20ee88 100644 --- a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -6,6 +6,15 @@ relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 # One value model, three encodings in vitaminc, and EQL v4 separated by producer +> **Amended 2026-10-11** by +> [`docs/plans/2026-10-11-order-and-equality-terms-layering.md`](../plans/2026-10-11-order-and-equality-terms-layering.md). +> Where the two disagree, the plan wins. In short: vitaminc sits at the +> bottom of the stack and the ORE schemes implement `vitaminc-ore`'s trait; +> equality and order terms take their own per-domain transforms instead of +> one shared canonical form; `vitaminc-prf` no longer depends on order +> encodings; and `orderable-bytes` is frozen. The plan lists every statement +> below that it replaces. + Every value Stack Encrypt handles is encoded three times: as a **ciphertext** (reversible and self-describing), as an **equality term** (a keyed hash that must be unambiguous) and as an **order term** (bytes whose order is the diff --git a/docs/plans/2026-10-11-order-and-equality-terms-layering.md b/docs/plans/2026-10-11-order-and-equality-terms-layering.md new file mode 100644 index 000000000..54f3d6947 --- /dev/null +++ b/docs/plans/2026-10-11-order-and-equality-terms-layering.md @@ -0,0 +1,331 @@ +# Order and equality terms: vitaminc at the bottom + +> **Plan, not specification.** This document is an indicative sketch of the +> steps required, written before the work was done. It is not kept in step +> with the implementation and must not be used as a formal specification or +> as a reference for reviewing what was actually built: the code, its rustdoc +> and the tests are the source of truth. Where the two disagree, the code wins +> and this document is simply out of date. + +**Status:** planned. Decisions settled 2026-10-11, except the one listed +under "Open decision". +**Date:** 2026-10-11 +**Amends:** ADR-0002 (`docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md`). +Where the two disagree, this document wins. +**Issues:** cipherstash/vitaminc#374 (`vitaminc-ore`), #1063 (index crates) + +## Why + +Starting on cipherstash/vitaminc#374 under ADR-0002 turned up four problems. + +- **Too many dependencies across repositories.** `vitaminc-ore` was to depend + on cllw-ore and ore-rs and adapt each to its `Scheme` trait, with + `orderable-bytes` underneath both. Order encodings, scheme traits and term + derivation ended up spread over three repositories, and every change needed + releases in all of them. +- **Raw bytes reach the schemes.** cllw-ore and ore-rs accept bytes that the + caller has already encoded. Bytes encoded by hand (little-endian integers, + an unflipped sign bit, `f64::to_bits`) encrypt without error and sort + wrong. Fixing that inside cllw-ore needed wrapper types in `orderable-bytes` + (cipherstash/ore.rs#103) and a feature split in cllw-ore + (cipherstash/cipherstash-suite#2308). Cargo features unify across a build, + so the feature split could not hold whenever anything else in the build + turned the feature on. +- **PRF depends on ordering.** `vitaminc-prf` reaches `orderable-bytes` + through `vitaminc-aead-value`'s `canonical` module, which reuses order + encodings as equality input. Equality needs a canonical form, not an order. +- **One canonical form for both term kinds is too rigid.** ADR-0002 builds + equality and order terms from the same canonical form. A leakage-tuned + domain needs them to differ, for example an exact timestamp for equality + and only the day for order. + +## Decisions + +### vitaminc is the bottom of the stack + +- `vitaminc-ore` owns: + - the order encodings; + - an `OrderScheme` trait; + - the derivation of order terms from `&Value`, generic over the scheme. + + It depends on no scheme crate. +- A scheme's input is a sealed type, `OrderPlaintext`, that only + `vitaminc-ore` can construct. It comes in fixed-length and variable-length + forms. A scheme can't be given bytes that skipped the encoding, so no + wrapper types or feature flags are needed to keep raw input out. +- `OrderScheme` returns a scheme-defined term, roughly + `type Term: Ord + AsRef<[u8]>`, whose length is only known at runtime. Each + scheme keeps its own comparison (CLLW ORE and block ORE don't compare byte + by byte) and its own stored bytes. Fixed-width types such as + `OreCllw8V1` stay as inherent APIs on the scheme crates, because stable + Rust can't name their width generically. +- The trait also covers the key type, domain separation (the term's domain + label goes in as the salt) and typed errors. +- cllw-ore and ore-rs depend on `vitaminc-ore` and implement `OrderScheme` + behind an optional `vitaminc` feature, so cipherstash-client doesn't take a + vitaminc dependency. Their existing typed APIs stay as they are, so + cipherstash-client's stored terms don't change. +- `vitaminc-ore` releases in lockstep with the other vitaminc crates. On each + vitaminc minor release, cllw-ore and ore-rs bump their optional + `vitaminc-ore` dependency before stack can upgrade. That is acceptable + while stack is the only user of the trait path. +- `vitaminc-ore`'s tests use a mock scheme defined inside the crate. A + dev-dependency on cllw-ore would form a cycle across repositories and pull + a second `vitaminc-ore` in from crates.io. + +### Equality and order are tuned per domain + +- There is no shared canonical form. Each kind has a **semantic baseline**, + which makes values that are equal under `Value`'s own semantics encode the + same: decimal scale (`1.0` = `1.00`), signed zero, NaN, NFC. +- On top of the baseline, each term's domain chooses its own **lossy + transforms**, such as timestamp resolution (exact, microsecond, day), + text folding (removing combining marks, case folding, with the Unicode + version pinned) and prefix truncation. +- Each transform is named in its term's domain label, for example + `…/timestamp/day/v1`, so terms built with different transforms can never + be compared. +- Equality and order can be **independent** (an exact timestamp for + equality, the day for order) or **locked** (the same transform value + passed to both). Stack's index spec (#1063) carries the choice per column. +- The semantic baseline lives in each term crate: `vitaminc-prf` and + `vitaminc-ore` each apply it. A conformance test checks that values equal + under `Value` semantics produce equal terms in both. There is no shared + canonicalisation crate. +- A column's leakage is the combined leakage of all its terms. Truncating the + order term to the day hides the order of events within a day, but an exact + equality term on the same column still reveals which rows share a + timestamp. Domain design must account for the combination. + +### PRF has no ordering dependency + +- `vitaminc-prf` depends only on `vitaminc-aead-value` and applies its own + baseline and transforms. +- `vitaminc-aead-value` does no term normalisation. Its `canonical` module and + its dependency on `orderable-bytes` are removed. +- The value-derived PRF domains (cipherstash/vitaminc#380) haven't been + released: they landed on 2026-10-08, after vitaminc 0.5.1. So `vitaminc-prf` + can choose its own canonical bytes for dates, timestamps and decimals + without affecting any stored term. + +### `orderable-bytes` is frozen + +- `orderable-bytes` stays as it is for cipherstash-client's stored terms. It + is not part of the target design. +- There is no 0.3. cipherstash/ore.rs#103 (binary wrapper types) is closed. +- 0.2.0 is not yanked: ore-rs 0.8.4 depends on it and nothing new needs it + gone. + +## Open decision: exhaustive matching over `Value` + +Once `canonical.rs` leaves `vitaminc-aead-value`, `vitaminc-prf` and +`vitaminc-ore` must each handle every kind of `Value`, and adding a kind must +fail to compile until each one does. Two approaches are on the table. + +### Prerequisite for either approach: variants that exist in every build + +`Value::Date` and `Value::Timestamp` exist only with `vitaminc-aead-value`'s +`chrono` feature, and `Value::Decimal` only with its `rust_decimal` feature. +Because features unify across a build, an exhaustive match outside +`vitaminc-aead-value` breaks: + +1. `vitaminc-prf` matches `Value` exhaustively, with its `Date` arm behind its + own `chrono` feature. +2. In a build where another crate turns on `vitaminc-aead-value/chrono` and + `vitaminc-prf`'s `chrono` is off, `Value::Date` exists and `vitaminc-prf` + has no arm for it. `vitaminc-prf` fails to compile in that build. + +A visitor fails the same way: a required `visit_date` that exists only with +`chrono` is a method an implementor can't know whether to write. A crate can +check its own features, never a dependency's. Today's design avoids this only +because the match sits inside `vitaminc-aead-value`, which checks its own +features. + +The fix is to make every variant exist in every build, using vitaminc's own +representation, with conversions to and from chrono and rust_decimal behind +the features: + +| Variant | Representation | +|---|---| +| `Date` | days from the start of the common era, `i32` | +| `Timestamp` | Unix seconds `i64`, nanoseconds `u32` | +| `Decimal` | mantissa and scale | + +The ciphertext wire format already uses these payloads (tags `0x12`–`0x14` +are independent of chrono), so stored ciphertexts don't change. The Rust +types do, which is a breaking change to `Value`. + +### The two approaches + +**A: drop `#[non_exhaustive]`.** `vitaminc-prf`, `vitaminc-ore` and the +bindings match `Value` exhaustively in their own crates. + +**B: a visitor trait, keeping `#[non_exhaustive]`.** `vitaminc-aead-value` +defines a `ValueVisitor` with one required method per kind and does the +dispatch. Crates that must handle every kind implement it. Other crates keep +matching with a `_ =>` arm. + +| | A: drop `#[non_exhaustive]` | B: visitor, keep `#[non_exhaustive]` | +|---|---|---| +| Who gets exhaustiveness | Every crate that matches `Value` | Crates that implement the visitor | +| Adding a kind | Breaking for every crate that matches | Breaking for visitor implementors, additive for others | +| Code to maintain | Nothing new | One trait, one dispatch function, a method per kind in each implementor | +| Grouping | Each match writes out every integer width | The trait can offer grouped hooks, such as one integer method that the per-width methods default to | +| Changing `Value`'s representation | Breaks every match | Hidden behind the method signatures | +| Main risk | Low | A default on a per-kind method lets a new kind compile without being handled. Per-kind methods need no defaults; only grouping hooks may have them. | + +**Evidence:** +- Outside `vitaminc-aead-value`, `Value` is matched in `vitaminc-prf` (through + `canonical`), in `aead-napi`'s `convert.rs` and `scalar.rs`, and in the hmac + tests. +- `aead-napi` already has `_ => Err("unsupported value kind")` + (`convert.rs`). That is the silent refusal `#[non_exhaustive]` invites. +- ADR-0002 requires every binding to handle every kind, so the bindings want + exhaustiveness as well. +- Under A they get it automatically. Under B each binding opts in by + implementing the visitor. + +**Leaning:** B. The decision waits on a closer look at the `Value` enum's +design, which may change the representation in ways that favour one +approach. + +## Current state + +Arrows point from a crate to what it depends on. cipherstash-client and +`vitaminc-protected` are left out. + +```mermaid +flowchart TB + subgraph stack + SE[stack-encrypt] + end + subgraph suite["cipherstash-suite"] + CLLW["cllw-ore 0.5
typed impls, own encodings"] + end + subgraph vitaminc + PRF[vitaminc-prf] + AV["vitaminc-aead-value
Value, canonical module"] + end + subgraph orers["ore.rs"] + ORERS["ore-rs 0.8"] + OB["orderable-bytes"] + end + SE --> PRF + SE --> AV + SE --> CLLW + PRF -- "canonical feature" --> AV + AV -- "git pin, pre-0.2" --> OB + CLLW -- "0.1, chrono and decimal" --> OB + ORERS -- "0.2" --> OB +``` + +- stack-encrypt calls cllw-ore directly for order terms. There is no block + ORE path. +- Equality bytes for dates, timestamps and decimals come from order + encodings, through `vitaminc-aead-value`. +- `orderable-bytes` is reached three ways, at three different versions. + +## Target state + +```mermaid +flowchart TB + subgraph stack + EQI["Equality index
picks transforms"] + ORI["Ore / Ope index
picks scheme and transforms"] + end + subgraph schemes["scheme crates"] + ORERS["ore-rs
block ORE"] + CLLW["cllw-ore
CLLW ORE and OPE"] + end + subgraph vitaminc + PRF["vitaminc-prf
equality terms, own baseline and transforms"] + VORE["vitaminc-ore
order encodings, transforms,
OrderScheme, sealed OrderPlaintext"] + AV["vitaminc-aead-value
Value"] + end + EQI --> PRF + ORI --> VORE + ORI --> ORERS + ORI --> CLLW + ORERS -- "impl OrderScheme" --> VORE + CLLW -- "impl OrderScheme" --> VORE + PRF --> AV + VORE --> AV +``` + +- vitaminc depends on nothing in the other repositories. +- The scheme crates implement `vitaminc-ore`'s trait. The index crates choose + a concrete scheme and the transforms for each column. +- `orderable-bytes` doesn't appear. It stays frozen for cipherstash-client. + +## Changes to ADR-0002 + +| ADR-0002 says | Now | +|---|---| +| "Equality and order terms are computed from one canonical form per kind, and both layers use the same one" | Each kind has a semantic baseline; lossy transforms are chosen per domain, independently or locked | +| Canonical forms table: timestamp truncated to microseconds, text order folded | These become per-domain transforms, named in each domain's label | +| Order: "The plaintext layer is `orderable-bytes`, which … gains a variable-length encoding" | `vitaminc-ore` owns the order encodings; `orderable-bytes` is frozen | +| "A `Scheme` trait, with block ORE (over ore-rs), CLLW ORE and CLLW OPE (over cllw-ore)" | `OrderScheme` lives in `vitaminc-ore`; ore-rs and cllw-ore implement it | +| "cllw-ore keeps its typed impls … behind a cargo feature that cipherstash-client enables, and gains a bytes-level entry point" | cllw-ore's typed API is unchanged and not feature-gated; it gains an `OrderScheme` impl behind an optional `vitaminc` feature | +| Review: "Using the canonical order bytes as the PRF input … Dan Draper signs it off" | PRF no longer uses order bytes; the review item applies only to domains that lock equality and order together | +| `Value` and `ValueKind` become `#[non_exhaustive]` | Open: see "Open decision" | +| Release order: "ore.rs (`orderable-bytes`) and cllw-ore first, then vitaminc 0.6.0" | vitaminc first, then cllw-ore and ore-rs, then stack | + +## Steps + +### Phase 1: tidy up work in flight + +1. Close cipherstash/ore.rs#103. Don't yank `orderable-bytes` 0.2.0. +2. Cut cipherstash/cipherstash-suite#2308 down to its first two commits: + known-answer tests that pin every cllw-ore encrypt path, and wiping the + plaintext bit buffer in `encrypt_ope_bits`. It becomes a cllw-ore patch + release. The known-answer tests then protect cipherstash-client's terms + through the rest of this work. + +### Phase 2: vitaminc + +3. Settle the open decision. If the `Value` changes are accepted, make every + variant exist in every build. +4. `vitaminc-aead-value` and `vitaminc-prf`: + - add the chosen exhaustiveness mechanism; + - remove `canonical.rs` and the `orderable-bytes` dependency from + `vitaminc-aead-value`; + - move the equality baseline and transforms into `vitaminc-prf`, with each + transform named in its domain label. +5. New crate `vitaminc-ore` (rewrite cipherstash/vitaminc#374 to match): + - `OrderScheme` and the sealed `OrderPlaintext`; + - order encodings ported from `orderable-bytes`, kept in `Protected`, + pinned by golden vectors, and checked against `orderable-bytes` where + they match byte for byte; + - per-domain transforms: timestamp resolution, text folding with a pinned + Unicode version, room for prefix truncation; + - order-term derivation from `&Value`, generic over the scheme; + - a mock scheme for tests; + - docs that state the combined-leakage rule. +6. A conformance test across `vitaminc-prf` and `vitaminc-ore`: every kind is + handled or listed as an exception, and values equal under `Value` + semantics produce equal terms. +7. Release vitaminc 0.6.0. + +### Phase 3: scheme crates + +8. cllw-ore: implement `OrderScheme` behind an optional `vitaminc` feature, + with known-answer tests for the trait path. This is an additive release. +9. ore-rs: the same. Block ORE uses each kind's natural width; + cipherstash-client keeps its 8-byte padding through the existing API. + Block ORE for text waits for cipherstash/ore.rs#95 and goes on the + exceptions list. +10. Release both. + +### Phase 4: stack + +11. Index crates (#1063): each index spec carries its scheme and transforms. + The Equality index calls `vitaminc-prf`; the Ore and Ope index builds the + concrete scheme and calls `vitaminc-ore`. stack-encrypt drops its direct + use of cllw-ore. +12. EQL v4 domain labels name their transforms. + +### Later + +- Block ORE for text, once cipherstash/ore.rs#95 is released. +- Retire cllw-ore's typed impls and `orderable-bytes` once cipherstash-client + is retired. From a0cfd86585416f390401d8ee9c393ac539424177 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Mon, 12 Oct 2026 19:40:46 +1100 Subject: [PATCH 3/4] docs(adr): correct ADR-0002 on what EQL v4's separate names enforce The ADR said Postgres refuses to compare an eql_v4 query term with an eql_v3 column at plan time. It does not. EQL's domains are over jsonb, and with no operator defined for the exact pair Postgres resolves `=` as the built-in jsonb = jsonb, so the query returns no rows silently, as a mismatched pair within v3 already does. Verified on Postgres 17 with both bundles from #1141 installed in one database. The decision stands: separate names still reject a wrong-version insert, make the function form ambiguous, and give typed clients distinct types. Consequences now says where the protection stops, and Deferred names the cross-version blocker operators that would close the gap (#1160). --- ...-three-encodings-and-eql-v4-by-producer.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md index 33a20ee88..d3c2f8ebe 100644 --- a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -76,9 +76,10 @@ the producer of the terms differs. hand-written SQL. 3. **One SQL source, emitted under two names.** The build writes the same source out as `eql_v3` (cipherstash-client terms) and `eql_v4` (Stack - Encrypt terms). A column's domain names its producer, and Postgres refuses - to compare an `eql_v4_*` query term with an `eql_v3_*` column at plan time. - Chosen. + Encrypt terms). A column's domain names its producer: a payload written + to the other producer's domain fails its check on insert, and a typed + client sees two different types. Names alone do not stop a cross-producer + `=` in SQL; see Consequences. Chosen. **How v3 and v4 ship.** Bumping `main` to v4 and patching v3 from a branch needs a maintenance release path for five lockstep artefacts that does not @@ -219,7 +220,7 @@ without an entry. check constraints test it, as the `eql_v3_*` ones test `3`. The version is one more build-time placeholder. A payload written to the other producer's domain then fails on insert, rather than only matching nothing when it is - queried. + queried. Queries are a different matter; see Consequences. - **New EQL targets are v4-only.** In v3 they stay refused, with a reason that points to v4. - **`@cipherstash/eql` 4.x ships both bundles** from `main`. A SQL fix lands in @@ -254,6 +255,17 @@ Dan Draper signs it off in writing before the `vitaminc-prf` change lands. - **cipherstash-client's terms do not change.** Its fixed-length `orderable-bytes` output, cllw-ore's typed impls and its block ORE padding are all kept, so every stored v3 payload stays queryable. +- **Separate names do not stop a cross-producer comparison in SQL.** EQL's + domains are over `jsonb`, and when no operator is defined for an exact pair + of types, Postgres treats a domain as its base type. `v3col = v4term` then + runs as the built-in `jsonb = jsonb` and returns no rows, silently, as + `eql_v3_text_eq = eql_v3.query_integer_eq` already does within v3. A join + between a v3 and a v4 column does the same, and + `eql_v3.eq_term(v3col) = eql_v4.eq_term(v4term)` compares the two terms as + text. Three things do fail loudly: an insert of the wrong version (the + domain check), the function form `eql_v3.eq(v3col, v4term)` ("not unique"), + and `OPERATOR(public.=)` between a v3 and a v4 column ("not unique"). + Verified on Postgres 17 with both bundles installed in one database. - **Moving a column from v3 to v4 means re-encrypting it.** The ciphertext and every term change. How that migration runs belongs to the decision that moves the TypeScript stack onto Stack Encrypt. @@ -272,6 +284,12 @@ Dan Draper signs it off in writing before the `vitaminc-prf` change lands. fully randomised and semantically secure, so a two-value domain leaks nothing through it. `Bool` could support that scheme alone, enforced by a marker trait on schemes. Not built until needed. +- **Cross-version blocker operators.** Blockers for every v3/v4 pair of the + same type family and capability, created by whichever bundle is installed + second, would turn the silent cross-producer `=` into an error. About 2,300 + operators, generated from the catalog like the existing blockers. Not built + yet. Until then, a cross-producer query is caught only by typed clients and + the function form. - **Locale-aware collation**, as its own domain with the collation version in its name. - **ASCII-packed text domains**, until a customer's column sizes make the From 19fabd5cceab7d66d5040b8775d31b4781cee2bf Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Mon, 12 Oct 2026 22:42:34 +1100 Subject: [PATCH 4/4] docs(adr): fold the layering plan into ADR-0002 and answer its review The ADR deferred to a plan whose own banner disclaims being a specification, so an implementer had no authoritative text for the canonical forms, the order-scheme layering or the release order. The amended decisions now live in the ADR body, with exhaustive handling of Value recorded as the one open decision. Review fixes: pin the timestamp microsecond rounding to Postgres's (round half to even, checked on 17.5) and the text fold's step order; let unassigned code points pass through instead of refusing writes; forbid lossy order transforms on _ord domains that answer = with the order term; choose a binding's kind from the declared EQL target, with a kind-to-family table, and take a Go date in the value's own location; copy the exact leaf byte layouts from tags.rs; say why float equality is now acceptable and that text order terms change; keep cllw-ore's Bool impl; reintroduce --eql-version deliberately on install and migration; list the eql_v3_-prefix detectors, installer schema lists and single-bundle release tooling v4 needs; and name the owner of the deferred cross-version blockers. --- ...-three-encodings-and-eql-v4-by-producer.md | 422 +++++++++++++----- docs/plans/2026-10-04-plan-builder.md | 2 +- ...10-11-order-and-equality-terms-layering.md | 18 +- 3 files changed, 325 insertions(+), 117 deletions(-) diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md index d3c2f8ebe..d2b21d3ae 100644 --- a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -1,32 +1,34 @@ --- status: accepted date: 2026-10-08 +revised: 2026-10-12 relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 --- # One value model, three encodings in vitaminc, and EQL v4 separated by producer -> **Amended 2026-10-11** by -> [`docs/plans/2026-10-11-order-and-equality-terms-layering.md`](../plans/2026-10-11-order-and-equality-terms-layering.md). -> Where the two disagree, the plan wins. In short: vitaminc sits at the -> bottom of the stack and the ORE schemes implement `vitaminc-ore`'s trait; -> equality and order terms take their own per-domain transforms instead of -> one shared canonical form; `vitaminc-prf` no longer depends on order -> encodings; and `orderable-bytes` is frozen. The plan lists every statement -> below that it replaces. +> **Revised 2026-10-12.** This ADR now carries the decisions from +> [`docs/plans/2026-10-11-order-and-equality-terms-layering.md`](../plans/2026-10-11-order-and-equality-terms-layering.md): +> vitaminc sits at the bottom of the stack, equality and order terms take +> per-domain transforms over a shared semantic baseline, `vitaminc-prf` no +> longer depends on order encodings, and `orderable-bytes` is frozen. The plan +> keeps the steps; this document is the decision. One question is still open, +> under "Open decision". Every value Stack Encrypt handles is encoded three times: as a **ciphertext** (reversible and self-describing), as an **equality term** (a keyed hash that must be unambiguous) and as an **order term** (bytes whose order is the value's order). This ADR decides where each encoding lives, which kinds of -value exist, the canonical form each kind takes, and how EQL keeps columns -written by Stack Encrypt apart from columns written by cipherstash-client. +value exist, how each kind is normalised before a term is derived from it, +and how EQL keeps columns written by Stack Encrypt apart from columns written +by cipherstash-client. ## The problem In October 2026 Stack Encrypt could produce one of EQL's 51 types, `TextEq`. -The number, date, timestamp and boolean families were blocked on encoding, -not code: +The number, date, timestamp and decimal families were blocked on encoding, +not code. (`Bool` was not: it already had a kind, and EQL's only boolean +type, `eql_v3_boolean`, stores a ciphertext and nothing else.) - **The three encodings disagreed about which types exist.** `i16` had an equality domain in `vitaminc-prf` and an order encoding, but no ciphertext @@ -40,11 +42,20 @@ not code: order-preserving encoding per type. cllw-ore uses it for chrono and decimal but hand-rolls integers and floats, and differs on `-0.0`. Block ORE was not derivable at all, so `TextOrdOre` and `TextSearchOre` were refused. +- **Order encodings were spread over three repositories.** Encodings lived in + ore.rs, schemes in ore.rs and cipherstash-suite, and term derivation in + vitaminc and Stack Encrypt. Every change needed releases in all of them, + and the scheme crates accepted raw bytes, so a hand-encoded integer + (little-endian, sign bit not flipped) encrypted without error and sorted + wrong. - **The existing writer's encodings are inconsistent.** cipherstash-client hashes a timestamp's milliseconds but orders by its nanoseconds, and hashes a decimal with its scale while ordering ignores it. Its block ORE text is ASCII-only, lowercased, maps every digit to one symbol and truncates to six - blocks. cllw-ore's text decomposes to NFD and strips accents. + blocks. cllw-ore's text order form (`orderize_string`) decomposes to NFD and + then drops every character that is not alphanumeric, whitespace or ASCII + punctuation, which removes symbols, emoji and non-ASCII punctuation as well + as accents, and does not fold case. - **Two producers wrote the same EQL domains.** A Stack Encrypt query term never matches a cipherstash-client term for the same value, and the `eql_v3_*` domains could not tell them apart. A query from one against a @@ -61,9 +72,20 @@ not code: vitaminc depend on Stack Encrypt (which depends on six vitaminc crates), or forks the frozen tag table into two copies. 2. **Keep it in vitaminc, and move term derivation there too.** All three - encodings then live in one repository, and one exhaustive `match` on the - value type per layer makes "a kind exists in every layer or in none" a - compile error rather than a convention. Chosen. + encodings then live in one repository, next to the value type they + encode. Chosen. How "a kind exists in every layer or in none" is enforced + is the open decision below; until it is settled, the conformance test + described under "Which kinds get which terms" is what enforces it. + +**Where order schemes sit.** + +1. **`vitaminc-ore` adapts the scheme crates.** It depends on cllw-ore and + ore-rs and wraps each in its own trait, with `orderable-bytes` underneath. + This is the spread over three repositories described above. +2. **vitaminc at the bottom.** `vitaminc-ore` owns the order encodings and an + `OrderScheme` trait and depends on no scheme crate; cllw-ore and ore-rs + implement the trait. A scheme's input is a sealed type only `vitaminc-ore` + can build, so raw bytes cannot reach a scheme. Chosen. **How EQL separates the two producers.** The SQL for both is identical; only the producer of the terms differs. @@ -86,7 +108,8 @@ needs a maintenance release path for five lockstep artefacts that does not exist, and the EQL publish script puts every stable release on `latest`, so a 3.x patch would move `latest` backwards. A second package doubles the trusted-publishing and release surface. One package carrying both bundles -needs neither. Chosen. +avoids both, at the cost of teaching the release tooling about a second +bundle (see Consequences). Chosen. ## Decision @@ -97,95 +120,177 @@ needs neither. Chosen. a `Value` variant and a ciphertext tag, in one breaking release. Rust `i128` and `u128` get `Encrypt` and `Decrypt` impls. - `FfiValue` is renamed `Value`, with `#[deprecated] pub type FfiValue = Value` - for one release. `Value` and `ValueKind` become `#[non_exhaustive]`, so later - kinds are additive. + for one release. Whether `Value` and `ValueKind` are `#[non_exhaustive]` is + the open decision. +- Every variant exists in every build. `Date`, `Timestamp` and `Decimal` use + vitaminc's own representation (the payloads in the table below), with + conversions to and from chrono and rust_decimal behind those features. A + variant that appeared only with a feature would break any exhaustive + handling outside `vitaminc-aead-value`, because features unify across a + build and a crate can check only its own. - `Value` implements `Clone` as a deep copy that rebuilds every leaf into a fresh `Protected`. A clone is under the same custody as its original. - The sealed leaf tag table (`tags.rs`) stays frozen and contiguous. The transport codec's framing tags move from `0x10`–`0x12` to `0xF0`–`0xF2`; transport carries no compatibility commitment and every user of it updates - together. The new leaves: + together. A leaf is `[tag] ++ payload`. The new leaves, which every binding + must produce byte for byte: | Tag | Kind | Payload | |---|---|---| - | `0x0C`–`0x11` | `Int8`, `UInt8`, `Int16`, `UInt16`, `Int128`, `UInt128` | 1, 1, 2, 2, 16, 16 bytes; two's complement for signed; little-endian | - | `0x12` | `Date` | `i32` days counted from 0001-01-01 (`num_days_from_ce`), little-endian | - | `0x13` | `Timestamp` | `i64` Unix seconds then `u32` nanoseconds, little-endian, UTC | - | `0x14` | `Decimal` | rust_decimal's 16-byte `serialize()`, which keeps the scale | + | `0x0C`, `0x0D` | `Int8`, `UInt8` | 1 byte; two's complement for `Int8` | + | `0x0E`, `0x0F` | `Int16`, `UInt16` | 2 bytes, little-endian; two's complement for `Int16` | + | `0x10`, `0x11` | `Int128`, `UInt128` | 16 bytes, little-endian; two's complement for `Int128` | + | `0x12` | `Date` | `i32`, little-endian: the day number in the proleptic Gregorian calendar with **0001-01-01 as day 1** (chrono's `num_days_from_ce`). 0001-01-01 is `01 00 00 00`, not zero | + | `0x13` | `Timestamp` | `i64` Unix seconds, then `u32` nanoseconds, both little-endian, UTC. Nanoseconds are below 1,000,000,000, except at a leap second (seconds modulo 60 is 59), where they may go up to 1,999,999,999 (chrono's representation). Anything else is refused | + | `0x14` | `Decimal` | 16 bytes: a little-endian `u32` of flags (sign in bit 31, scale in bits 16–23 and at most 28, every other bit zero), then the low, middle and high little-endian `u32` words of the 96-bit mantissa. This is rust_decimal's `serialize()` and keeps the scale. There is no NaN or infinity | - `transport` stays a module of `vitaminc-aead-value`. -### Canonical forms +### Semantic baseline and per-domain transforms The ciphertext keeps the value exactly as given: `1.50` decrypts as `1.50`, -and a timestamp keeps its nanoseconds. Equality and order terms are computed -from one canonical form per kind, and both layers use the same one, so -equality and ordering agree by construction. +and a timestamp keeps its nanoseconds. Terms are derived in two steps. -| Kind | Canonical form for terms | +**The semantic baseline** makes values that are equal under `Value`'s own +semantics encode the same. Every term applies it; it loses nothing a user +would call a different value. + +| Kind | Baseline | |---|---| -| `Timestamp` | truncated to microseconds, Postgres's precision | -| `Decimal` | scale normalised (`1`, `1.0` and `1.00` are equal). NaN and ±Infinity are refused at encode time; rust_decimal cannot represent them | -| `Float32`, `Float64` | `-0.0` folded into `+0.0`; every NaN replaced by one positive quiet NaN, which sorts above +Infinity. This matches Postgres | -| text, equality | NFC | -| text, order | NFC, then NFD, combining marks removed, Unicode default case folding | - -Text normalisation is pinned. The Unicode version is part of the encoding's -domain label (for example `text-nfc/unicode-16/v1`), the -`unicode-normalization` crate is pinned to it, and strings containing -unassigned code points are refused. Order terms fold accent and case because -code-point order puts `é` after `z`; a fixed, pinned fold approximates the -first level of Unicode collation without depending on ICU, whose sort keys -change between versions. Equality is not folded: a folded order term only -produces ties, while a folded equality term produces false matches. A -case-insensitive equality is its own domain. - -Truncation and alphabet packing are settings of an EQL domain, named in its -label, never part of the shared encoding. +| `Decimal` | scale normalised: `1`, `1.0` and `1.00` are equal | +| `Float32`, `Float64` | `-0.0` folded into `+0.0`; every NaN replaced by one positive quiet NaN, which sorts above +Infinity. This is Postgres's own `float8` semantics: `'NaN' = 'NaN'`, `'-0' = '0'` and `'NaN' > 'Infinity'` are all true | +| text | NFC | +| every other kind | none | + +**Lossy transforms** sit on top of the baseline, chosen by each term's +domain: timestamp resolution, text folding and prefix truncation. Each +transform and its version is named in the domain label, for example +`…/timestamp/day/v1`. The ones defined now: + +- **Timestamp resolution**: exact, microsecond or day. Microsecond rounds to + the nearest microsecond, ties to even, on the signed microsecond count + since the epoch. That is what Postgres does when it parses a timestamp with + more digits than it keeps (`.1234565` becomes `.123456` and `.1234575` + becomes `.123458`, before and after 1970; checked on Postgres 17.5). A + value that already has microsecond precision is unchanged. Day takes the + UTC calendar day. +- **Text fold, v1**: NFD, then full Unicode case folding (`CaseFolding.txt` + statuses C and F, so `ß` becomes `ss`), then NFD again, then removal of + every code point with General_Category `Mn`, then NFC. The order of the + steps is part of the transform, because folding can produce text that is + no longer normalised and the steps do not commute. +- **Prefix truncation**: a number of code points kept after any other + transform. Alphabet packing, if ever built, is a transform of the same + kind. + +Text transforms are pinned to a Unicode version, which is part of the label +(for example `text-fold/unicode-16/v1`), and the `unicode-normalization` and +case-folding tables are pinned to it. Code points unassigned in that version +pass through unchanged (canonical combining class 0, no decomposition, no +case mapping), so the transform is a fixed function and nothing is refused on +write. The cost is that two strings using characters added after the pin, +which a later Unicode would treat as equivalent, may not match. Moving the +pin is a new label: existing columns keep the label they were written under +and stay queryable, and a column moves only by re-encryption. + +Order terms fold case and accents because code-point order puts `é` after +`z`; a fixed fold approximates the first level of Unicode collation without +depending on ICU, whose sort keys change between versions. Equality on text +is not folded by default: a folded order term only produces ties, while a +folded equality term produces false matches. A case-insensitive equality is +its own domain. + +**Equality and order may take different transforms**, for example an exact +timestamp for equality and the day for order, or the same one ("locked"). +Stack's index spec carries the choice per column. Two rules bound it: + +- **A non-injective order transform needs an equality term beside it.** In + the EQL SQL, the numeric, date and timestamp `_ord`, `_ord_ope` and + `_ord_ore` domains store no `hm` and answer `=` with the order term + (`timestamp_ord_functions.sql`: `eq` is `ord_term(a) = ord_term(b)`). A + day-resolution order term there would make `=` match every row from that + day. So a lossy order transform is allowed only on a domain that also + stores an equality term and answers `=` through it, the shape `text_ord` + already has (`hm` and `op`). On the other domains the order transform is + injective: exact for numbers and dates, and for timestamps whatever + resolution the equality term would have used. +- **Labels separate domains; they do not make terms incomparable.** Terms + built with different transforms are unrelated bytes, so a probe built with + one matches nothing built with the other. Nothing in a stored term records + its label. Keeping a query on the same transforms as the column is the + index spec's job, and the EQL domain is what ties a column to its spec. + +A column's leakage is the combined leakage of all its terms. Truncating the +order term to the day hides the order of events within a day, but an exact +equality term on the same column still reveals which rows share a timestamp. ### Which kinds get which terms - **Every scalar kind has a ciphertext.** Containers and null have no terms. - **Equality:** every scalar kind except `Bool`, including floats over their - canonical bits. -- **Order:** every scalar kind except `Bool`, under all three schemes. + baseline bits. Stack Encrypt refused float equality until now, on the + grounds that equality on IEEE-754 values is a modelling error. The baseline + removes the two cases where bitwise equality and numeric equality disagree + (signed zero and NaN payloads), and what is left is exactly Postgres's + `float8 =`, which EQL's `real_eq` and `double_eq` domains already expose. + Whether a computed float is the value the caller meant is the caller's + problem, as it is for a plaintext float column. +- **Order:** every scalar kind except `Bool`, under all three schemes, with + one exception: block ORE for text, which waits for ore-rs's variable-length + block ORE (see Deferred). - **`Bool` has a ciphertext only.** A keyed hash or an order term over a domain of two values hides nothing: it splits the rows into two groups, and - an order term also says which group is `true`. The existing CLLW order terms - for `Bool` are removed. + an order term also says which group is `true`. Stack Encrypt stops + producing CLLW order terms for `Bool`, and `vitaminc-ore` has no `Bool` + encoding. cllw-ore keeps its `Bool` impl, so cipherstash-client and its + stored v3 terms are unaffected. These exceptions are an explicit, documented list kept next to vitaminc's -conformance test, which fails for any kind that is missing from a layer -without an entry. +conformance test. The test fails for any kind that is missing from +`vitaminc-prf` or `vitaminc-ore` without an entry, and checks that values +equal under `Value` semantics produce equal terms in both. ### Where each encoding lives -- **Ciphertext:** `vitaminc-aead-value`. -- **Equality:** `vitaminc-prf`, which gains domains for every new kind, floats - and `Decimal`, over the canonical bytes. -- **Order:** a new `vitaminc-ore` crate. - - The plaintext layer is `orderable-bytes`, which stays its own crate in - ore.rs and gains a variable-length encoding for text and bytes. Its - existing fixed-length output does not change. - - A `Scheme` trait, with block ORE (over ore-rs), CLLW ORE and CLLW OPE - (over cllw-ore). A scheme only encrypts the bytes the plaintext layer - produces, so every orderable kind works under every scheme. - - Block ORE uses each kind's natural width. The 8-byte padding is a - cipherstash-client detail kept for its stored terms. - - cllw-ore keeps its typed impls, unchanged, behind a cargo feature that - cipherstash-client enables, and gains a bytes-level entry point that - `vitaminc-ore` calls. -- **Term derivation** takes a `&Value` in `vitaminc-prf` and `vitaminc-ore`, - with one exhaustive match per layer that keeps leaves inside `Protected`. A - pairing a layer does not support returns a typed error from vitaminc. +- **Ciphertext:** `vitaminc-aead-value`. It does no term normalisation: its + `canonical` module and its dependency on `orderable-bytes` are removed. +- **Equality:** `vitaminc-prf`, which depends only on `vitaminc-aead-value`, + gains domains for every new kind, floats and `Decimal`, and applies the + baseline and its own transforms. It has no ordering dependency. The + value-derived PRF domains landed after vitaminc 0.5.1 and are unreleased, + so their bytes are still free to choose. +- **Order:** a new `vitaminc-ore` crate, released in lockstep with the other + vitaminc crates. + - It owns the order encodings (ported from `orderable-bytes`, kept in + `Protected`, pinned by golden vectors), the baseline, the order + transforms, and derivation of order terms from `&Value`. + - Its `OrderScheme` trait takes a sealed `OrderPlaintext`, fixed-length or + variable-length, that only `vitaminc-ore` can build, and returns a + scheme-defined term, roughly `type Term: Ord + AsRef<[u8]>`. Each scheme + keeps its own comparison and stored bytes. The trait also covers the key + type, domain separation (the label goes in as the salt) and typed errors. + - It depends on no scheme crate; its tests use a mock scheme. +- **Schemes:** cllw-ore (CLLW ORE and OPE) and ore-rs (block ORE) implement + `OrderScheme` behind an optional `vitaminc` feature, so cipherstash-client + takes no vitaminc dependency. Their typed APIs are unchanged and not + feature-gated. Block ORE through the trait uses each kind's natural width; + the 8-byte padding stays a detail of the typed API cipherstash-client uses. +- **`orderable-bytes` is frozen.** It stays as it is for cipherstash-client's + stored terms and is not part of the target design. +- **Term derivation** takes a `&Value` in `vitaminc-prf` and `vitaminc-ore` + and keeps leaves inside `Protected`. A pairing a layer does not support + returns a typed error from vitaminc. ### Stack Encrypt - `dynamic::Value` and `dynamic::term::Scalar` are deleted. `dynamic::term` takes a `&vitaminc_aead_value::Value`. `admits` keeps only the rules that belong to Stack Encrypt, such as `Match` taking text only. -- Order terms go through `vitaminc-ore`, and a block ORE term kind joins CLLW - ORE and OPE. Stack Encrypt no longer depends on cllw-ore directly. +- Order terms go through `vitaminc-ore`. The index for an ORE or OPE column + picks the concrete scheme (cllw-ore or ore-rs, through `OrderScheme`) and + its transforms; Stack Encrypt no longer calls cllw-ore's typed API. A block + ORE term kind joins CLLW ORE and OPE. - `TextEq` normalises to NFC and moves to EQL v4. The Stack Encrypt targets on `eql_v3_*` domains are deleted. The `stack-encrypt:1:` ciphertext prefix stays: decryption does not pass through Postgres types, and the prefix is @@ -193,18 +298,48 @@ without an entry. ### Host languages -- **Go.** `int8`, `int16`, `uint8` and `uint16` map to their own kinds instead - of widening to 32 bits; this lands before the Go SDK ships, so no stored - data uses the old mapping. `Int128` and `Uint128` are SDK value types. - `time.Time` means `Timestamp`, and `encrypt.Date{Year, Month, Day}` is a - date. A field whose EQL target is a date family accepts `time.Time`, - truncated to its UTC calendar day. -- **JavaScript.** A `BigInt` maps to the smallest kind that holds it, up to - 128 bits, and decodes as a `BigInt`. `Date` means `Timestamp`; a date has - its own wrapper. +A binding picks the kind from the field's **declared EQL target**, never +from the host type. This keeps the rule `eql-bindings` already enforces: a +target accepts exactly one kind, and nothing below the binding converts. The +binding converts a host value into the target's kind when it can do so +without loss, and refuses it otherwise. + +| Kind | EQL family | | +|---|---|---| +| `Int16` | `smallint` | | +| `Int32` | `integer` | | +| `Int64` | `bigint` | | +| `Float32` | `real` | | +| `Float64` | `double` | | +| `Decimal` | `numeric` | | +| `Date` | `date` | | +| `Timestamp` | `timestamp` | | +| `String` | `text` | | +| `Bool` | `boolean` | | +| `Int8`, `UInt8`, `UInt16`, `UInt32`, `UInt64`, `Int128`, `UInt128`, `Bytes` | none | ciphertext and terms through Stack Encrypt's own targets; no EQL domain yet | + +- **Go.** Any Go integer type whose every value fits the target's kind is + accepted (`int8` into `smallint`, `int32` into `bigint`); a wider one is + range-checked and refused if the value does not fit. `int8`, `int16`, + `uint8` and `uint16` have their own kinds for targets that name them. + `Int128` and `Uint128` are SDK value types. `time.Time` means `Timestamp`, + and `encrypt.Date{Year, Month, Day}` is a date. A field whose target is a + date family also accepts `time.Time` and takes its calendar day **in the + value's own location** (`t.Date()`), which is how Go code builds a date; + taking the UTC day would store the previous day for a local midnight east + of UTC. +- **JavaScript.** A `BigInt` takes the target's kind, and is refused if it + does not fit. It decodes as a `BigInt`. `Date` means `Timestamp`; a date + has its own wrapper. +- **Decimal.** A host decimal (Go's `shopspring/decimal`, a JavaScript + string) is refused if it is NaN or infinite, has more than 28 digits after + the point, or does not fit a 96-bit mantissa at its scale. Postgres + `numeric` holds values that `Decimal` cannot, and a binding refuses them + rather than rounding. - **Every binding** runs vitaminc's shared test vectors: a host value, the - kind it must map to, and the exact `[tag] ++ payload` leaf bytes, including - normalisation cases. The leaf is deterministic even though the AEAD is not. + target, the kind it must map to, and the exact `[tag] ++ payload` leaf + bytes, including normalisation and refusal cases. The leaf is + deterministic even though the AEAD is not. ### EQL v4 @@ -222,39 +357,70 @@ without an entry. domain then fails on insert, rather than only matching nothing when it is queried. Queries are a different matter; see Consequences. - **New EQL targets are v4-only.** In v3 they stay refused, with a reason - that points to v4. + that points to v4. A domain that a lossy order transform needs, with an + `hm` beside its `op`, is one of these, and is the first point where the v4 + SQL differs from the v3 SQL. - **`@cipherstash/eql` 4.x ships both bundles** from `main`. A SQL fix lands in both names in one release. -- **`stash eql install --eql-version 3|4|all`** chooses the bundle and defaults - to 3, which is today's behaviour. The default changes to 4, announced in - advance, when the TypeScript stack has moved to Stack Encrypt. +- **`--eql-version` comes back, on purpose.** It was removed when v3 became + the only installable generation, and the CLI now refuses it. It returns as + `--eql-version 3|4|all` on both `stash eql install` and + `stash eql migration`, so the migration-first paths (Drizzle, and Supabase + with a local `supabase/` directory, where a direct install does not survive + `supabase db reset`) can install v4 too. The migration they write carries + the chosen bundles, and `--supabase` grants on each bundle's schemas. The + default is 3, which is today's behaviour, and changes to 4, announced in + advance, when the TypeScript stack has moved to Stack Encrypt. `2` keeps + its current message pointing at the EQL 2.3.1 release, rather than + becoming an ordinary invalid value. - **During the overlap**, expected to last a quarter or more, cipherstash-client takes fixes only. New kinds and domains are produced through Stack Encrypt and v4. An exception is a decision written down in its issue. ### Review -Using the canonical order bytes as the PRF input is the simplest way to make -equality and ordering agree, and it is frozen once data is stored under it. -Dan Draper signs it off in writing before the `vitaminc-prf` change lands. +A domain that locks equality and order together derives both from the same +transformed bytes, and that choice is frozen once data is stored under it. +Dan Draper signs off each such domain in writing before it ships. Domains +whose equality and order transforms are independent need no such sign-off. + +## Open decision: exhaustive handling of `Value` + +`vitaminc-prf`, `vitaminc-ore` and every binding must handle every kind of +`Value`, and adding a kind should fail to build until each one does. Two +approaches remain: + +- **A: drop `#[non_exhaustive]`.** Every crate that matches `Value` gets + exhaustiveness, and adding a kind is a breaking change for all of them. +- **B: a `ValueVisitor` trait with one required method per kind, keeping + `#[non_exhaustive]`.** Crates that must handle every kind implement it, and + adding a kind is breaking only for them; other crates keep a `_ =>` arm. + +The leaning is B. The layering plan sets out the comparison. Either way, +the conformance test is the backstop: a `#[non_exhaustive]` enum lets a +downstream match compile with a wildcard, so the compiler alone does not +guarantee coverage under B. ## Consequences -- **The release order is fixed.** ore.rs (`orderable-bytes`) and cllw-ore - first, then vitaminc 0.6.0, then the Stack Encrypt breaking release. Stack - Encrypt must reach crates.io before `eql-bindings` uses its new API, since - `cargo publish` builds `eql-bindings` against the registry. The EQL v4 - bundle and the Go SDK changes follow. -- **Adding a kind later is additive**, because `Value` and `ValueKind` are - `#[non_exhaustive]`. It still takes a tag, a PRF domain, an order encoding, - a codec in every binding, and test vectors, or an entry in the exceptions - list. +- **The release order is fixed.** vitaminc 0.6.0 first, then cllw-ore and + ore-rs (additive releases implementing `OrderScheme`), then the Stack + Encrypt breaking release. Stack Encrypt must reach crates.io before + `eql-bindings` uses its new API, since `cargo publish` builds + `eql-bindings` against the registry. The EQL v4 bundle and the Go SDK + changes follow. +- **Adding a kind later** takes a tag, PRF and order encodings, a codec in + every binding, test vectors and a row in the kind-to-family table, or an + entry in the exceptions list. Whether it is a breaking change depends on + the open decision. - **Stack Encrypt's terms change** for `-0.0` and negative-sign NaN floats, - for non-NFC text equality, and for `Bool` order terms, which are removed. - None is deployed, so the breaking release carries them without migration. -- **cipherstash-client's terms do not change.** Its fixed-length - `orderable-bytes` output, cllw-ore's typed impls and its block ORE padding - are all kept, so every stored v3 payload stays queryable. + for non-NFC text equality, for `Bool` order terms, which are removed, and + for every text order term: `orderize_string` drops symbols and keeps case, + while the fold keeps symbols and folds case. None is deployed, so the + breaking release carries them without migration. +- **cipherstash-client's terms do not change.** `orderable-bytes`, cllw-ore's + typed impls (`Bool` included) and its block ORE padding are all kept, so + every stored v3 payload stays queryable. - **Separate names do not stop a cross-producer comparison in SQL.** EQL's domains are over `jsonb`, and when no operator is defined for an exact pair of types, Postgres treats a domain as its base type. `v3col = v4term` then @@ -266,13 +432,37 @@ Dan Draper signs it off in writing before the `vitaminc-prf` change lands. domain check), the function form `eql_v3.eq(v3col, v4term)` ("not unique"), and `OPERATOR(public.=)` between a v3 and a v4 column ("not unique"). Verified on Postgres 17 with both bundles installed in one database. +- **Tooling that recognises encrypted columns by the `eql_v3_` prefix must + learn `eql_v4_` before any v4 column can exist.** Today that is + `stash init`'s introspection (`cli/src/commands/init/lib/introspect.ts`), + the wizard's tools (`wizard/src/tools/wizard-tools.ts`) and + `classifyEqlDomain` (`migrate/src/version.ts`), which `stash status` and + `stash encrypt status` use. Getting this wrong shows an encrypted column as + plaintext, which invites encrypting it a second time. The installer's + ADR-0001 capture, rebuild and verify logic hard-codes the disposable + schemas as `('eql_v3', 'eql_v3_internal')` + (`installer/derived-search-index-restoration.ts`, + `installer/installation-state.ts`); without `eql_v4` and + `eql_v4_internal` there, a v4 reinstall would capture and restore none of + its own indexes. Both land with the first released v4 bundle. +- **The release tooling assumes one bundle.** The lockstep stamp is read from + `COMMENT ON SCHEMA eql_v3` (`scripts/sync-lockstep-versions.mjs`, + `packages/eql/tasks/release/prepare-bindings-assets.sh`); there is one + `release-manifest.json` per artefact, which the CLI's `bundle-digest.ts` + and `eql-sql-asset-freshness.test.mjs` check; `eql_bindings::sql` embeds + only the v3 files; and `_build-eql-sql.yml` uploads only them. The v4 + bundle needs its own stamp, manifest entries, freshness check, crate + constants and release upload before it ships, or v4 drift goes unnoticed + with every digest still verifying. The `postgres-eql` image states which + bundles it installs. - **Moving a column from v3 to v4 means re-encrypting it.** The ciphertext and every term change. How that migration runs belongs to the decision that moves the TypeScript stack onto Stack Encrypt. -- **The CLI surface changes** (`--eql-version`), so `skills/stash-cli`, - `skills/stash-indexing` and `skills/stash-postgres` change in the same PR. +- **The CLI surface changes** (`--eql-version` on `stash eql install` and + `stash eql migration`), so `skills/stash-cli`, `skills/stash-indexing` and + `skills/stash-postgres` change in the same PR. - **This replaces a definition in a plan.** `docs/plans/2026-10-04-plan-builder.md` - calls "EQL v4" a name for a Stack Encrypt payload in the v3 envelope and + called "EQL v4" a name for a Stack Encrypt payload in the v3 envelope and the `eql_v3_*` domains. That plan is updated to this ADR. ## Deferred @@ -285,11 +475,17 @@ Dan Draper signs it off in writing before the `vitaminc-prf` change lands. nothing through it. `Bool` could support that scheme alone, enforced by a marker trait on schemes. Not built until needed. - **Cross-version blocker operators.** Blockers for every v3/v4 pair of the - same type family and capability, created by whichever bundle is installed - second, would turn the silent cross-producer `=` into an error. About 2,300 - operators, generated from the catalog like the existing blockers. Not built - yet. Until then, a cross-producer query is caught only by typed clients and - the function form. + same type family and capability would turn the silent cross-producer `=` + into an error. About 2,300 operators, generated from the catalog like the + existing blockers. Not built yet; until then, a cross-producer query is + caught only by typed clients and the function form. When they are built, + **the v4 bundle owns them**: it creates them in `eql_v4_internal`, against + whichever `public.eql_v3_*` domains exist, on every install. A v4 reinstall + drops them with its schema and creates them again. A v3 reinstall leaves + them alone, because they depend only on the `public` domains, which + survive, and not on anything in `eql_v3`. Installing v3 after v4 into a + database that had no v3 domains leaves them absent until v4 is reinstalled, + so `--eql-version all` installs v3 first. - **Locale-aware collation**, as its own domain with the collation version in its name. - **ASCII-packed text domains**, until a customer's column sizes make the diff --git a/docs/plans/2026-10-04-plan-builder.md b/docs/plans/2026-10-04-plan-builder.md index e70ea3b11..5037c67ad 100644 --- a/docs/plans/2026-10-04-plan-builder.md +++ b/docs/plans/2026-10-04-plan-builder.md @@ -662,7 +662,7 @@ In the `Text` family, the three `Ord` suffixes carry equality too. Each type has a query type, with `Query` after its name: `TextEqQuery`. The value of `encrypt_into` is the Go type name. -An EQL value from the engine is an EQL v4 value: its version field is `4`, and Postgres stores it in an `eql_v4_*` domain. +Once EQL v4 lands, an EQL value from the engine is an EQL v4 value: its version field is `4`, and Postgres stores it in an `eql_v4_*` domain. The ciphertext inside it is a Stack Encrypt ciphertext, which starts with `stack-encrypt:1:`. EQL v4 is the v3 SQL, emitted from the same source under a second name, so that a column written by Stack Encrypt and one written by cipherstash-client are different Postgres types ([ADR-0002](../adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md)). This replaces this plan's first definition, under which "EQL v4" named a Stack Encrypt payload in the v3 envelope and an `eql_v3` domain; Postgres could not tell the two producers apart, and a query from one against a column written by the other matched nothing. diff --git a/docs/plans/2026-10-11-order-and-equality-terms-layering.md b/docs/plans/2026-10-11-order-and-equality-terms-layering.md index 54f3d6947..aacd27a2d 100644 --- a/docs/plans/2026-10-11-order-and-equality-terms-layering.md +++ b/docs/plans/2026-10-11-order-and-equality-terms-layering.md @@ -11,7 +11,8 @@ under "Open decision". **Date:** 2026-10-11 **Amends:** ADR-0002 (`docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md`). -Where the two disagree, this document wins. +The decisions below were folded into the ADR on 2026-10-12, and the ADR is +now the reference for them. This plan keeps the steps. **Issues:** cipherstash/vitaminc#374 (`vitaminc-ore`), #1063 (index crates) ## Why @@ -83,11 +84,20 @@ Starting on cipherstash/vitaminc#374 under ADR-0002 turned up four problems. text folding (removing combining marks, case folding, with the Unicode version pinned) and prefix truncation. - Each transform is named in its term's domain label, for example - `…/timestamp/day/v1`, so terms built with different transforms can never - be compared. + `…/timestamp/day/v1`, so terms built with different transforms are + domain-separated: a probe built with one matches nothing built with the + other. A stored term does not record its label, so they are not made + incomparable. Keeping a query on the column's transforms is the index + spec's job. - Equality and order can be **independent** (an exact timestamp for equality, the day for order) or **locked** (the same transform value passed to both). Stack's index spec (#1063) carries the choice per column. +- A **lossy order transform needs an equality term beside it.** EQL's + numeric, date and timestamp `_ord` domains store no `hm` and answer `=` + with the order term, so a day-resolution order term there would make `=` + match every row from that day. A non-injective order transform is allowed + only on a domain that stores an equality term and answers `=` through it, + as `text_ord` does. Those domains are new, v4-only EQL targets. - The semantic baseline lives in each term crate: `vitaminc-prf` and `vitaminc-ore` each apply it. A conformance test checks that values equal under `Value` semantics produce equal terms in both. There is no shared @@ -259,6 +269,8 @@ flowchart TB ## Changes to ADR-0002 +These have been made in the ADR itself. + | ADR-0002 says | Now | |---|---| | "Equality and order terms are computed from one canonical form per kind, and both layers use the same one" | Each kind has a semantic baseline; lossy transforms are chosen per domain, independently or locked |