quackscience · lmangani · Aug 22, 2026 · Aug 18, 2026 · Aug 18, 2026 · Aug 18, 2026
diff --git a/.github/workflows/MainDistributionPipeline.yml b/.github/workflows/MainDistributionPipeline.yml
@@ -24,10 +24,10 @@ concurrency:
 jobs:
   duckdb-stable-build:
     name: Build extension binaries
-    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5-variegata
+    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5.5
     with:
       duckdb_version: v1.5.5
-      ci_tools_version: v1.5-variegata
+      ci_tools_version: v1.5.5
       extension_name: rawduck
       # Windows is excluded until the VS2026 runner image is compatible with
       # duckdb v1.5's vendored fmt (stdext::checked_array_iterator was
@@ -36,9 +36,9 @@ jobs:
 
   code-quality-check:
     name: Code Quality Check
-    uses: duckdb/extension-ci-tools/.github/workflows/_extension_code_quality.yml@v1.5-variegata
+    uses: duckdb/extension-ci-tools/.github/workflows/_extension_code_quality.yml@v1.5.5
     with:
       duckdb_version: v1.5.5
-      ci_tools_version: v1.5-variegata
+      ci_tools_version: v1.5.5
       extension_name: rawduck
       format_checks: 'format;tidy'
diff --git a/.github/workflows/QuackIntegration.yml b/.github/workflows/QuackIntegration.yml
@@ -21,10 +21,10 @@ concurrency:
 jobs:
   quack-integration:
     name: rawduck:quack (linux_amd64)
-    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5-variegata
+    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5.5
     with:
       duckdb_version: v1.5.5
-      ci_tools_version: v1.5-variegata
+      ci_tools_version: v1.5.5
       extension_name: rawduck
       opt_in_archs: 'linux_amd64'
       use_merged_vcpkg_manifest: '1'

diff --git a/.github/workflows/Release.yml b/.github/workflows/Release.yml
@@ -53,10 +53,10 @@ env:
 jobs:
   build:
     name: Build extension binaries
-    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5-variegata
+    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5.5
     with:
       duckdb_version: v1.5.5
-      ci_tools_version: v1.5-variegata
+      ci_tools_version: v1.5.5
       extension_name: rawduck
       # build exactly the four published platforms: linux/osx x amd64/arm64.
       # windows stays excluded for the VS2026/fmt incompatibility (see

diff --git a/AGENTS.md b/AGENTS.md
@@ -11,9 +11,11 @@ real typed columns at ingest (one-time cost) so every query runs at native colum
 instead of storing opaque JSON strings and paying `->>` extraction on every scan (45–265× slower,
 see BENCHMARK.md).
 
-Pinned to DuckDB **v1.5.5**: the `duckdb/` submodule commit and the versions in
-`.github/workflows/MainDistributionPipeline.yml` must stay in sync. Build with `GEN=ninja make
-release`; test with `./build/release/test/unittest --test-dir . "test/sql/*"`; format with
+Pinned to DuckDB **v1.5.5** (latest stable as of this pin): the `duckdb/` submodule
+tag, `extension-ci-tools` on the matching `v1.5.5` tip, and `duckdb_version` /
+`ci_tools_version` in `.github/workflows/{MainDistributionPipeline,Release,QuackIntegration}.yml`
+must stay in sync. Build with `GEN=ninja make release`; test with
+`./build/release/test/unittest --test-dir . "test/sql/*"`; format with
 `make format-fix` (CI enforces it).
 
 ## Source map (src/)
@@ -133,6 +135,8 @@ sqllogictests in `test/sql/`: `rawduck.test` (core types/records), `raw_ingest.t
 (stores, transactions, persistence), `raw_api.test` (server lifecycle), `ducklake.test` (`require ducklake`, skips when absent).
 OTEL ingest performance: `./scripts/benchmark/run_otel.sh` + `./scripts/benchmark/compare.sh`
 (see `scripts/benchmark/README.md` and `BENCHMARK.md`).
+VARIANT vs RawDuck (v1.5.5, traces): `./scripts/benchmark/run_variant.sh` on branch
+`feat/variant-benchmark`.
 Every feature needs: happy path, evolution interaction, error case, and—for anything that can
 return wrong data—a proof test (e.g. tampering with a projection to prove the rewrite engaged).
 `raw_ingest` output is `(table, created, columns_added, columns_widened, rows, errors)`;

diff --git a/BENCHMARK.md b/BENCHMARK.md
@@ -1,119 +1,135 @@
 # RawDuck Benchmark
 
-RawDuck's bet: shred schema-less event JSON into real typed columns at ingest so every later query
-runs at native columnar speed, instead of keeping opaque JSON and paying `->>` extraction on every
-scan. The primary benchmark is the realistic workload — **OTEL telemetry** (OTLP/JSON logs, metrics,
-traces) — with the GH Archive run kept below as a historical wide-schema stress test.
+Primary workload: **OTEL telemetry** (OTLP/JSON logs, metrics, traces). GH Archive is a
+wide-schema stress test in the appendix.
+
+All published numbers: DuckDB **v1.5.5**, default RawDuck settings, Apple **M3 Ultra**
+(32 cores, 512 GiB) unless noted. Cold ingest = first `raw_ingest_file` in a fresh database;
+warm = second ingest in the same process after `DELETE` (`columns_added = 0`). Report the
+best of N sessions unless noted. See `scripts/benchmark/README.md` for metric definitions.
 
 ### Harness
 
 ```sh
 GEN=ninja make release
-./scripts/benchmark/run_otel.sh --quick                      # CI / remote smoke (100k)
-./scripts/benchmark/compare.sh benchmark/results/smoke.json  # vs committed baseline
-./scripts/benchmark/run_otel.sh --records 1000000 --runs 5   # full publishable numbers
-```
 
-Each session is **one DuckDB process**: cold ingest (schema discovery) then warm ingest (same
-process, fresh timestamps, `columns_added = 0`). See `scripts/benchmark/README.md`.
+# OTEL bulk ingest (NDJSON file)
+./scripts/benchmark/run_otel.sh --quick
+./scripts/benchmark/run_otel.sh --records 1000000 --runs 5
+
+# OTEL streaming ingest (OpenTelemetry SDK → raw_serve HTTP)
+./scripts/benchmark/run_otel_streaming.sh --quick
+./scripts/benchmark/run_otel_streaming.sh --workers 16 --spans-per-worker 60000
 
-## OTEL ingestion (primary)
+# VARIANT vs RawDuck (same trace dataset)
+./scripts/benchmark/run_variant.sh --quick
+./scripts/benchmark/run_variant.sh --records 1000000 --runs 3
+```
+
+## OTEL bulk ingest
 
-Published results — multi-core commodity hardware, DuckDB v1.5.5, 1,000,000 records per signal,
-OTLP/JSON export envelopes (the exact bytes an OpenTelemetry Collector posts to an OTLP/HTTP json
-endpoint), default settings (no manual tuning):
+1,000,000 records per signal, OTLP/JSON export envelopes (collector POST bodies), best of 5
+sessions:
 
-| signal | records | columns | source NDJSON | ingest | records/s | throughput | on disk |
-|---|---:|---:|---:|---:|---:|---:|---:|
-| traces  | 1,000,000 | 15 | 435 MB | 0.74 s | 1.35M | 586 MB/s | 50 MB |
-| logs    | 1,000,000 | 11 | 294 MB | 0.80 s | 1.26M | 369 MB/s | 9 MB |
-| metrics | 1,000,000 | 9  | 353 MB | 1.03 s | 970k | 342 MB/s | 88 MB |
+| signal | records | source NDJSON | cold ingest | records/s | throughput |
+|---|---:|---:|---:|---:|---:|
+| traces | 1,000,000 | 435 MB | 1.19 s | 841k | 366 MB/s |
+| logs | 1,000,000 | 294 MB | 0.87 s | 1.15M | 338 MB/s |
+| metrics | 1,000,000 | 353 MB | 1.13 s | 889k | 314 MB/s |
 
-3M telemetry records shredded into typed columns in **2.6 s (~1.2M records/s)**. The OTLP transform
-explodes the nested `resource → scope → record` envelopes, flattens KeyValue attributes
-(`http.status_code`, `service.name`, …) into typed columns, and normalizes byte ids to hex — all in
-the parallel parse stage. Because each NDJSON line is a fat export envelope that explodes into many
-records, the loader batches by bytes (not just line count) so the parse threads stay fed
-automatically; no `batch_size` tuning is needed.
+3M telemetry records in **3.2 s** (~940k records/s average). Warm ingest matches cold within
+~2% on each signal.
 
 ### Query speed (1,000,000 spans)
 
-Same spans, identical results — shredded typed columns vs the "just keep the JSON" baseline (one
-JSON object per span, queried with `->>`):
+Same spans — shredded typed columns vs one JSON object per span (`->>`), best of 3 runs:
 
 | query | JSON `->>` | RawDuck | speedup |
 |---|---:|---:|---:|
-| error count by service (`status>=500`) | 71 ms | 2 ms | 36× |
-| p99 latency by route | 136 ms | 5 ms | 27× |
-| status-code distribution | 63 ms | 1 ms | 63× |
-| storage | 232 MB | 38 MB | 6× smaller |
-
-This baseline is the *favorable* one. Real OTLP keeps attributes as KeyValue **arrays**, so querying
-them without shredding means `UNNEST`-ing the `resource → scope → span` nesting and scanning each
-attribute array by key — hundreds of times slower than reading a typed column.
+| error count by service (`status>=500`) | 39 ms | 1.5 ms | 26× |
+| p99 latency by route | 99 ms | 3.2 ms | 31× |
+| status-code distribution | 35 ms | 2.5 ms | 14× |
+| storage | 143 MB | 39.5 MB | 3.6× smaller |
 
 ### Reproduce
 
-Generate OTLP/JSON envelopes (one `Export*ServiceRequest` per line):
-
-```python
-# gen_otlp.py  ->  python3 gen_otlp.py 1000000
-import json, os, random, sys
-random.seed(11)
-SVC = ["checkout","cart","payments","search","auth","inventory","shipping","frontend"]
-RT  = ["/api/v1/orders","/api/v1/cart","/api/v1/pay","/api/v1/search","/login","/health"]
-def kv(k,v):
-    if isinstance(v,bool): return {"key":k,"value":{"boolValue":v}}
-    if isinstance(v,int):  return {"key":k,"value":{"intValue":str(v)}}
-    if isinstance(v,float):return {"key":k,"value":{"doubleValue":v}}
-    return {"key":k,"value":{"stringValue":str(v)}}
-def res(s): return {"attributes":[kv("service.name",s),kv("deployment.environment","production"),
-                                  kv("cloud.region","us-east-1"),kv("host.name","pod-%d"%random.randint(1,400))]}
-def span(ts):
-    st=random.choice([200,200,200,201,400,404,500]); d=random.randint(2*10**5,8*10**8)
-    return {"traceId":os.urandom(16).hex(),"spanId":os.urandom(8).hex(),"name":random.choice(RT),
-            "kind":random.randint(1,5),"startTimeUnixNano":str(ts),"endTimeUnixNano":str(ts+d),
-            "attributes":[kv("http.method",random.choice(["GET","POST","PUT","DELETE"])),
-                          kv("http.route",random.choice(RT)),kv("http.status_code",st),
-                          kv("retry",random.choice([True,False]))],"status":{"code":2 if st>=500 else 1}}
-def gen(name,total,per,rec,wrap):
-    ts=1_700_000_000_000_000_000; w=0
-    with open(name+".ndjson","w") as f:
-        while w<total:
-            n=min(per,total-w)
-            f.write(json.dumps(wrap(random.choice(SVC),[rec(ts+(w+j)*1000) for j in range(n)]))+"\n"); w+=n
-n=int(sys.argv[1]) if len(sys.argv)>1 else 1_000_000
-gen("traces",n,80,span,lambda s,r:{"resourceSpans":[{"resource":res(s),"scopeSpans":[{"spans":r}]}]})
-# logs/metrics envelopes follow the same shape with resourceLogs.scopeLogs.logRecords /
-# resourceMetrics.scopeMetrics.metrics (see the repo's bench scripts for the full generator)
+```sh
+./scripts/benchmark/run_otel.sh --records 1000000 --runs 5
+./scripts/benchmark/run_variant.sh --records 1000000 --runs 3   # queries + storage above
 ```
 
-Ingest and query:
+## OTEL streaming ingest
 
-```sql
--- bduck otel.db
-.timer on
-CALL raw_ingest_file('traces', 'traces.ndjson', transform := 'otlp-traces');   -- and otlp-logs / otlp-metrics
-CHECKPOINT;
+Real OTLP/HTTP **protobuf** traffic via the OpenTelemetry Python SDK into `raw_serve()`
+(concurrent exporter processes, not bulk NDJSON):
 
--- baseline: identical spans as a JSON blob per row
-CREATE TABLE traces_json AS SELECT to_json(traces)::JSON AS j FROM traces;
+| workers | spans | wall | spans/s |
+|---:|---:|---:|---:|
+| 4 | 20,000 | 0.75 s | 27k |
+| 16 | 960,000 | 3.85 s | 250k |
+
+`rows_ingested` must equal `total_spans_sent` (checked by the harness).
+
+### Reproduce
 
--- typed columns vs ->> extraction
-SELECT "resource.service.name", count(*) FROM traces WHERE "http.status_code" >= 500 GROUP BY 1;
-SELECT j->>'resource.service.name', count(*) FROM traces_json
-  WHERE CAST(j->>'http.status_code' AS BIGINT) >= 500 GROUP BY 1;
+```sh
+./scripts/benchmark/run_otel_streaming.sh --workers 16 --spans-per-worker 60000
 ```
 
-Run each query three times (against a `-readonly` database) and report the best.
+First run creates `benchmark/work/otel-streaming-venv` (OpenTelemetry SDK dependency).
+
+## VARIANT vs RawDuck (DuckDB v1.5.5)
+
+Same 1,000,000 OTLP/JSON trace spans. Paths:
+
+| path | definition |
+|---|---|
+| RawDuck | `raw_ingest_file(..., transform := 'otlp-traces')` → typed columns |
+| VARIANT OTLP | SQL unnest → one `VARIANT` `{resource, span}` per span (KeyValue arrays kept) |
+| JSON OTLP | same exploded shape as `JSON` |
+| VARIANT-flat | `to_json(traces)::VARIANT` of shredded RawDuck rows (encode/query only) |
+| JSON-flat | same shredded rows as `JSON`, queried with `->>` |
+
+Disk = `used_blocks × block_size` after cold `CHECKPOINT`. Parenthetical file size is after
+warm re-ingest (includes free-list holes; not comparable across paths). VARIANT requires
+`STORAGE_VERSION 'v1.5.0'`.
+
+### Ingest + storage (1,000,000 spans)
 
-## Appendix: GH Archive (historical, wide-schema stress test)
+| path | M3 Ultra | Spark GB10 aarch64 (`--threads 8`) |
+|---|---|---|
+| RawDuck | 0.99 s · 1.01M rec/s · 39.5 MB (108 MB file) | 1.18 s · 850k rec/s · 35.5 MB (91 MB file) |
+| VARIANT-flat | encode · 35.8 MB | encode · 39.5 MB |
+| VARIANT OTLP | 11.96 s · 84k · 53.5 MB (106 MB file) | 7.96 s · 126k · 54.5 MB (114 MB file) |
+| JSON-flat | encode · 143 MB | encode · 142 MB |
+| JSON OTLP | 4.72 s · 212k · 241 MB (484 MB file) | 5.71 s · 175k · 242 MB (484 MB file) |
 
-One hour of real [GH Archive](https://www.gharchive.org/) data — 247,199 events / 956 MB NDJSON
-exploding to a **914-column** schema. This is the worst case for shredding (extreme, sparse schema
-churn), kept as a stress test rather than the representative workload.
+Spark GB10: 20 cores, 122 GiB, Linux aarch64.
 
-Published results (Apple Silicon, 10 cores, DuckDB v1.5.5):
+### Queries (best of 3, ms)
+
+| encoding | M3 Ultra | Spark GB10 |
+|---|---|---|
+| | errors / p99 / status | errors / p99 / status |
+| RawDuck | 1.5 / 3.2 / 2.5 | 1.3 / 4.9 / 5.1 |
+| JSON-flat | 39 / 99 / 35 | 65 / 136 / 63 |
+| JSON OTLP positional | 213 / 303 / 193 | 253 / 415 / 215 |
+| JSON OTLP key lookup | 344 / 418 / 304 | 397 / 580 / 366 |
+| VARIANT-flat | 436 / 1225 / 416 | 700 / 1988 / 696 |
+| VARIANT OTLP positional | 1227 / 3479 / 1162 | 2013 / 5947 / 1852 |
+| VARIANT OTLP key lookup | 1493 / 3658 / 1404 | 2107 / 5996 / 2049 |
+
+### Reproduce
+
+```sh
+./scripts/benchmark/run_variant.sh --records 1000000 --runs 3
+./scripts/benchmark/run_variant.sh --records 1000000 --runs 3 --threads 8   # many-core ARM
+```
+
+## Appendix: GH Archive (wide-schema stress test)
+
+One hour of [GH Archive](https://www.gharchive.org/) data — 247,199 events / 956 MB NDJSON /
+914 columns. Apple Silicon, 10 cores, DuckDB v1.5.5:
 
 | | JSON column | RawDuck | |
 |---|---:|---:|---|
@@ -122,73 +138,23 @@ Published results (Apple Silicon, 10 cores, DuckDB v1.5.5):
 | distinct repos per actor | 457 ms | 10 ms | 46× |
 | sum of push payload sizes | 265 ms | 1 ms | 265× |
 | events per minute | 236 ms | 3 ms | 79× |
-| ingest | 1.4 s | ~13 s | one-time cost |
+| cold ingest | 1.4 s | ~13 s | one-time cost |
+| warm re-ingest | — | ~4.9 s | steady state |
 | storage | 1.05 GB | 636 MB | 40% smaller |
 
+### Reproduce
+
 ```sh
-curl -sL https://data.gharchive.org/2024-01-15-10.json.gz -o gh.json.gz   # raw_ingest_file reads .gz directly
+curl -sL https://data.gharchive.org/2024-01-15-10.json.gz -o gh.json.gz
 ```
 
 ```sql
--- RawDuck: one call shreds the whole hour (914 typed columns, evolution included)
 CALL raw_ingest_file('gh_events', 'gh.json.gz');
 CHECKPOINT;
-
--- baseline keeps raw JSON (records='false' alone still infers a STRUCT; the columns clause keeps it raw)
-CREATE TABLE gh_raw AS SELECT json
-  FROM read_json('gh.json.gz', format='newline_delimited', records='false', columns={json: 'JSON'});
-CHECKPOINT;
-```
-
-RawDuck queries / baseline (`->>`) queries:
-
-```sql
-SELECT type, count(*) AS n FROM gh_events GROUP BY type ORDER BY n DESC;
-SELECT "repo.name", count(*) AS n FROM gh_events WHERE type='PushEvent' GROUP BY 1 ORDER BY n DESC LIMIT 10;
-SELECT "actor.login", count(DISTINCT "repo.name") AS r FROM gh_events GROUP BY 1 ORDER BY r DESC LIMIT 10;
-SELECT sum("payload.size") FROM gh_events WHERE type='PushEvent';
-SELECT date_trunc('minute', created_at) AS m, count(*) FROM gh_events GROUP BY m ORDER BY m;
-
-SELECT json->>'$.type' AS type, count(*) AS n FROM gh_raw GROUP BY type ORDER BY n DESC;
-SELECT json->>'$.repo.name' AS repo, count(*) AS n FROM gh_raw WHERE json->>'$.type'='PushEvent' GROUP BY 1 ORDER BY n DESC LIMIT 10;
-SELECT json->>'$.actor.login' AS a, count(DISTINCT json->>'$.repo.name') AS r FROM gh_raw GROUP BY 1 ORDER BY r DESC LIMIT 10;
-SELECT sum(CAST(json->>'$.payload.size' AS BIGINT)) FROM gh_raw WHERE json->>'$.type'='PushEvent';
-SELECT date_trunc('minute', CAST(json->>'$.created_at' AS TIMESTAMP)) AS m, count(*) FROM gh_raw GROUP BY m ORDER BY m;
-```
-
-### Warm-table ingest
-
-The cold GH numbers include schema discovery (CREATE + evolution sync points). Re-ingesting the same
-hour into the already-evolved table runs fully parallel: **~4.9 s** — the steady-state rate once a
-table's shape has stabilized (the realistic OTEL case, where the schema is stable after warmup).
-
-### INSERT-syntax streaming (fastest path)
-
-`INSERT INTO raw.ingest.t SELECT ...` streams any SQL source through a parallel zero-copy sink:
-
-```sql
-ATTACH 'rawduck:store.db' AS raw;
-INSERT INTO raw.ingest.narrow SELECT '{"a":' || range || '}' FROM range(5000000);
--- 5M narrow rows in ~0.8 s  (~6.1M rows/s)
-```
-
-### Adaptive layout and projections
-
-```sql
-CALL raw_stats();
-CALL raw_optimize('gh_events');     -- physically reorders by hottest columns
-CALL raw_project('gh_events');      -- materializes the hottest aggregation
-SET rawduck_use_projections = true; -- transparent rewrite of eligible count(*) queries
 ```
 
 ## Pitfalls
 
-- Don't split NDJSON with Python's `splitlines()` — it splits on `\u2028`/`\u2029` which appear raw
-  inside real-world strings and corrupts records. Split on `\n` only.
-- The shell reports query times with `.timer on`; dot-commands don't work via `duckdb -c`, use
-  `-f script.sql`.
-- A shallow duckdb submodule clone without tags makes the shell report `v0.0.1`; fetch the release
-  tag (`git -C duckdb fetch --depth 1 origin tag v1.5.5`) or extension installs 404.
-- For large imports where each line is a fat container (OTLP envelopes, CloudWatch log groups), the
-  loader auto-parallelizes via byte-aware batching; you only need `batch_size` to *raise* the line
-  cap for very small flat records.
+- Split NDJSON on `\n` only (not `splitlines()` — `\u2028`/`\u2029` appear in strings).
+- Use `.timer on` via `duckdb -f script.sql` (not `duckdb -c`).
+- Shallow duckdb clones without tags report `v0.0.1`; fetch tag `v1.5.5`.