otel benchmark harness

.gitignore

@@ -8,3 +8,9 @@ test/python/__pycache__/
 **/__pycache__/
 .Rhistory
 .cache/
+benchmark/data
+benchmark/work
+# Local/ephemeral benchmark outputs; keep the committed CI baseline only.
+benchmark/results/*
+!benchmark/results/baseline-otel.json
+!benchmark/results/.gitkeep

AGENTS.md

@@ -131,6 +131,8 @@ release`; test with `./build/release/test/unittest --test-dir . "test/sql/*"`; f
 sqllogictests in `test/sql/`: `rawduck.test` (core types/records), `raw_ingest.test` (evolution),
 `raw_advanced.test` (streaming, transforms, pool, optimize, projections), `raw_attach.test`
 (stores, transactions, persistence), `raw_api.test` (server lifecycle), `ducklake.test` (`require ducklake`, skips when absent).
+OTEL ingest performance: `./scripts/benchmark/run_otel.sh` + `./scripts/benchmark/compare.sh`
+(see `scripts/benchmark/README.md` and `BENCHMARK.md`).
 Every feature needs: happy path, evolution interaction, error case, and—for anything that can
 return wrong data—a proof test (e.g. tampering with a projection to prove the rewrite engaged).
 `raw_ingest` output is `(table, created, columns_added, columns_widened, rows, errors)`;

BENCHMARK.md

@@ -5,19 +5,31 @@ runs at native columnar speed, instead of keeping opaque JSON and paying `->>` e
 scan. The primary benchmark is the realistic workload — **OTEL telemetry** (OTLP/JSON logs, metrics,
 traces) — with the GH Archive run kept below as a historical wide-schema stress test.
 
+### Harness
+
+```sh
+GEN=ninja make release
+./scripts/benchmark/run_otel.sh --quick                      # CI / remote smoke (100k)
+./scripts/benchmark/compare.sh benchmark/results/smoke.json  # vs committed baseline
+./scripts/benchmark/run_otel.sh --records 1000000 --runs 5   # full publishable numbers
+```
+
+Each session is **one DuckDB process**: cold ingest (schema discovery) then warm ingest (same
+process, fresh timestamps, `columns_added = 0`). See `scripts/benchmark/README.md`.
+
 ## OTEL ingestion (primary)
 
-Published results — Apple Silicon, 10 cores, DuckDB v1.5.5, 1,000,000 records per signal, OTLP/JSON
-export envelopes (the exact bytes an OpenTelemetry Collector posts to an OTLP/HTTP json endpoint),
-default settings (no tuning):
+Published results — multi-core commodity hardware, DuckDB v1.5.5, 1,000,000 records per signal,
+OTLP/JSON export envelopes (the exact bytes an OpenTelemetry Collector posts to an OTLP/HTTP json
+endpoint), default settings (no manual tuning):
 
 | signal | records | columns | source NDJSON | ingest | records/s | throughput | on disk |
 |---|---:|---:|---:|---:|---:|---:|---:|
-| traces  | 1,000,000 | 23 | 704 MB | 1.65 s | 604k | 426 MB/s | 72 MB |
-| logs    | 1,000,000 | 20 | 598 MB | 1.71 s | 586k | 350 MB/s | 61 MB |
-| metrics | 1,000,000 | 13 | 495 MB | 1.30 s | 771k | 381 MB/s | 56 MB |
+| traces  | 1,000,000 | 15 | 435 MB | 0.74 s | 1.35M | 586 MB/s | 50 MB |
+| logs    | 1,000,000 | 11 | 294 MB | 0.80 s | 1.26M | 369 MB/s | 9 MB |
+| metrics | 1,000,000 | 9  | 353 MB | 1.03 s | 970k | 342 MB/s | 88 MB |
 
-3M telemetry records shredded into typed columns in **4.66 s (~644k records/s)**. The OTLP transform
+3M telemetry records shredded into typed columns in **2.6 s (~1.2M records/s)**. The OTLP transform
 explodes the nested `resource → scope → record` envelopes, flattens KeyValue attributes
 (`http.status_code`, `service.name`, …) into typed columns, and normalizes byte ids to hex — all in
 the parallel parse stage. Because each NDJSON line is a fat export envelope that explodes into many
@@ -31,10 +43,10 @@ JSON object per span, queried with `->>`):
 
 | query | JSON `->>` | RawDuck | speedup |
 |---|---:|---:|---:|
-| error count by service (`status>=500`) | 115 ms | 3 ms | 38× |
-| p99 latency by route | 283 ms | 9 ms | 31× |
-| status-code distribution | 92 ms | 6 ms | 15× |
-| storage | 250 MB | 72 MB | 3.5× smaller |
+| error count by service (`status>=500`) | 71 ms | 2 ms | 36× |
+| p99 latency by route | 136 ms | 5 ms | 27× |
+| status-code distribution | 63 ms | 1 ms | 63× |
+| storage | 232 MB | 38 MB | 6× smaller |
 
 This baseline is the *favorable* one. Real OTLP keeps attributes as KeyValue **arrays**, so querying
 them without shredding means `UNNEST`-ing the `resource → scope → span` nesting and scanning each

README.md

@@ -11,7 +11,7 @@ flattens nested objects into real columns, transforms and evolves the schema as
 ### ⚡ Benefits
 No `CREATE TABLE`, no schema declarations, no `json_extract` at query time. Because data lands
 shredded into native typed columns instead of opaque JSON strings, analytical queries run
-**15–38× faster** on telemetry queries, **3.5× smaller** on disk — see benchmark.
+**27–63× faster** on telemetry queries, **6× smaller** on disk — see benchmark.
 
 ### ⚙️ Under the hood
 RawDuck delivers a complete engine rather than a parser: ingestion is transactional, pipelined, and
@@ -84,22 +84,22 @@ want to project or filter their result columns.
 ## Benchmark: OTEL at line speed
 
 Real OTLP/JSON export envelopes — logs, metrics, traces — shredded into typed columns on
-ingest. Apple Silicon, DuckDB v1.5.5, 1M records per signal:
+ingest. DuckDB v1.5.5, 1M records per signal:
 | signal | records | columns | source NDJSON | ingest | records/s | throughput | on disk |
 |---|---:|---:|---:|---:|---:|---:|---:|
-| traces  | 1,000,000 | 23 | 704 MB | 1.65 s | 604k | 426 MB/s | 72 MB |
-| logs    | 1,000,000 | 20 | 598 MB | 1.71 s | 586k | 350 MB/s | 61 MB |
-| metrics | 1,000,000 | 13 | 495 MB | 1.30 s | 771k | 381 MB/s | 56 MB |
+| traces  | 1,000,000 | 15 | 435 MB | 0.74 s | 1.35M | 586 MB/s | 50 MB |
+| logs    | 1,000,000 | 11 | 294 MB | 0.80 s | 1.26M | 369 MB/s | 9 MB |
+| metrics | 1,000,000 | 9  | 353 MB | 1.03 s | 970k | 342 MB/s | 88 MB |
 
-**3M telemetry records in 4.7 s.** Queries on shredded spans run **15–38× faster** than a JSON
-column with identical results; storage is **3.5× smaller**. One call handles envelope explode,
-KeyValue attribute flattening, and byte-id normalization — no schema upfront:
+**3M telemetry records in 2.6 s (~1.2M records/s).** Queries on shredded spans run **27–63× faster**
+than a JSON column with identical results; storage is **6× smaller**. One call handles envelope
+explode, KeyValue attribute flattening, and byte-id normalization — no schema upfront:
 
 ```sql
 CALL raw_ingest_file('traces', 'export.ndjson', transform := 'otlp-traces');
 
 SELECT "resource.service.name", count(*) FROM traces
-WHERE "http.status_code" >= 500 GROUP BY 1;   -- 3 ms
+WHERE "http.status_code" >= 500 GROUP BY 1;   -- 2 ms
 ```
 
 As a wide-schema stress test, one hour of [GH Archive](https://www.gharchive.org/) data (914

benchmark/results/baseline-otel.json

@@ -0,0 +1,33 @@
+{
+  "benchmark": "otel_ingest",
+  "notes": "Conservative 100k floor for main; refresh after verified ingest wins. Gate is 98% of these values.",
+  "records_per_signal": 100000,
+  "runs": 3,
+  "session": "single_process_cold_then_warm",
+  "results": {
+    "traces_cold": {
+      "records": 100000,
+      "records_per_sec": 325000
+    },
+    "traces_warm": {
+      "records": 100000,
+      "records_per_sec": 325000
+    },
+    "logs_cold": {
+      "records": 100000,
+      "records_per_sec": 570000
+    },
+    "logs_warm": {
+      "records": 100000,
+      "records_per_sec": 355000
+    },
+    "metrics_cold": {
+      "records": 100000,
+      "records_per_sec": 285000
+    },
+    "metrics_warm": {
+      "records": 100000,
+      "records_per_sec": 285000
+    }
+  }
+}

scripts/benchmark/README.md

@@ -0,0 +1,96 @@
+# RawDuck benchmark scripts
+
+Reproducible OTEL ingest timings. Data lands in `benchmark/data/` (gitignored);
+JSON results in `benchmark/results/` (gitignored except the committed baseline).
+
+**Policy:** only merge ingest changes that improve or match baseline on every
+metric (cold and warm, all signals). Features that regress any default path are
+removed, not shipped as opt-in toggles.
+
+## Session model (cold then warm)
+
+Each **session** is one DuckDB process (`run_otel_session.py`):
+
+1. `LOAD rawduck`
+2. **Cold** — `raw_ingest_file` + `CHECKPOINT` (timed; schema discovery)
+3. `DELETE FROM table` (untimed; keep evolved DDL, empty rows)
+4. **Warm** — `raw_ingest_file` on a second NDJSON with shifted timestamps +
+   `CHECKPOINT` (timed; must report `columns_added = 0` and `columns_widened = 0`)
+
+Warm keeps the in-memory schema cache hot (same process). This models a collector
+hub that already absorbed an OTLP shape and receives a fresh batch — not a new
+DuckDB startup on an old database file.
+
+Host timing uses `time.perf_counter()` around each timed window (not SQL
+`epoch_ms`, which is too coarse for short runs).
+
+## Remote / new machine (post-build)
+
+```sh
+# after: git clone --recurse-submodules … && GEN=ninja make release
+./scripts/benchmark/run_otel.sh --quick --output benchmark/results/smoke.json
+./scripts/benchmark/compare.sh benchmark/results/smoke.json
+
+# publishable numbers
+./scripts/benchmark/run_otel.sh --records 1000000 --runs 5 \
+  --output "benchmark/results/otel_1m_$(hostname -s)_$(date -u +%Y%m%dT%H%M%SZ).json"
+```
+
+Send back: the JSON file, host CPU/cores/RAM/OS, and `git rev-parse HEAD`.
+
+## Quick smoke (~100k / signal)
+
+```sh
+GEN=ninja make release
+./scripts/benchmark/run_otel.sh --quick --output benchmark/results/smoke.json
+./scripts/benchmark/compare.sh benchmark/results/smoke.json
+```
+
+## Full baseline (1M records, best of 5 sessions)
+
+```sh
+./scripts/benchmark/run_otel.sh --records 1000000 --runs 5 \
+  --output benchmark/results/otel_1m.json
+```
+
+### Metric definitions
+
+- **Cold:** first `raw_ingest_file` in a fresh database — CREATE TABLE + column adds.
+- **Warm:** second `raw_ingest_file` in the **same** process after `DELETE FROM table`
+  — evolved schema, no DDL, fresh timestamps in `*_warm.ndjson`.
+
+## Compare against baseline
+
+`benchmark/results/baseline-otel.json` holds committed thresholds (100k quick run).
+Fails if any metric drops below 98% of baseline:
+
+```sh
+./scripts/benchmark/compare.sh benchmark/results/smoke.json
+./scripts/benchmark/compare.sh result.json --min-ratio 0.95
+```
+
+Refresh the committed baseline only after a verified improvement on all six metrics:
+
+```sh
+./scripts/benchmark/run_otel.sh --quick --output benchmark/results/baseline-otel.json
+# review, then commit benchmark/results/baseline-otel.json
+```
+
+## Generate data only
+
+```sh
+python3 scripts/benchmark/gen_otlp.py all 1000000
+python3 scripts/benchmark/gen_otlp.py traces 1000000 benchmark/data 1700086400000000000 _warm
+```
+
+## Files
+
+| Script | Role |
+|---|---|
+| `lib.sh` | Paths + build checks (bash) |
+| `gen_otlp.py` | OTLP/JSON NDJSON generator (`ts_base` + suffix for warm files) |
+| `run_otel_session.py` | Single-process cold→warm driver (host timing) |
+| `run_otel.sh` | Orchestrator: data gen, N sessions, JSON output (bash) |
+| `compare.sh` | Regression gate vs `baseline-otel.json` (bash) |
+
+Requires: bash, python3, a release build (`build/release/duckdb` + extension).

scripts/benchmark/compare.sh

@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# Compare a benchmark JSON result against a saved baseline (within tolerance).
+#
+# usage: ./scripts/benchmark/compare.sh RESULT.json [BASELINE.json] [--min-ratio 0.98]
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+RESULT="${1:?result json required}"
+shift || true
+
+BASELINE="${ROOT}/benchmark/results/baseline-otel.json"
+MIN_RATIO=0.98
+
+while (( $# > 0 )); do
+	case "$1" in
+		--min-ratio)
+			MIN_RATIO="${2:?}"
+			shift 2
+			;;
+		*)
+			BASELINE="$1"
+			shift
+			;;
+	esac
+done
+
+if [[ ! -f "${RESULT}" ]]; then
+	echo "missing result: ${RESULT}" >&2
+	exit 1
+fi
+if [[ ! -f "${BASELINE}" ]]; then
+	echo "missing baseline: ${BASELINE} (run ./scripts/benchmark/run_otel.sh --quick first)" >&2
+	exit 1
+fi
+
+PYTHON="${PYTHON:-$(command -v python3 || true)}"
+if [[ -z "${PYTHON}" || ! -x "${PYTHON}" ]]; then
+	echo "python3 not found (set PYTHON=...)" >&2
+	exit 1
+fi
+
+"${PYTHON}" - "${RESULT}" "${BASELINE}" "${MIN_RATIO}" <<'PY'
+import json
+import sys
+
+result_path, baseline_path, min_ratio = sys.argv[1:4]
+min_ratio = float(min_ratio)
+
+with open(result_path) as f:
+    result = json.load(f)
+with open(baseline_path) as f:
+    baseline = json.load(f)
+
+failures = []
+for key, base in baseline.get("results", {}).items():
+    cur = result.get("results", {}).get(key)
+    if not cur:
+        failures.append(f"missing metric: {key}")
+        continue
+    base_rps = base.get("records_per_sec", 0)
+    cur_rps = cur.get("records_per_sec", 0)
+    if base_rps <= 0:
+        continue
+    ratio = cur_rps / base_rps
+    print(f"{key}: {cur_rps} rec/s vs baseline {base_rps} ({ratio:.1%})")
+    if ratio < min_ratio:
+        failures.append(f"{key}: {ratio:.1%} < {min_ratio:.0%} of baseline")
+
+if failures:
+    print("REGRESSION:", file=sys.stderr)
+    for f in failures:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+print("OK: within tolerance")
+PY