Add Apache Kudu

impala-kudu/README.md

@@ -0,0 +1,36 @@
+# Impala with Apache Kudu
+
+This configuration uses Apache Impala as the SQL execution layer and Apache Kudu as the persistent mutable storage layer. It is separate from `impala/`, which queries the source Parquet file directly.
+
+## What runs
+
+The single-node stack contains one Kudu master, one Kudu tablet server, Hive Metastore, Impala statestored, catalogd, impalad, and an Impala shell client. The Kudu table uses replication factor 1 and a composite `(WatchID, EventTime)` primary key so all 99,997,497 source rows are preserved. A view restores ClickBench's canonical 105-column order.
+
+`benchmark.sh` invokes the shared ClickBench driver. `load` first registers the downloaded Parquet file as the source table, creates the Kudu table, and loads it through 10 sequential hash buckets. The batches are sequential rather than parallel; a single full-table `UPSERT ... SELECT` exceeded the Impala sort-node memory limit on the target machine.
+
+## Caching and restart behavior
+
+The result is tagged `no-cold`. The shared driver drops the host page cache before the first try, but it does not restart the stack between cold tries. Restarting the existing Impala quickstart stack loses the in-memory catalog state needed by the benchmark; the existing Parquet `impala/` configuration uses the same exception. The two later tries are warm-cache measurements.
+
+## Storage measurement
+
+`data-size` reports allocated filesystem blocks for all persistent Kudu master, Kudu tablet server, and Hive Metastore paths. This avoids counting unallocated extents in Kudu's sparse WAL files. Input Parquet is excluded because it is only the load source and is not part of the Kudu-backed database after loading.
+
+## Reproduction
+
+The dated result was measured on an AWS `c6a.2xlarge` instance with Ubuntu 24.04, 8 vCPUs, 16 GiB RAM, and a 150 GB gp3 root volume. From a fresh ClickBench checkout:
+
+```bash
+cd impala-kudu
+./benchmark.sh
+```
+
+The benchmark requires x86-64 with AVX support because the Apache Impala images do not start on ARM hosts. `install` installs Docker only when absent, preserves an already-working Compose plugin, downloads the HMS dependency, and pulls the container images before the measured run.
+
+The exact pre-run source manifest and raw logs were retained as external review evidence; they are not part of this upstream result package. The formal query sweep remains the unmodified 43-query ClickBench workload in `queries.sql`; any deterministic diagnostic queries used to investigate tied or floating-point outputs are kept outside this upstream system directory.
+
+## Result qualification
+
+The retained formal run preserved all `99,997,497` rows and produced all 129 timing cells. Its validator accepted Q1–Q41 and Q43; the original Q42 selected a different row at a `PageViews = 1` tie boundary because its `ORDER BY PageViews DESC LIMIT 10 OFFSET 10000` does not define an order among tied groups. A validation-only variant ordering additionally by `WindowClientWidth, WindowClientHeight` matched exactly. The formal timed SQL was not changed.
+
+Q4's observed relative difference was `8.1792e-14`, inside the predeclared `1e-12` tolerance. Independent exact diagnostics also matched `COUNT(*)`, `COUNT(UserID)`, and `SUM(CAST(UserID AS DECIMAL(38,0)))` between the Parquet and Kudu tables.

impala-kudu/benchmark.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+set -euo pipefail
+
+# Apache Kudu is a durable shared server, but restarting catalogd between
+# queries loses the in-memory Impala catalog used by this single-node setup.
+# We therefore flush only the OS page cache and mark the result no-cold.
+export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-single"
+export BENCH_RESTARTABLE=no
+export BENCH_DURABLE=yes
+export BENCH_TRIES=3
+export BENCH_QUERIES_FILE="queries.sql"
+export BENCH_CHECK_TIMEOUT=900
+export BENCH_CONCURRENT_CONNECTIONS=10
+export BENCH_CONCURRENT_DURATION=600
+export KUDU_LOAD_BUCKETS="${KUDU_LOAD_BUCKETS:-10}"
+
+exec ../lib/benchmark-common.sh

impala-kudu/check

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if docker info >/dev/null 2>&1; then
+    DOCKER=(docker)
+else
+    DOCKER=(sudo docker)
+fi
+
+dump_diagnostics() {
+    "${DOCKER[@]}" ps -a --format '  {{.Names}}: {{.Status}}' \
+        --filter 'name=impala-' --filter 'name=kudu-' >&2 || true
+    local container
+    while IFS= read -r container; do
+        [ -n "$container" ] || continue
+        printf '%s\n' "--- $container (last 50 log lines) ---" >&2
+        "${DOCKER[@]}" logs --tail 50 "$container" 2>&1 | sed 's/^/  /' >&2 || true
+    done < <("${DOCKER[@]}" ps -a --format '{{.Names}}' \
+        --filter 'name=impala-' --filter 'name=kudu-')
+}
+
+out=$(curl -sSf -m 5 http://localhost:25000/healthz 2>&1) || {
+    printf 'impala-kudu/check: /healthz unreachable: %s\n' "$out" >&2
+    dump_diagnostics
+    exit 1
+}
+if [ "$out" != "OK" ]; then
+    printf 'impala-kudu/check: /healthz returned: %s\n' "$out" >&2
+    dump_diagnostics
+    exit 1
+fi
+
+kudu_out=$("${DOCKER[@]}" exec kudu-master \
+    kudu cluster ksck kudu-master:7051 2>&1) || {
+    printf 'impala-kudu/check: kudu cluster ksck failed:\n%s\n' "$kudu_out" >&2
+    dump_diagnostics
+    exit 1
+}

impala-kudu/create.sql

@@ -0,0 +1,156 @@
+CREATE DATABASE IF NOT EXISTS clickbench;
+USE clickbench;
+
+DROP VIEW IF EXISTS hits;
+DROP TABLE IF EXISTS hits_raw;
+
+CREATE EXTERNAL TABLE hits_raw (
+    WatchID bigint,
+    JavaEnable smallint,
+    Title string,
+    GoodEvent smallint,
+    EventTime bigint,
+    EventDate int,
+    CounterID int,
+    ClientIP int,
+    RegionID int,
+    UserID bigint,
+    CounterClass smallint,
+    OS smallint,
+    UserAgent smallint,
+    URL string,
+    Referer string,
+    IsRefresh smallint,
+    RefererCategoryID smallint,
+    RefererRegionID int,
+    URLCategoryID smallint,
+    URLRegionID int,
+    ResolutionWidth smallint,
+    ResolutionHeight smallint,
+    ResolutionDepth smallint,
+    FlashMajor smallint,
+    FlashMinor smallint,
+    FlashMinor2 string,
+    NetMajor smallint,
+    NetMinor smallint,
+    UserAgentMajor smallint,
+    UserAgentMinor string,
+    CookieEnable smallint,
+    JavascriptEnable smallint,
+    IsMobile smallint,
+    MobilePhone smallint,
+    MobilePhoneModel string,
+    Params string,
+    IPNetworkID int,
+    TraficSourceID smallint,
+    SearchEngineID smallint,
+    SearchPhrase string,
+    AdvEngineID smallint,
+    IsArtifical smallint,
+    WindowClientWidth smallint,
+    WindowClientHeight smallint,
+    ClientTimeZone smallint,
+    ClientEventTime bigint,
+    SilverlightVersion1 smallint,
+    SilverlightVersion2 smallint,
+    SilverlightVersion3 int,
+    SilverlightVersion4 smallint,
+    PageCharset string,
+    CodeVersion int,
+    IsLink smallint,
+    IsDownload smallint,
+    IsNotBounce smallint,
+    FUniqID bigint,
+    OriginalURL string,
+    HID int,
+    IsOldCounter smallint,
+    IsEvent smallint,
+    IsParameter smallint,
+    DontCountHits smallint,
+    WithHash smallint,
+    HitColor string,
+    LocalEventTime bigint,
+    Age smallint,
+    Sex smallint,
+    Income smallint,
+    Interests smallint,
+    Robotness smallint,
+    RemoteIP int,
+    WindowName int,
+    OpenerName int,
+    HistoryLength smallint,
+    BrowserLanguage string,
+    BrowserCountry string,
+    SocialNetwork string,
+    SocialAction string,
+    HTTPError smallint,
+    SendTiming int,
+    DNSTiming int,
+    ConnectTiming int,
+    ResponseStartTiming int,
+    ResponseEndTiming int,
+    FetchTiming int,
+    SocialSourceNetworkID smallint,
+    SocialSourcePage string,
+    ParamPrice bigint,
+    ParamOrderID string,
+    ParamCurrency string,
+    ParamCurrencyID smallint,
+    OpenstatServiceName string,
+    OpenstatCampaignID string,
+    OpenstatAdID string,
+    OpenstatSourceID string,
+    UTMSource string,
+    UTMMedium string,
+    UTMCampaign string,
+    UTMContent string,
+    UTMTerm string,
+    FromTag string,
+    HasGCLID smallint,
+    RefererHash bigint,
+    URLHash bigint,
+    CLID int
+)
+STORED AS PARQUET
+LOCATION '/clickbench/hits';
+
+REFRESH hits_raw;
+
+-- The Parquet file stores EventTime/ClientEventTime/LocalEventTime as Unix epoch seconds (BIGINT)
+-- and EventDate as days since 1970-01-01 (INT). Wrap the raw table in a view that exposes the
+-- standard ClickBench types so queries.sql matches the canonical text. CAST(from_unixtime(...) AS
+-- TIMESTAMP) is the idiomatic Impala BIGINT-epoch -> TIMESTAMP conversion, and days_add() yields a
+-- DATE from an int day count.
+CREATE VIEW hits AS
+SELECT
+    WatchID, JavaEnable, Title, GoodEvent,
+    CAST(from_unixtime(EventTime) AS TIMESTAMP) AS EventTime,
+    days_add(DATE '1970-01-01', EventDate) AS EventDate,
+    CounterID, ClientIP, RegionID, UserID, CounterClass, OS, UserAgent,
+    URL, Referer, IsRefresh, RefererCategoryID, RefererRegionID,
+    URLCategoryID, URLRegionID, ResolutionWidth, ResolutionHeight,
+    ResolutionDepth, FlashMajor, FlashMinor, FlashMinor2, NetMajor, NetMinor,
+    UserAgentMajor, UserAgentMinor, CookieEnable, JavascriptEnable,
+    IsMobile, MobilePhone, MobilePhoneModel, Params, IPNetworkID,
+    TraficSourceID, SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical,
+    WindowClientWidth, WindowClientHeight, ClientTimeZone,
+    CAST(from_unixtime(ClientEventTime) AS TIMESTAMP) AS ClientEventTime,
+    SilverlightVersion1, SilverlightVersion2, SilverlightVersion3,
+    SilverlightVersion4, PageCharset, CodeVersion, IsLink, IsDownload,
+    IsNotBounce, FUniqID, OriginalURL, HID, IsOldCounter, IsEvent,
+    IsParameter, DontCountHits, WithHash, HitColor,
+    CAST(from_unixtime(LocalEventTime) AS TIMESTAMP) AS LocalEventTime,
+    Age, Sex, Income, Interests, Robotness, RemoteIP, WindowName,
+    OpenerName, HistoryLength, BrowserLanguage, BrowserCountry,
+    SocialNetwork, SocialAction, HTTPError, SendTiming, DNSTiming,
+    ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming,
+    SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID,
+    ParamCurrency, ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID,
+    OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign,
+    UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash, URLHash, CLID
+FROM hits_raw;
+
+-- Make sure Impala's planner has up-to-date stats; without this the
+-- first query pays the metadata-collection wall-clock as if it were
+-- query work, and pure scan queries plan as full broadcasts.
+COMPUTE STATS hits_raw;

impala-kudu/create_kudu.sql

@@ -0,0 +1,143 @@
+USE clickbench;
+
+DROP VIEW IF EXISTS hits_kudu;
+DROP TABLE IF EXISTS hits_kudu_raw;
+
+CREATE TABLE hits_kudu_raw (
+    WatchID bigint,
+    EventTime timestamp,
+    JavaEnable smallint,
+    Title string,
+    GoodEvent smallint,
+    EventDate date,
+    CounterID int,
+    ClientIP int,
+    RegionID int,
+    UserID bigint,
+    CounterClass smallint,
+    OS smallint,
+    UserAgent smallint,
+    URL string,
+    Referer string,
+    IsRefresh smallint,
+    RefererCategoryID smallint,
+    RefererRegionID int,
+    URLCategoryID smallint,
+    URLRegionID int,
+    ResolutionWidth smallint,
+    ResolutionHeight smallint,
+    ResolutionDepth smallint,
+    FlashMajor smallint,
+    FlashMinor smallint,
+    FlashMinor2 string,
+    NetMajor smallint,
+    NetMinor smallint,
+    UserAgentMajor smallint,
+    UserAgentMinor string,
+    CookieEnable smallint,
+    JavascriptEnable smallint,
+    IsMobile smallint,
+    MobilePhone smallint,
+    MobilePhoneModel string,
+    Params string,
+    IPNetworkID int,
+    TraficSourceID smallint,
+    SearchEngineID smallint,
+    SearchPhrase string,
+    AdvEngineID smallint,
+    IsArtifical smallint,
+    WindowClientWidth smallint,
+    WindowClientHeight smallint,
+    ClientTimeZone smallint,
+    ClientEventTime timestamp,
+    SilverlightVersion1 smallint,
+    SilverlightVersion2 smallint,
+    SilverlightVersion3 int,
+    SilverlightVersion4 smallint,
+    PageCharset string,
+    CodeVersion int,
+    IsLink smallint,
+    IsDownload smallint,
+    IsNotBounce smallint,
+    FUniqID bigint,
+    OriginalURL string,
+    HID int,
+    IsOldCounter smallint,
+    IsEvent smallint,
+    IsParameter smallint,
+    DontCountHits smallint,
+    WithHash smallint,
+    HitColor string,
+    LocalEventTime timestamp,
+    Age smallint,
+    Sex smallint,
+    Income smallint,
+    Interests smallint,
+    Robotness smallint,
+    RemoteIP int,
+    WindowName int,
+    OpenerName int,
+    HistoryLength smallint,
+    BrowserLanguage string,
+    BrowserCountry string,
+    SocialNetwork string,
+    SocialAction string,
+    HTTPError smallint,
+    SendTiming int,
+    DNSTiming int,
+    ConnectTiming int,
+    ResponseStartTiming int,
+    ResponseEndTiming int,
+    FetchTiming int,
+    SocialSourceNetworkID smallint,
+    SocialSourcePage string,
+    ParamPrice bigint,
+    ParamOrderID string,
+    ParamCurrency string,
+    ParamCurrencyID smallint,
+    OpenstatServiceName string,
+    OpenstatCampaignID string,
+    OpenstatAdID string,
+    OpenstatSourceID string,
+    UTMSource string,
+    UTMMedium string,
+    UTMCampaign string,
+    UTMContent string,
+    UTMTerm string,
+    FromTag string,
+    HasGCLID smallint,
+    RefererHash bigint,
+    URLHash bigint,
+    CLID int,
+    PRIMARY KEY (WatchID, EventTime)
+)
+PARTITION BY HASH (WatchID) PARTITIONS 2
+STORED AS KUDU
+TBLPROPERTIES (
+    'kudu.master_addresses' = 'kudu-master:7051',
+    'kudu.num_tablet_replicas' = '1'
+);
+
+-- Kudu requires primary-key columns to lead the physical schema. Expose a
+-- projection in canonical ClickBench order so SELECT * remains comparable.
+CREATE VIEW hits_kudu AS
+SELECT
+    WatchID, JavaEnable, Title, GoodEvent, EventTime, EventDate,
+    CounterID, ClientIP, RegionID, UserID, CounterClass, OS,
+    UserAgent, URL, Referer, IsRefresh, RefererCategoryID, RefererRegionID,
+    URLCategoryID, URLRegionID, ResolutionWidth, ResolutionHeight, ResolutionDepth, FlashMajor,
+    FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor, UserAgentMinor,
+    CookieEnable, JavascriptEnable, IsMobile, MobilePhone, MobilePhoneModel, Params,
+    IPNetworkID, TraficSourceID, SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical,
+    WindowClientWidth, WindowClientHeight, ClientTimeZone, ClientEventTime, SilverlightVersion1, SilverlightVersion2,
+    SilverlightVersion3, SilverlightVersion4, PageCharset, CodeVersion, IsLink, IsDownload,
+    IsNotBounce, FUniqID, OriginalURL, HID, IsOldCounter, IsEvent,
+    IsParameter, DontCountHits, WithHash, HitColor, LocalEventTime, Age,
+    Sex, Income, Interests, Robotness, RemoteIP, WindowName,
+    OpenerName, HistoryLength, BrowserLanguage, BrowserCountry, SocialNetwork, SocialAction,
+    HTTPError, SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming,
+    FetchTiming, SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID, ParamCurrency,
+    ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID, UTMSource,
+    UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID,
+    RefererHash, URLHash, CLID
+FROM hits_kudu_raw;