fix(precompute): value_column ignored at ingest - COUNT(DISTINCT col) estimates cardinality of the wrong column #432

Summary

For a query such as:

SELECT srcip, COUNT(DISTINCT dstip)
FROM netflow_table
GROUP BY srcip;

the planner, inference configuration, and streaming configuration are all generated correctly:

• Aggregation type: HLL
• Grouping keys: [srcip]
• value_column: dstip

However, the ingest → precompute path never reads AggregationConfig.value_column. Instead, it always feeds the scalar value that arrived on the wire (the dataset's configured value column, e.g. pkt_len) into the accumulator.

As a result, ASAP computes:

COUNT(DISTINCT pkt_len) GROUP BY srcip

while ClickHouse correctly computes:

COUNT(DISTINCT dstip) GROUP BY srcip

The two systems are answering different queries, making fidelity and benchmark comparisons invalid.

Root Cause

Although value_column is propagated correctly through planning and configuration, it is never used during ingest or precomputation.

JSON Ingest

json_ingest.rs reads only JsonFileIngestConfig.value_col (the dataset default value column).

Remote Write Ingest

Remote-write ingestion forwards the Prometheus sample value directly:

Sample.value

Routing

ingest_source.rs forwards:

(labels, ts, s.value)

for every aggregation and performs no per-aggregation value_column resolution.

Worker Execution

worker.rs::apply_sample() hashes the supplied value into the accumulator and never consults config.value_column:

fn apply_sample(updater, series_key, val, ts, config) {
    if updater.is_keyed() {
        ...
        updater.update_keyed(&key, val, ts);
    } else {
        updater.update_single(val, ts);
    }
}

For HLL aggregations, val is therefore the dataset's wire value (e.g. pkt_len) rather than the requested column (dstip).

Consequently, the HLL accumulator estimates:

COUNT(DISTINCT pkt_len)

instead of:

COUNT(DISTINCT dstip)

Impact

Correctness

COUNT(DISTINCT col) returns the cardinality of the dataset's configured value column rather than the column requested by the query.

Benchmark Validity

Comparisons between ASAP and ClickHouse for cardinality queries are invalid because the systems are computing distinct counts over different columns.

Silent Failure

No error or warning is emitted:

• The planner generates the expected configuration.
• value_column appears correctly populated.
• The query executes successfully.
• Results look plausible but are incorrect.

This makes the issue particularly difficult to detect and can lead to misleading fidelity and performance conclusions.

Acceptance Criteria

• COUNT(DISTINCT dstip) GROUP BY srcip over the netflow workload yields ASAP cardinality matching ClickHouse uniq(dstip) within HLL tolerance.
• SUM / quantile / top-k queries are unaffected (still aggregate the wire value column).
• Regression test: two samples with the same srcip, different dstip, identical pkt_len → HLL estimate ≈ 2 (currently 1).