From 940325044dccb45c46377b9a3d76ef9cfbf7477d Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Mon, 12 Oct 2026 18:08:12 +0000 Subject: [PATCH 1/2] Remove file-based demo and legacy surfaces Drop the asap-otap-demo binary, the file-backed nightly and scale-sweep benchmarks, the criterion bench, the Grafana verify script and KLL dashboard, the demo guide, the multiprocess e2e test, the flat RecordBatch encode/records codec with its tests, and the adapter and control-channel stubs. Update the READMEs, benchmark docs, Compose file and run-nightly.sh (which ran the removed criterion bench) to match. --- .dockerignore | 5 +- README.md | 57 +- .../benches/asap_sketch_pipeline.rs | 398 ------ asap-precompute-rs/src/adapter.rs | 64 - asap-precompute-rs/src/bin/asap_otap_demo.rs | 1066 ----------------- .../src/bin/demo/prometheus_export.rs | 202 ---- asap-precompute-rs/src/control_channel.rs | 60 - asap-precompute-rs/src/otap/decode.rs | 670 ----------- asap-precompute-rs/src/otap/encode.rs | 261 ---- asap-precompute-rs/src/otap/records.rs | 997 --------------- asap-precompute-rs/tests/api_surface.rs | 260 ---- asap-precompute-rs/tests/otap_codec.rs | 237 ---- .../tests/otap_multiprocess_e2e.rs | 244 ---- benchmarks/Dockerfile | 9 - benchmarks/PROFILING.md | 2 +- benchmarks/README.md | 206 +--- benchmarks/STREAMING.md | 11 +- benchmarks/nightly/asap-sketch.yaml | 65 - benchmarks/nightly/scenario-steps.yaml.j2 | 47 - benchmarks/run-file-nightly.sh | 55 - benchmarks/run-nightly.sh | 4 - benchmarks/run-pipeline-component.sh | 100 -- benchmarks/run-scale-sweep.py | 272 ----- benchmarks/summarize-stages.py | 46 - demos/kll-grafana/compose.yaml | 4 +- demos/kll-grafana/grafana/dashboards/kll.json | 81 -- demos/kll-grafana/verify.py | 53 - docs/demo-guide.md | 153 --- docs/kll-windows.md | 12 +- 29 files changed, 45 insertions(+), 5596 deletions(-) delete mode 100644 asap-precompute-rs/benches/asap_sketch_pipeline.rs delete mode 100644 asap-precompute-rs/src/adapter.rs delete mode 100644 asap-precompute-rs/src/bin/asap_otap_demo.rs delete mode 100644 asap-precompute-rs/src/bin/demo/prometheus_export.rs delete mode 100644 asap-precompute-rs/src/control_channel.rs delete mode 100644 asap-precompute-rs/src/otap/decode.rs delete mode 100644 asap-precompute-rs/src/otap/encode.rs delete mode 100644 asap-precompute-rs/src/otap/records.rs delete mode 100644 asap-precompute-rs/tests/api_surface.rs delete mode 100644 asap-precompute-rs/tests/otap_codec.rs delete mode 100644 asap-precompute-rs/tests/otap_multiprocess_e2e.rs delete mode 100644 benchmarks/Dockerfile delete mode 100644 benchmarks/nightly/asap-sketch.yaml delete mode 100644 benchmarks/nightly/scenario-steps.yaml.j2 delete mode 100755 benchmarks/run-file-nightly.sh delete mode 100755 benchmarks/run-pipeline-component.sh delete mode 100644 benchmarks/run-scale-sweep.py delete mode 100644 benchmarks/summarize-stages.py delete mode 100644 demos/kll-grafana/grafana/dashboards/kll.json delete mode 100644 demos/kll-grafana/verify.py delete mode 100644 docs/demo-guide.md diff --git a/.dockerignore b/.dockerignore index cb07d08..6033086 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,7 +1,4 @@ .cache .git benchmark-results -asap-precompute-rs/target/* -!asap-precompute-rs/target/release/ -asap-precompute-rs/target/release/* -!asap-precompute-rs/target/release/asap-otap-demo +asap-precompute-rs/target diff --git a/README.md b/README.md index 0f8431b..4c23198 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,8 @@ ASAP's [OTAP Dataflow](https://github.com/open-telemetry/otel-arrow/tree/main/ru processor. It ports ASAP's edge **precompute + sketching** runtime (windowed -DDSketch / KLL / HLL / CountSketch / CountMin-Sketch aggregation) onto +DDSketch / KLL / HLL / CountSketch / Count-Min / Count-Min-with-heap +aggregation) onto OTAP Dataflow's Arrow-native streaming model as a single unified `asap_sketches` processor. @@ -17,8 +18,8 @@ overlay step: | Feature | What it adds | Extra dependency | |---|---|---| -| *(default, no features)* | Host-neutral precompute runtime — the five sketch types, windowing, overflow policy. No Arrow, no OTAP, no Tokio. | none | -| `otap` | The standalone Arrow codec (`encode_batch`/`decode_batch`, `OtapMetricRecords`). | `arrow-array`, `arrow-schema` | +| *(default, no features)* | Host-neutral precompute runtime — the sketch types, windowing, overflow policy. No Arrow, no OTAP, no Tokio. | none | +| `otap` | Plugin config (`otap::config`) and the `sketch.*` carrier keys. | `arrow-array`, `arrow-schema` | | `otap-engine` | The real OTAP node: `AsapSketchesProcessor` (a genuine `local::Processor`, `linkme`-registered) and its direct `SketchEnvelope <-> OtapPdata` binding (`otap::codec`). | the real `otel-arrow-dfe-*` crates, via a plain git dependency pinned to a specific `open-telemetry/otel-arrow` commit — see `Cargo.toml` | `otap-engine`'s git dependency is what used to require staging this @@ -44,11 +45,10 @@ same way it already does for the `asap_sketchlib` git dependency — `SketchEnvelope <-> OtapPdata` binding: implements OTAP's own `MetricsView` trait family straight over `&[SketchEnvelope]` (encode) and decodes a real `OtapPdata` straight into - `Vec` (decode). No intermediate `RecordBatch` or - `OtapMetricRecords` two-batch hop — those exist as a general, - real-OTAP-free carrier format for other potential Strategy-B - adapters (Telegraf, Vector), but this repo only ever ships to OTAP, - so the OTAP-facing path skips straight to the real type. + `Vec` (decode). A sketch record's `sketch.envelope` is + the sketch's full state as the self-describing ASAPv1 envelope from + `asap_sketchlib`; the codec rejects bytes that do not decode as the + declared `sketch.type`. There are no deltas. **There is exactly one transport: the pipeline.** `AsapSketchesProcessor` sends only via `effect_handler.send_message_with_source_node` — an @@ -70,38 +70,28 @@ pipeline", for the full rationale and the real schema this produces. `cargo fmt --check`, against `open-telemetry/otel-arrow` commit `3e85c346` (2026-08-24, pinned in `Cargo.toml`). -## Configuration + standalone Arrow codec +## Configuration -| File | Responsibility | -|---|---| -| [`config.rs`](./asap-precompute-rs/src/otap/config.rs) | `PluginConfig` + `resolve()` — the 5-sketch `sketch_type` dispatch table (factory + observer + `SketchType`). | -| [`decode.rs`](./asap-precompute-rs/src/otap/decode.rs) / [`encode.rs`](./asap-precompute-rs/src/otap/encode.rs) | Standalone Arrow codec: `RecordBatch ↔ Vec` / `[SketchEnvelope]`, keyed on well-known columns + Strategy-B `_asap_*` carrier keys ([`schema.rs`](./asap-precompute-rs/src/otap/schema.rs)). | -| [`records.rs`](./asap-precompute-rs/src/otap/records.rs) | `OtapMetricRecords` + `flatten`/`lift` — projects an `OtapArrowRecords`-shaped sibling-batch family (metrics + per-row attribute child batch joined by `parent_id`) ↔ flat `RecordBatch`. Real-OTAP-free; same scope note as above. | - -All of this sits on the host-neutral runtime in the same crate -(`precompute`, `window`, `snapshot_cache`, `sketches/*`). +[`config.rs`](./asap-precompute-rs/src/otap/config.rs) holds `PluginConfig` +and `resolve()` — the `sketch_type` dispatch table (factory + observer + +`SketchType`). All of this sits on the host-neutral runtime in the same +crate (`precompute`, `window`, `sketches/*`). ## Build & test -For a presentation-ready walkthrough of the real YAML pipeline integration, -see the [live OTAP pipeline demo guide](./docs/demo-guide.md). The guide also -includes a [Prometheus + Grafana setup](./docs/demo-guide.md#write-kll-estimates-to-prometheus-and-view-them-in-grafana) -that writes the KLL p50/p99 estimates to Prometheus and displays them in a -provisioned Grafana dashboard. For a continuous KLL vs exact comparison with -every processor in its own pinned process, see the -[multi-process streaming KLL demo](./docs/kll-windows.md). +For a continuous KLL vs exact comparison with every processor in its own +pinned process, see the [multi-process streaming KLL demo](./docs/kll-windows.md). ```sh cd asap-precompute-rs -cargo build --features otap-engine -cargo test --features otap-engine # runtime + codec + lifecycle + real pipeline tests -cargo clippy --all-targets --features otap-engine -- -D warnings +cargo test --features otap-engine,stream-benchmark # runtime + codec + real pipeline tests +cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings cargo fmt --check ``` The default (no-feature) build excludes Arrow/Tokio and compiles the -row-oriented runtime only; `otap` turns on the Arrow codec + legacy -transport + plugin lifecycle; `otap-engine` additionally turns on the +row-oriented runtime only; `otap` turns on the plugin config; +`otap-engine` additionally turns on the real OTAP node (and pulls in the real OTAP Dataflow crates — a heavier, slower build: datafusion, tonic, and the rest of that dependency tree). @@ -114,20 +104,13 @@ a specific commit in `Cargo.toml`. ## Status -Phases **B** (Arrow codec) and **C** (five-sketch runtime) are complete and -tested. Phase **D** (the `linkme` registration + the +The sketch runtime is complete and tested. Phase **D** (the `linkme` registration + the real `OtapPdata` binding) is complete and build/lint/test-verified by this repo's own build. `tests/otap_pipeline_e2e.rs` parses a genuine OTAP pipeline YAML, builds it through `OTAP_PIPELINE_FACTORY`, runs the resulting `RuntimePipeline`, injects real scalar `OtapPdata`, and verifies a two-source, four-processor creation → fan-in merge → estimation flow at the configured exporter. -Run the same live pipeline as a terminal demo with: - -```sh -cd asap-precompute-rs -cargo run --bin asap-otap-demo --features otap-engine -``` The real OTAP adapter merges an upstream processor's self-describing sketch directly from `OtapPdata`. The multi-process demo uses standard OTLP metrics diff --git a/asap-precompute-rs/benches/asap_sketch_pipeline.rs b/asap-precompute-rs/benches/asap_sketch_pipeline.rs deleted file mode 100644 index f1b431a..0000000 --- a/asap-precompute-rs/benches/asap_sketch_pipeline.rs +++ /dev/null @@ -1,398 +0,0 @@ -//! Nightly traffic -> SUT -> backend benchmark. -//! -//! All three paths consume the same pre-generated -//! native `OtapPdata`, decode it at ingress, encode their results as native -//! pdata, and decode/validate those results in the simulated backend. - -use asap_precompute_rs::envelope::{Encoding, SketchEnvelope, SketchType}; -use asap_precompute_rs::observation::{KeyValue, Observation}; -use asap_precompute_rs::otap::codec::{decode_pdata_to_observations, encode_envelopes_to_pdata}; -use asap_precompute_rs::precompute::{QuantileSketch, Sketch}; -use asap_precompute_rs::sketches::KLLWrapper; -use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; -use otel_arrow_dfe_otap::pdata::OtapPdata; -use std::hint::black_box; - -const SIGNAL_COUNTS: [usize; 3] = [1_024, 16_384, 131_072]; -const SOURCES: usize = 2; -const INPUT_BATCH_SIZE: usize = 4_096; - -fn semantic_http_durations(count: usize) -> Vec { - (0..count) - .map(|i| { - // Deterministic SplitMix64 output gives the duration field the - // high-diversity continuous distribution produced by a traffic - // generator, without making benchmark runs non-reproducible. - let mut mixed = (i as u64).wrapping_add(0x9e37_79b9_7f4a_7c15); - mixed = (mixed ^ (mixed >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9); - mixed = (mixed ^ (mixed >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb); - mixed ^= mixed >> 31; - let unit = (mixed >> 11) as f64 / (1_u64 << 53) as f64; - if i % 20 == 0 { - 1.0 + unit * 0.4 - } else { - 0.005 + unit * 0.1 - } - }) - .collect() -} - -fn scalar_envelope(value: f64) -> SketchEnvelope { - SketchEnvelope { - schema_version: 1, - sketch_type: SketchType::Unspecified, - agg_id: 0, - resource_labels: vec![KeyValue::new("service.name", "checkout")], - labels: vec![ - KeyValue::new("http.request.method", "GET"), - KeyValue::new("http.route", "/checkout"), - KeyValue::new("http.response.status_code", "200"), - ], - window_start_ms: 0, - window_end_ms: 1, - encoding: Encoding::Unspecified, - payload: vec![], - hash_spec: None, - metric_name: "http.server.request.duration".to_owned(), - count: 0, - aggregation_temporality: 0, - value, - } -} - -fn encode_values(values: &[f64]) -> Vec { - values - .chunks(INPUT_BATCH_SIZE) - .map(|batch| { - let envelopes = batch - .iter() - .copied() - .map(scalar_envelope) - .collect::>(); - encode_envelopes_to_pdata(&envelopes).expect("traffic generator pdata") - }) - .collect() -} - -fn generated_pdata(values: &[f64]) -> [Vec; SOURCES] { - let midpoint = values.len() / 2; - [ - encode_values(&values[..midpoint]), - encode_values(&values[midpoint..]), - ] -} - -fn exact_quantiles(values: &[f64]) -> (f64, f64) { - let mut sorted = values.to_vec(); - sorted.sort_by(f64::total_cmp); - let at = |q: f64| sorted[((sorted.len() - 1) as f64 * q).round() as usize]; - (at(0.5), at(0.99)) -} - -fn quantile_pdata(p50: f64, p99: f64) -> OtapPdata { - let output = [("0.5", p50), ("0.99", p99)] - .into_iter() - .map(|(quantile, value)| { - let mut envelope = scalar_envelope(value); - envelope.metric_name = "http.server.request.duration.estimate".into(); - envelope.labels.push(KeyValue::new("quantile", quantile)); - envelope - }) - .collect::>(); - encode_envelopes_to_pdata(&output).expect("quantile output pdata") -} - -fn backend_decode(pdata: OtapPdata) -> Vec { - decode_pdata_to_observations(pdata) - .expect("simulated backend decode") - .observations -} - -fn pass_stage(input: Vec) -> Vec { - let mut output_batches = Vec::new(); - for batch in input { - let observations = backend_decode(batch); - let output = observations - .iter() - .map(|observation| { - let mut envelope = scalar_envelope(observation.value.float); - envelope.resource_labels = observation.resource_labels.clone(); - envelope.labels = observation.labels.clone(); - envelope.metric_name = observation.metric.clone(); - envelope - }) - .collect::>(); - output_batches.push(encode_envelopes_to_pdata(&output).expect("pass output pdata")); - } - output_batches -} - -fn control_pipeline(input: [Vec; SOURCES]) -> Vec { - let [a, b] = input; - let mut joined = pass_stage(a); - joined.extend(pass_stage(b)); - let output = pass_stage(pass_stage(joined)); - let mut backend = Vec::new(); - for batch in output { - backend.extend(backend_decode(batch)); - } - backend -} - -fn decoded_values(input: Vec) -> Vec { - input - .into_iter() - .flat_map(backend_decode) - .map(|o| o.value.float) - .collect() -} - -fn sort_stage(input: Vec) -> Vec { - let mut values = decoded_values(input); - values.sort_by(f64::total_cmp); - encode_values(&values) -} - -fn merge_sorted_stage(left: Vec, right: Vec) -> Vec { - let (left, right) = (decoded_values(left), decoded_values(right)); - let (mut a, mut b) = (0, 0); - let mut merged = Vec::with_capacity(left.len() + right.len()); - while a < left.len() && b < right.len() { - if left[a].total_cmp(&right[b]).is_le() { - merged.push(left[a]); - a += 1; - } else { - merged.push(right[b]); - b += 1; - } - } - merged.extend_from_slice(&left[a..]); - merged.extend_from_slice(&right[b..]); - encode_values(&merged) -} - -fn exact_estimate_stage(input: Vec) -> OtapPdata { - let values = decoded_values(input); - let at = |q: f64| values[((values.len() - 1) as f64 * q).round() as usize]; - quantile_pdata(at(0.5), at(0.99)) -} - -fn otap_exact_quantile_pipeline(input: [Vec; SOURCES]) -> Vec { - let [a, b] = input; - backend_decode(exact_estimate_stage(merge_sorted_stage( - sort_stage(a), - sort_stage(b), - ))) -} - -fn kll_create_stage(input: Vec, seed: u64) -> OtapPdata { - let mut sketch = KLLWrapper::new(400, Some(seed)).with_wire_encoding(Encoding::Msgpack); - let mut count = 0; - for batch in input { - for observation in backend_decode(batch) { - sketch.update(observation.value.float); - count += 1; - } - } - let mut env = scalar_envelope(0.0); - env.sketch_type = SketchType::KLLSketch; - env.agg_id = 7; - env.encoding = Encoding::Msgpack; - env.payload = sketch.snapshot().expect("KLL snapshot"); - env.count = count; - env.metric_name = "http.server.request.duration.sketch".into(); - encode_envelopes_to_pdata(&[env]).expect("KLL create pdata") -} - -fn decode_kll(input: OtapPdata) -> KLLWrapper { - let obs = backend_decode(input).pop().expect("one sketch"); - let env = obs.value.envelope.expect("sketch envelope"); - let mut sketch = KLLWrapper::new(400, Some(99)).with_wire_encoding(Encoding::Msgpack); - sketch - .apply_delta_encoded(&env.payload, env.encoding) - .expect("ASAPv1 decode"); - sketch -} - -fn kll_merge_stage(left: OtapPdata, right: OtapPdata) -> OtapPdata { - let mut merged = KLLWrapper::new(400, Some(99)).with_wire_encoding(Encoding::Msgpack); - merged.merge(&decode_kll(left)).expect("merge left"); - merged.merge(&decode_kll(right)).expect("merge right"); - let mut env = scalar_envelope(0.0); - env.sketch_type = SketchType::KLLSketch; - env.agg_id = 7; - env.encoding = Encoding::Msgpack; - env.payload = merged.snapshot().expect("merged snapshot"); - env.metric_name = "http.server.request.duration.merged_sketch".into(); - encode_envelopes_to_pdata(&[env]).expect("KLL merge pdata") -} - -fn asap_pipeline(input: [Vec; SOURCES]) -> Vec { - let [a, b] = input; - let merged = kll_merge_stage(kll_create_stage(a, 1), kll_create_stage(b, 2)); - let sketch = decode_kll(merged); - backend_decode(quantile_pdata(sketch.quantile(0.5), sketch.quantile(0.99))) -} - -fn assert_control(input: &[Vec; SOURCES], values: &[f64]) { - let output = control_pipeline(input.clone()); - assert_eq!(output.len(), values.len()); - assert_eq!(output[0].value.float, values[0]); - assert!(output[0] - .resource_labels - .contains(&KeyValue::new("service.name", "checkout"))); - assert!(output[0] - .labels - .contains(&KeyValue::new("http.route", "/checkout"))); -} - -fn assert_quantile_output( - pipeline: &str, - output: Vec, - exact: (f64, f64), - maximum_error: f64, -) { - assert_eq!(output.len(), 2, "{pipeline} output shape"); - for (name, quantile, want) in [("p50", "0.5", exact.0), ("p99", "0.99", exact.1)] { - let observation = output - .iter() - .find(|observation| { - observation - .labels - .contains(&KeyValue::new("quantile", quantile)) - }) - .unwrap_or_else(|| panic!("{pipeline} missing {name}")); - assert_eq!(observation.metric, "http.server.request.duration.estimate"); - assert!(observation - .resource_labels - .contains(&KeyValue::new("service.name", "checkout"))); - let got = observation.value.float; - let relative_error = (got - want).abs() / want.max(f64::EPSILON); - assert!( - relative_error <= maximum_error, - "{pipeline} {name}: got {got}, want {want}" - ); - } -} - -fn requested(scenario: &str, count: usize) -> bool { - let filters = std::env::args() - .skip(1) - .filter(|arg| !arg.starts_with('-')) - .collect::>(); - filters.is_empty() - || filters - .iter() - .any(|filter| filter.contains(&format!("{scenario}/{count}"))) -} - -fn assert_exact(input: &[Vec; SOURCES], values: &[f64]) { - let exact = exact_quantiles(values); - assert_quantile_output( - "OTAP exact", - otap_exact_quantile_pipeline(input.clone()), - exact, - 0.0, - ); -} - -fn assert_kll(input: &[Vec; SOURCES], values: &[f64]) { - assert_quantile_output( - "ASAP KLL", - asap_pipeline(input.clone()), - exact_quantiles(values), - 0.05, - ); -} - -fn benchmark(c: &mut Criterion) { - let mut group = c.benchmark_group("http.server.request.duration"); - for count in SIGNAL_COUNTS { - let run_control = requested("otap_control_pipeline", count); - let run_exact = requested("otap_exact_quantile_pipeline", count); - let run_kll = requested("asap_kll_pipeline", count); - if !run_control && !run_exact && !run_kll { - continue; - } - let values = semantic_http_durations(count); - let input = generated_pdata(&values); - group.throughput(Throughput::Elements(count as u64)); - if run_exact { - assert_exact(&input, &values); - group.bench_with_input( - BenchmarkId::new("otap_exact_quantile_pipeline", count), - &input, - |b, p| b.iter(|| black_box(otap_exact_quantile_pipeline(black_box(p.clone())))), - ); - } - if run_control { - assert_control(&input, &values); - group.bench_with_input( - BenchmarkId::new("otap_control_pipeline", count), - &input, - |b, p| b.iter(|| black_box(control_pipeline(black_box(p.clone())))), - ); - } - if run_kll { - assert_kll(&input, &values); - group.bench_with_input( - BenchmarkId::new("asap_kll_pipeline", count), - &input, - |b, p| b.iter(|| black_box(asap_pipeline(black_box(p.clone())))), - ); - } - } - group.finish(); -} - -fn benchmark_kll_stages(c: &mut Criterion) { - let mut group = c.benchmark_group("kll_isolated_stages"); - for count in SIGNAL_COUNTS { - let values = semantic_http_durations(count); - let make_sketch = |slice: &[f64], seed| { - let mut sketch = KLLWrapper::new(400, Some(seed)).with_wire_encoding(Encoding::Msgpack); - for value in slice { - sketch.update(*value); - } - sketch - }; - let left = make_sketch(&values[..count / 2], 1); - let right = make_sketch(&values[count / 2..], 2); - group.throughput(Throughput::Elements(count as u64)); - group.bench_function(BenchmarkId::new("create", count), |b| { - b.iter(|| { - let sketch = make_sketch(black_box(&values), 3); - black_box(sketch); - }) - }); - group.bench_function(BenchmarkId::new("merge", count), |b| { - b.iter(|| { - let mut merged = - KLLWrapper::new(400, Some(3)).with_wire_encoding(Encoding::Msgpack); - merged.merge(black_box(&left)).unwrap(); - merged.merge(black_box(&right)).unwrap(); - black_box(merged); - }) - }); - group.throughput(Throughput::Elements((count / 2) as u64)); - group.bench_function(BenchmarkId::new("estimate", count), |b| { - b.iter(|| { - black_box(( - left.quantile(black_box(0.5)), - left.quantile(black_box(0.99)), - )); - }) - }); - group.bench_function(BenchmarkId::new("serialize_asapv1", count), |b| { - b.iter(|| { - let bytes = left.snapshot().unwrap(); - assert!(bytes.starts_with(b"ASAPv1")); - black_box(bytes); - }) - }); - } - group.finish(); -} - -criterion_group!(benches, benchmark, benchmark_kll_stages); -criterion_main!(benches); diff --git a/asap-precompute-rs/src/adapter.rs b/asap-precompute-rs/src/adapter.rs deleted file mode 100644 index c35450a..0000000 --- a/asap-precompute-rs/src/adapter.rs +++ /dev/null @@ -1,64 +0,0 @@ -//! [`Adapter`] trait. -//! -//! Adapters are the Layer-4 platform shims (OTel / Telegraf / -//! Vector / OTAP) that translate their host's native event into -//! [`crate::observation::Observation`], hand it to a -//! [`crate::precompute::Precompute`], and translate the runtime's -//! emitted [`crate::envelope::SketchEnvelope`] back to the host's -//! native event. - -use std::time::Duration; - -use crate::envelope::SketchEnvelope; -use crate::observation::Observation; -use crate::precompute::{PrecomputeError, StatsSnapshot}; - -/// Cancel handle returned by [`Adapter::schedule_tick`]. Dropping -/// the handle stops the periodic callback. -/// -/// Boxed `FnOnce` so adapters can return any cancellation -/// closure (channel send, atomic flag flip, JoinHandle::abort, -/// etc.). -pub type CancelTick = Box; - -/// Contract every Layer-4 platform shim implements. -/// -/// The associated `Event` type is the host's native event / data -/// model: -/// - OTel: `pmetric::Metrics` -/// - Telegraf: `telegraf::Metric` -/// - Vector: `vector::Event` -/// - OTAP: `arrow::RecordBatch` -/// -/// Using an associated type keeps each adapter monomorphically typed -/// without any runtime type-erasure dance. -pub trait Adapter { - /// Host's native event / data model. - type Event; - - /// Converts a host-native event into one or more host-neutral - /// [`Observation`]s. - /// - /// MUST recognize sketch-typed inputs and produce - /// [`crate::observation::ObservationValueKind::Envelope`] rather - /// than expanding them to scalars (bandwidth invariant). - fn decode(&self, ev: Self::Event) -> Result, PrecomputeError>; - - /// Converts a slice of host-neutral [`SketchEnvelope`]s back - /// into a host-native event suitable for handoff to the host's - /// downstream pipeline. - fn encode(&self, envelopes: &[SketchEnvelope]) -> Result; - - /// Installs a periodic callback at `period`. The callback runs - /// on a thread / task the adapter owns; returns a cancel handle - /// to stop the worker on shutdown. - /// - /// The callback typically calls - /// [`crate::precompute::Precompute::tick`] and feeds output - /// back via [`Self::encode`] + the host's downstream dispatch. - fn schedule_tick(&self, period: Duration, cb: Box) -> CancelTick; - - /// Publishes the [`crate::precompute::Precompute`]'s runtime - /// stats through the host's native metric channel. - fn emit_telemetry(&self, stats: &StatsSnapshot); -} diff --git a/asap-precompute-rs/src/bin/asap_otap_demo.rs b/asap-precompute-rs/src/bin/asap_otap_demo.rs deleted file mode 100644 index fed0671..0000000 --- a/asap-precompute-rs/src/bin/asap_otap_demo.rs +++ /dev/null @@ -1,1066 +0,0 @@ -//! Multi-process OTAP sketch create -> merge -> estimate demonstration. -//! Each ASAP processor runs in its own OS process and OTAP RuntimePipeline. - -#[path = "demo/prometheus_export.rs"] -mod prometheus_export; - -use asap_precompute_rs::envelope::{Encoding, SketchEnvelope, SketchType}; -use asap_precompute_rs::observation::KeyValue; -use asap_precompute_rs::otap::codec::{decode_pdata_to_observations, encode_envelopes_to_pdata}; -use async_trait::async_trait; -use linkme::distributed_slice; -use otel_arrow_dfe_config::node::NodeUserConfig; -use otel_arrow_dfe_config::observed_state::{ObservedStateSettings, SendPolicy}; -use otel_arrow_dfe_config::pipeline::PipelineConfig; -use otel_arrow_dfe_config::policy::{ChannelCapacityPolicy, TelemetryPolicy}; -use otel_arrow_dfe_config::{DeployedPipelineKey, PipelineGroupId, PipelineId, SignalType}; -use otel_arrow_dfe_core_nodes as _; -use otel_arrow_dfe_engine::capability::registry::Capabilities; -use otel_arrow_dfe_engine::config::{ExporterConfig, ProcessorConfig, ReceiverConfig}; -use otel_arrow_dfe_engine::context::{ControllerContext, PipelineContext}; -use otel_arrow_dfe_engine::control::{ - pipeline_completion_msg_channel, runtime_ctrl_msg_channel, NodeControlMsg, RuntimeControlMsg, -}; -use otel_arrow_dfe_engine::error::ProcessorErrorKind; -use otel_arrow_dfe_engine::exporter::ExporterWrapper; -use otel_arrow_dfe_engine::local::{ - exporter as local_exporter, processor as local_processor, receiver as local_receiver, -}; -use otel_arrow_dfe_engine::message::{ExporterInbox, Message}; -use otel_arrow_dfe_engine::node::NodeId; -use otel_arrow_dfe_engine::processor::ProcessorWrapper; -use otel_arrow_dfe_engine::receiver::ReceiverWrapper; -use otel_arrow_dfe_engine::terminal_state::TerminalState; -use otel_arrow_dfe_engine::{ - ExporterFactory, MessageSourceLocalEffectHandlerExtension, ProcessorFactory, ReceiverFactory, -}; -use otel_arrow_dfe_otap::pdata::OtapPdata; -use otel_arrow_dfe_otap::{ - OTAP_EXPORTER_FACTORIES, OTAP_PIPELINE_FACTORY, OTAP_PROCESSOR_FACTORIES, - OTAP_RECEIVER_FACTORIES, -}; -use otel_arrow_dfe_pdata::{OtlpProtoBytes, TryIntoWithOptions}; -use otel_arrow_dfe_state::store::ObservedStateStore; -use otel_arrow_dfe_telemetry::InternalTelemetrySystem; -use serde::{Deserialize, Serialize}; -use std::path::{Path, PathBuf}; -use std::process::{Child, Command}; -use std::sync::{Arc, Condvar, Mutex, OnceLock}; -use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; -use std::{env, fs}; - -const WORKER_ARG: &str = "--df-worker"; -const SOURCE_URN: &str = "urn:asap:receiver:otlp_file"; -const SINK_URN: &str = "urn:asap:exporter:otlp_file"; -const BASELINE_URN: &str = "urn:asap:processor:benchmark_baseline"; -const TRAFFIC_SEED: u64 = 0x9e37_79b9_7f4a_7c15; -static INPUTS: OnceLock> = OnceLock::new(); -static CAPTURED: OnceLock<(Mutex>, Condvar)> = OnceLock::new(); -fn captured() -> &'static (Mutex>, Condvar) { - CAPTURED.get_or_init(|| (Mutex::new(Vec::new()), Condvar::new())) -} - -fn scalar_input(start: u64, end: u64) -> OtapPdata { - let values = (start..=end) - .map(|value| SketchEnvelope { - schema_version: 1, - sketch_type: SketchType::Unspecified, - agg_id: 0, - resource_labels: vec![KeyValue::new("service.name", "checkout")], - labels: vec![ - KeyValue::new("http.request.method", "GET"), - KeyValue::new("http.route", "/checkout"), - KeyValue::new("http.response.status_code", "200"), - ], - window_start_ms: 1_000, - window_end_ms: 2_000, - encoding: Encoding::Unspecified, - payload: vec![], - hash_spec: None, - metric_name: "http.server.request.duration".into(), - count: 0, - aggregation_temporality: 0, - value: value as f64, - }) - .collect::>(); - encode_envelopes_to_pdata(&values).expect("encode input") -} - -// SplitMix64 gives reproducible, high-cardinality durations with a long tail. -fn traffic_value(index: u64) -> f64 { - let mut x = index.wrapping_add(TRAFFIC_SEED); - x = (x ^ (x >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9); - x = (x ^ (x >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb); - x ^= x >> 31; - let unit = (x >> 11) as f64 / (1_u64 << 53) as f64; - if index.is_multiple_of(20) { - 1.0 + unit * 0.4 - } else { - 0.005 + unit * 0.1 - } -} - -fn generated_input(start: u64, end: u64, threads: usize, semantic: bool) -> OtapPdata { - if !semantic { - return scalar_input(start, end); - } - let count = (end - start + 1) as usize; - let chunk = count.div_ceil(threads); - let mut handles = Vec::new(); - for thread in 0..threads { - let first = start + (thread * chunk) as u64; - if first > end { - break; - } - let last = (first + chunk as u64 - 1).min(end); - handles.push(std::thread::spawn(move || { - (first..=last).map(traffic_value).collect::>() - })); - } - let values = handles - .into_iter() - .flat_map(|handle| handle.join().unwrap()) - .collect::>(); - assert_eq!(values.len(), count); - let envelopes = values - .into_iter() - .map(|value| SketchEnvelope { - schema_version: 1, - sketch_type: SketchType::Unspecified, - agg_id: 0, - resource_labels: vec![KeyValue::new("service.name", "checkout")], - labels: vec![ - KeyValue::new("http.request.method", "GET"), - KeyValue::new("http.route", "/checkout"), - KeyValue::new("http.response.status_code", "200"), - ], - window_start_ms: 1_000, - window_end_ms: 2_000, - encoding: Encoding::Unspecified, - payload: vec![], - hash_spec: None, - metric_name: "http.server.request.duration".into(), - count: 0, - aggregation_temporality: 0, - value, - }) - .collect::>(); - encode_envelopes_to_pdata(&envelopes).expect("encode generated input") -} - -#[derive(Serialize)] -struct StageMetrics { - elapsed_nanoseconds: u128, - cpu_nanoseconds: u128, - peak_rss_kib: u64, - serialization_nanoseconds: u128, - output_bytes: u64, -} - -fn process_cpu_nanoseconds() -> Result { - let mut stamp = std::mem::MaybeUninit::::uninit(); - if unsafe { libc::clock_gettime(libc::CLOCK_PROCESS_CPUTIME_ID, stamp.as_mut_ptr()) } != 0 { - return Err(format!( - "read process CPU clock: {}", - std::io::Error::last_os_error() - )); - } - let stamp = unsafe { stamp.assume_init() }; - Ok(stamp.tv_sec as u128 * 1_000_000_000 + stamp.tv_nsec as u128) -} - -fn write_stage_metrics( - output: &Path, - start: Instant, - cpu_start: u128, - serialization_nanoseconds: u128, -) -> Result<(), String> { - let status = fs::read_to_string("/proc/self/status").map_err(|e| e.to_string())?; - let peak_rss_kib = status - .lines() - .find_map(|line| line.strip_prefix("VmHWM:")) - .and_then(|value| value.split_whitespace().next()) - .and_then(|value| value.parse().ok()) - .unwrap_or(0); - let metrics = StageMetrics { - elapsed_nanoseconds: start.elapsed().as_nanos(), - cpu_nanoseconds: process_cpu_nanoseconds()?.saturating_sub(cpu_start), - peak_rss_kib, - serialization_nanoseconds, - output_bytes: fs::metadata(output).map_err(|e| e.to_string())?.len(), - }; - fs::write( - output.with_extension("metrics.json"), - serde_json::to_vec(&metrics).map_err(|e| e.to_string())?, - ) - .map_err(|e| e.to_string()) -} - -fn write_otlp(path: &Path, pdata: OtapPdata) -> Result<(), String> { - let (_, payload) = pdata.into_parts(); - let encoded = <_ as TryIntoWithOptions>::try_into_with_default(payload) - .map_err(|e| e.to_string())?; - match encoded { - OtlpProtoBytes::ExportMetricsRequest(bytes) => { - fs::write(path, bytes).map_err(|e| e.to_string()) - } - _ => Err("non-metrics signal".into()), - } -} -fn read_otlp(path: &Path) -> Result { - let bytes = fs::read(path).map_err(|e| e.to_string())?; - Ok(OtapPdata::new_todo_context( - OtlpProtoBytes::new_from_bytes(SignalType::Metrics, bytes).into(), - )) -} - -struct FileReceiver; -#[async_trait(?Send)] -impl local_receiver::Receiver for FileReceiver { - async fn start( - self: Box, - mut ctrl: local_receiver::ControlChannel, - effects: local_receiver::EffectHandler, - ) -> Result { - for path in INPUTS.get().expect("inputs initialized") { - effects - .send_message_with_source_node(read_otlp(path).expect("read OTLP input")) - .await?; - } - loop { - match ctrl.recv().await { - Ok(NodeControlMsg::Shutdown { .. }) | Err(_) => break, - Ok(_) => {} - } - } - Ok(TerminalState::default()) - } -} -fn create_source( - _ctx: PipelineContext, - node: NodeId, - config: Arc, - runtime: &ReceiverConfig, - _caps: &Capabilities, -) -> Result, otel_arrow_dfe_config::error::Error> { - Ok(ReceiverWrapper::local(FileReceiver, node, config, runtime)) -} -#[distributed_slice(OTAP_RECEIVER_FACTORIES)] -static SOURCE_FACTORY: ReceiverFactory = ReceiverFactory { - name: SOURCE_URN, - create: create_source, - wiring_contract: otel_arrow_dfe_engine::wiring_contract::WiringContract::UNRESTRICTED, - validate_config: otel_arrow_dfe_config::validation::no_config, -}; - -struct FileExporter; -#[async_trait(?Send)] -impl local_exporter::Exporter for FileExporter { - async fn start( - self: Box, - mut inbox: ExporterInbox, - _effects: local_exporter::EffectHandler, - ) -> Result { - loop { - match inbox.recv().await? { - Message::PData(pdata) => { - let (outputs, ready) = captured(); - outputs.lock().expect("capture mutex").push(pdata); - ready.notify_all(); - } - Message::Control(NodeControlMsg::Shutdown { .. }) => break, - Message::Control(_) => {} - } - } - Ok(TerminalState::default()) - } -} -fn create_sink( - _ctx: PipelineContext, - node: NodeId, - config: Arc, - runtime: &ExporterConfig, - _caps: &Capabilities, -) -> Result, otel_arrow_dfe_config::error::Error> { - Ok(ExporterWrapper::local(FileExporter, node, config, runtime)) -} -#[distributed_slice(OTAP_EXPORTER_FACTORIES)] -static SINK_FACTORY: ExporterFactory = ExporterFactory { - name: SINK_URN, - create: create_sink, - wiring_contract: otel_arrow_dfe_engine::wiring_contract::WiringContract::UNRESTRICTED, - validate_config: otel_arrow_dfe_config::validation::no_config, -}; - -#[derive(Clone, Deserialize)] -#[serde(deny_unknown_fields)] -struct BaselineConfig { - operation: String, - #[serde(default = "one_input")] - expected_inputs: usize, -} - -const fn one_input() -> usize { - 1 -} - -struct BaselineProcessor { - config: BaselineConfig, - values: Vec, - runs: Vec>, - received: usize, -} - -fn merge_sorted_runs(left: &[f64], right: &[f64]) -> Vec { - let (mut left_index, mut right_index) = (0, 0); - let mut merged = Vec::with_capacity(left.len() + right.len()); - while left_index < left.len() && right_index < right.len() { - if left[left_index].total_cmp(&right[right_index]).is_le() { - merged.push(left[left_index]); - left_index += 1; - } else { - merged.push(right[right_index]); - right_index += 1; - } - } - merged.extend_from_slice(&left[left_index..]); - merged.extend_from_slice(&right[right_index..]); - merged -} - -fn values_from_pdata(pdata: OtapPdata) -> Result, String> { - Ok(decode_pdata_to_observations(pdata) - .map_err(|error| error.to_string())? - .observations - .into_iter() - .map(|observation| observation.value.float) - .collect()) -} - -fn values_pdata(values: &[f64], metric_name: &str) -> Result { - let envelopes = values - .iter() - .copied() - .map(|value| SketchEnvelope { - schema_version: 1, - sketch_type: SketchType::Unspecified, - agg_id: 0, - resource_labels: vec![KeyValue::new("service.name", "checkout")], - labels: vec![ - KeyValue::new("http.request.method", "GET"), - KeyValue::new("http.route", "/checkout"), - KeyValue::new("http.response.status_code", "200"), - ], - window_start_ms: 1_000, - window_end_ms: 2_000, - encoding: Encoding::Unspecified, - payload: vec![], - hash_spec: None, - metric_name: metric_name.into(), - count: 0, - aggregation_temporality: 0, - value, - }) - .collect::>(); - encode_envelopes_to_pdata(&envelopes).map_err(|error| error.to_string()) -} - -fn exact_quantile_pdata(p50: f64, p99: f64) -> Result { - let mut envelopes = Vec::new(); - for (quantile, value) in [("0.5", p50), ("0.99", p99)] { - let mut envelope = SketchEnvelope { - schema_version: 1, - sketch_type: SketchType::Unspecified, - agg_id: 0, - resource_labels: vec![KeyValue::new("service.name", "checkout")], - labels: vec![KeyValue::new("http.route", "/checkout")], - window_start_ms: 1_000, - window_end_ms: 2_000, - encoding: Encoding::Unspecified, - payload: vec![], - hash_spec: None, - metric_name: "request.duration.exact_estimate".into(), - count: 0, - aggregation_temporality: 0, - value, - }; - envelope.labels.push(KeyValue::new("quantile", quantile)); - envelopes.push(envelope); - } - encode_envelopes_to_pdata(&envelopes).map_err(|error| error.to_string()) -} - -#[async_trait(?Send)] -impl local_processor::Processor for BaselineProcessor { - async fn process( - &mut self, - msg: Message, - effects: &mut local_processor::EffectHandler, - ) -> Result<(), otel_arrow_dfe_engine::error::Error> { - if let Message::PData(pdata) = msg { - if self.config.operation == "pass" && self.config.expected_inputs == 1 { - effects.send_message_with_source_node(pdata).await?; - return Ok(()); - } - let mut values = values_from_pdata(pdata).map_err(|error| { - otel_arrow_dfe_engine::error::Error::ProcessorError { - processor: effects.processor_id(), - kind: ProcessorErrorKind::Other, - error, - source_detail: String::new(), - } - })?; - if self.config.operation == "sort" { - values.sort_by(f64::total_cmp); - } - if self.config.operation == "merge_sorted" { - self.runs.push(values); - } else { - self.values.extend(values); - } - self.received += 1; - if self.received == self.config.expected_inputs { - if self.config.operation == "merge_sorted" { - self.values = merge_sorted_runs(&self.runs[0], &self.runs[1]); - } - let output = if self.config.operation == "estimate" { - let at = - |q: f64| self.values[((self.values.len() - 1) as f64 * q).round() as usize]; - exact_quantile_pdata(at(0.5), at(0.99)) - } else { - values_pdata(&self.values, "request.duration") - } - .map_err(|error| { - otel_arrow_dfe_engine::error::Error::ProcessorError { - processor: effects.processor_id(), - kind: ProcessorErrorKind::Other, - error, - source_detail: String::new(), - } - })?; - effects.send_message_with_source_node(output).await?; - } - } - Ok(()) - } -} - -fn create_baseline( - _ctx: PipelineContext, - node: NodeId, - config: Arc, - runtime: &ProcessorConfig, - _caps: &Capabilities, -) -> Result, otel_arrow_dfe_config::error::Error> { - let parsed = serde_json::from_value(config.config.clone()).map_err(|error| { - otel_arrow_dfe_config::error::Error::InvalidUserConfig { - error: error.to_string(), - } - })?; - Ok(ProcessorWrapper::local( - BaselineProcessor { - config: parsed, - values: Vec::new(), - runs: Vec::new(), - received: 0, - }, - node, - config, - runtime, - )) -} - -#[distributed_slice(OTAP_PROCESSOR_FACTORIES)] -static BASELINE_FACTORY: ProcessorFactory = ProcessorFactory { - name: BASELINE_URN, - create: create_baseline, - wiring_contract: otel_arrow_dfe_engine::wiring_contract::WiringContract::UNRESTRICTED, - validate_config: otel_arrow_dfe_config::validation::no_config, -}; - -fn pipeline_yaml(role: &str, trace: &Path) -> Result { - let processor = match role { - "create_a" | "create_b" => r#"type: "urn:asap:processor:asap_sketches" - config: - sketch_type: "kll" - encoding: "Msgpack" - window_size: "20ms" - output_metric_name: "request.duration.sketch" - agg_id: 7 - sketch_params: { k: 400 } - transmit_sketch: true - quantiles: []"# - .to_owned(), - "merge" | "estimate" => { - let (name, transmit, quantiles) = if role == "merge" { - ("request.duration.merged_sketch", true, "[]") - } else { - ("request.duration.estimate", false, "[0.5, 0.99]") - }; - format!( - r#"type: "urn:asap:processor:asap_sketches" - config: - sketch_type: "kll" - encoding: "Msgpack" - window_size: "20ms" - output_metric_name: "{name}" - agg_id: 7 - sketch_params: {{ k: 400 }} - transmit_sketch: {transmit} - quantiles: {quantiles}"# - ) - } - "raw_a" | "raw_b" | "raw_final" => { - format!( - r#"type: "{BASELINE_URN}" - config: {{ operation: "pass", expected_inputs: 1 }}"# - ) - } - "raw_merge" => format!( - r#"type: "{BASELINE_URN}" - config: {{ operation: "pass", expected_inputs: 2 }}"# - ), - "exact_a" | "exact_b" => format!( - r#"type: "{BASELINE_URN}" - config: {{ operation: "sort", expected_inputs: 1 }}"# - ), - "exact_merge" => format!( - r#"type: "{BASELINE_URN}" - config: {{ operation: "merge_sorted", expected_inputs: 2 }}"# - ), - "exact_estimate" => format!( - r#"type: "{BASELINE_URN}" - config: {{ operation: "estimate", expected_inputs: 1 }}"# - ), - _ => return Err(format!("unknown role {role}")), - }; - if env::var("ASAP_BENCH_DISABLE_DEBUG").as_deref() == Ok("1") { - return Ok(format!( - r#" -nodes: - source: {{ type: "{SOURCE_URN}" }} - processor: - {processor} - sink: {{ type: "{SINK_URN}" }} -connections: - - {{ from: source, to: processor }} - - {{ from: processor, to: sink }} -"# - )); - } - let trace = serde_json::to_string(&trace.to_string_lossy()).unwrap(); - Ok(format!( - r#" -nodes: - source: {{ type: "{SOURCE_URN}" }} - processor: - {processor} - debug: - type: "urn:otel:processor:debug" - config: - verbosity: detailed - mode: batch - signals: [metrics] - output: {trace} - sink: {{ type: "{SINK_URN}" }} -connections: - - {{ from: source, to: processor }} - - {{ from: processor, to: debug }} - - {{ from: debug, to: sink }} -"# - )) -} - -fn run_worker( - role: &str, - inputs: Vec, - output: PathBuf, - trace: PathBuf, -) -> Result<(), String> { - let stage_start = Instant::now(); - let cpu_start = process_cpu_nanoseconds()?; - INPUTS - .set(inputs) - .map_err(|_| "inputs already initialized".to_owned())?; - captured().0.lock().unwrap().clear(); - let config = PipelineConfig::from_yaml( - "asap-demo".into(), - role.to_owned().into(), - &pipeline_yaml(role, &trace)?, - ) - .map_err(|e| e.to_string())?; - let telemetry = InternalTelemetrySystem::default(); - let ctx = ControllerContext::new(telemetry.registry()).pipeline_context_with( - PipelineGroupId::from("asap-demo"), - PipelineId::from(role.to_owned()), - 0, - 1, - 0, - ); - let entity = ctx.register_pipeline_entity(); - let runtime = OTAP_PIPELINE_FACTORY - .build( - ctx.clone(), - config, - ChannelCapacityPolicy::default(), - TelemetryPolicy::default(), - None, - Default::default(), - None, - None, - ) - .map_err(|e| e.to_string())?; - let policy = ChannelCapacityPolicy::default(); - let (runtime_tx, runtime_rx) = runtime_ctrl_msg_channel(policy.control.pipeline); - let (completion_tx, completion_rx) = pipeline_completion_msg_channel(policy.control.completion); - let observed = ObservedStateStore::new(&ObservedStateSettings::default(), telemetry.registry()); - let shutdown_tx = runtime_tx.clone(); - let shutdown = std::thread::spawn(move || { - let (outputs, ready) = captured(); - let guard = outputs.lock().expect("capture mutex"); - let (_guard, wait) = ready - .wait_timeout_while(guard, Duration::from_secs(60), |items| items.is_empty()) - .expect("capture wait"); - shutdown_tx - .try_send(RuntimeControlMsg::Shutdown { - deadline: Instant::now() + Duration::from_secs(2), - reason: if wait.timed_out() { - "timed out waiting for pipeline output".into() - } else { - "pipeline output complete".into() - }, - }) - .expect("shutdown"); - }); - let key = DeployedPipelineKey { - pipeline_group_id: ctx.pipeline_group_id(), - pipeline_id: ctx.pipeline_id(), - core_id: 0, - deployment_generation: 0, - }; - let (_, pressure_rx) = tokio::sync::watch::channel( - otel_arrow_dfe_engine::memory_limiter::MemoryPressureChanged::initial(), - ); - let _guard = otel_arrow_dfe_engine::entity_context::set_pipeline_entity_key( - ctx.metrics_registry(), - entity, - ); - runtime - .run_forever( - key, - ctx, - observed.reporter(SendPolicy::default()), - telemetry.reporter(), - Duration::from_secs(1), - pressure_rx, - runtime_tx, - runtime_rx, - completion_tx, - completion_rx, - ) - .map_err(|e| e.to_string())?; - shutdown - .join() - .map_err(|_| "shutdown thread panic".to_owned())?; - let mut outputs = captured().0.lock().unwrap(); - if outputs.len() != 1 { - return Err(format!("expected one output, got {}", outputs.len())); - } - let serialization_start = Instant::now(); - write_otlp(&output, outputs.pop().unwrap())?; - write_stage_metrics( - &output, - stage_start, - cpu_start, - serialization_start.elapsed().as_nanos(), - ) -} - -fn spawn(exe: &Path, role: &str, inputs: &[&Path], output: &Path, trace: &Path) -> Child { - let cores = env::var("ASAP_PROCESSOR_CORES").ok(); - let mut cmd = pinned_command(exe, cores.as_deref()); - cmd.arg(WORKER_ARG).arg(role); - for input in inputs { - cmd.arg(input); - } - cmd.arg(output).arg(trace); - let child = cmd.spawn().unwrap_or_else(|e| panic!("launch {role}: {e}")); - println!( - "launched {role} DF processor pid={} trace={}", - child.id(), - trace.display() - ); - child -} - -fn pinned_command(exe: &Path, cores: Option<&str>) -> Command { - if let Some(cores) = cores { - let mut command = Command::new("taskset"); - command.arg("-c").arg(cores).arg(exe); - command - } else { - Command::new(exe) - } -} - -fn spawn_generator(exe: &Path, start: u64, end: u64, output: &Path, threads: usize) -> Child { - let mut cmd = pinned_command(exe, env::var("ASAP_GENERATOR_CORES").ok().as_deref()); - cmd.arg("--generate") - .arg(start.to_string()) - .arg(end.to_string()) - .arg(output) - .arg(threads.to_string()); - cmd.spawn().expect("launch traffic generator") -} -fn wait(mut child: Child, role: &str) { - let status = child.wait().unwrap(); - assert!(status.success(), "{role} failed: {status}"); -} - -fn validate_quantiles(results: &[(String, f64)], expected: [(f64, f64); 2]) -> Result<(), String> { - if results.len() != 2 { - return Err(format!( - "expected p50 and p99, got {} result(s)", - results.len() - )); - } - for (quantile, expected) in expected { - let quantile = quantile.to_string(); - let value = results - .iter() - .find_map(|(label, value)| (label == &quantile).then_some(*value)) - .ok_or_else(|| format!("missing quantile {quantile}"))?; - let tolerance = (expected.abs() * 0.05).max(0.001); - if (value - expected).abs() > tolerance { - return Err(format!( - "quantile {quantile} out of tolerance: got {value}, expected {expected} +/- {tolerance}" - )); - } - } - Ok(()) -} - -#[derive(Serialize)] -struct ProcessorRun { - role: &'static str, - pid: u32, -} - -#[derive(Serialize)] -struct DemoManifest { - parent_pid: u32, - points_per_source: u64, - pipeline_elapsed_nanoseconds: u128, - validation_elapsed_nanoseconds: u128, - traffic_mode: &'static str, - generator_threads: usize, - traffic_seed: u64, - processors: Vec, -} - -struct ParentOptions { - prometheus_otlp_endpoint: Option, - output_dir: Option, - result_manifest: Option, - points_per_source: u64, - scenario: String, - semantic_traffic: bool, - generator_threads: usize, -} - -fn parent_options(args: &[std::ffi::OsString]) -> Result { - let mut options = ParentOptions { - prometheus_otlp_endpoint: None, - output_dir: None, - result_manifest: None, - points_per_source: 100, - scenario: "kll".into(), - semantic_traffic: false, - generator_threads: 1, - }; - let mut index = 0; - while index < args.len() { - let flag = args[index].to_string_lossy(); - let value = args - .get(index + 1) - .ok_or_else(|| format!("missing value for {flag}"))?; - match flag.as_ref() { - "--prometheus-otlp-endpoint" => { - options.prometheus_otlp_endpoint = Some(value.to_string_lossy().into_owned()); - } - "--output-dir" => options.output_dir = Some(PathBuf::from(value)), - "--result-manifest" => options.result_manifest = Some(PathBuf::from(value)), - "--points-per-source" => { - options.points_per_source = value - .to_string_lossy() - .parse() - .map_err(|_| "--points-per-source must be a positive integer".to_owned())?; - if options.points_per_source == 0 { - return Err("--points-per-source must be positive".into()); - } - } - "--scenario" => options.scenario = value.to_string_lossy().into_owned(), - "--traffic" => { - options.semantic_traffic = match value.to_string_lossy().as_ref() { - "sequential" => false, - "semantic" => true, - _ => return Err("--traffic must be sequential or semantic".into()), - } - } - "--generator-threads" => { - options.generator_threads = value - .to_string_lossy() - .parse() - .map_err(|_| "--generator-threads must be positive".to_owned())?; - if options.generator_threads == 0 { - return Err("--generator-threads must be positive".into()); - } - } - _ => return Err(format!("unknown argument {flag}")), - } - index += 2; - } - if options.prometheus_otlp_endpoint.is_some() && options.scenario != "kll" { - return Err("--prometheus-otlp-endpoint requires --scenario kll".into()); - } - Ok(options) -} - -fn run_parent(options: ParentOptions) -> Result<(), String> { - println!("ASAP OTAP multi-process demo: OTLP Metrics boundaries, ASAPv1 sketches"); - let id = SystemTime::now() - .duration_since(UNIX_EPOCH) - .unwrap() - .as_nanos(); - let dir = options.output_dir.unwrap_or_else(|| { - env::temp_dir().join(format!("asap-otap-demo-{}-{id}", std::process::id())) - }); - fs::create_dir_all(&dir).map_err(|e| e.to_string())?; - let p = |name: &str| dir.join(name); - let ia = p("a.otlp"); - let ib = p("b.otlp"); - let sa = p("sa.otlp"); - let sb = p("sb.otlp"); - let merged = p("merged.otlp"); - let out = p("out.otlp"); - let n = options.points_per_source; - let exe = env::current_exe().map_err(|e| e.to_string())?; - let pipeline_start = Instant::now(); - if options.semantic_traffic { - let ga = spawn_generator(&exe, 1, n, &ia, options.generator_threads); - let gb = spawn_generator(&exe, n + 1, n * 2, &ib, options.generator_threads); - wait(ga, "traffic_a"); - wait(gb, "traffic_b"); - } else { - write_otlp(&ia, scalar_input(1, n))?; - write_otlp(&ib, scalar_input(n + 1, n * 2))?; - } - let (role_a, role_b, role_merge, role_estimate) = match options.scenario.as_str() { - "kll" => ("create_a", "create_b", "merge", "estimate"), - "raw" => ("raw_a", "raw_b", "raw_merge", "raw_final"), - "exact" => ("exact_a", "exact_b", "exact_merge", "exact_estimate"), - other => { - return Err(format!( - "unknown scenario {other}; expected raw, exact, or kll" - )) - } - }; - let a = spawn(&exe, role_a, &[&ia], &sa, &p("create-a.debug.log")); - let b = spawn(&exe, role_b, &[&ib], &sb, &p("create-b.debug.log")); - let mut processors = vec![ - ProcessorRun { - role: role_a, - pid: a.id(), - }, - ProcessorRun { - role: role_b, - pid: b.id(), - }, - ]; - wait(a, role_a); - wait(b, role_b); - let m = spawn( - &exe, - role_merge, - &[&sa, &sb], - &merged, - &p("merge.debug.log"), - ); - processors.push(ProcessorRun { - role: role_merge, - pid: m.id(), - }); - wait(m, role_merge); - let e = spawn( - &exe, - role_estimate, - &[&merged], - &out, - &p("estimate.debug.log"), - ); - processors.push(ProcessorRun { - role: role_estimate, - pid: e.id(), - }); - wait(e, role_estimate); - let decoded = decode_pdata_to_observations(read_otlp(&out)?) - .map_err(|e| e.to_string())? - .observations; - let pipeline_elapsed_nanoseconds = pipeline_start.elapsed().as_nanos(); - let validation_start = Instant::now(); - if options.scenario == "raw" && decoded.len() != (n * 2) as usize { - return Err(format!( - "raw backend received {} of {} signals", - decoded.len(), - n * 2 - )); - } - let mut quantiles = Vec::with_capacity(2); - for o in &decoded { - if options.scenario == "raw" { - continue; - } - let q = o - .labels - .iter() - .find(|kv| kv.key == "quantile") - .map_or("?", |kv| kv.value.as_str()); - println!( - "result metric={} quantile={} value={:.3}", - o.metric, q, o.value.float - ); - if options.scenario != "raw" { - let expected_metric = if options.scenario == "kll" { - "request.duration.estimate" - } else { - "request.duration.exact_estimate" - }; - assert_eq!(o.metric, expected_metric); - } - quantiles.push((q.to_owned(), o.value.float)); - } - if options.scenario != "raw" { - let expected = if options.semantic_traffic { - let mut values = (1..=n * 2).map(traffic_value).collect::>(); - values.sort_by(f64::total_cmp); - let at = |q: f64| values[((values.len() - 1) as f64 * q).round() as usize]; - [at(0.5), at(0.99)] - } else { - [n as f64 + 1.0, ((n * 2 - 1) as f64 * 0.99).round() + 1.0] - }; - if options.scenario == "kll" { - validate_quantiles(&quantiles, [(0.5, expected[0]), (0.99, expected[1])])?; - } else { - for (quantile, expected) in [("0.5", expected[0]), ("0.99", expected[1])] { - let actual = quantiles - .iter() - .find_map(|(label, value)| (label == quantile).then_some(*value)) - .ok_or_else(|| format!("missing exact quantile {quantile}"))?; - if actual != expected { - return Err(format!( - "exact quantile {quantile}: got {actual}, expected {expected}" - )); - } - } - } - } - let validation_elapsed_nanoseconds = validation_start.elapsed().as_nanos(); - if let Some(endpoint) = options.prometheus_otlp_endpoint { - prometheus_export::export(&out, &endpoint)?; - println!("KLL estimates exported to Prometheus"); - } - if let Some(path) = options.result_manifest { - let manifest = DemoManifest { - parent_pid: std::process::id(), - points_per_source: n, - pipeline_elapsed_nanoseconds, - validation_elapsed_nanoseconds, - traffic_mode: if options.semantic_traffic { - "semantic" - } else { - "sequential" - }, - generator_threads: options.generator_threads, - traffic_seed: TRAFFIC_SEED, - processors, - }; - fs::write( - path, - serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?, - ) - .map_err(|e| e.to_string())?; - } - println!( - "success: {} ran four DF processors in four child OS processes", - options.scenario - ); - println!("official detailed debug traces: {}", dir.display()); - Ok(()) -} - -fn main() { - let args = env::args_os().skip(1).collect::>(); - if args.first().is_some_and(|a| a == "--generate") { - if args.len() != 5 { - eprintln!("usage: --generate START END OUTPUT THREADS"); - std::process::exit(2); - } - let start: u64 = args[1].to_string_lossy().parse().expect("start"); - let end: u64 = args[2].to_string_lossy().parse().expect("end"); - let output = PathBuf::from(&args[3]); - let threads: usize = args[4].to_string_lossy().parse().expect("threads"); - let started = Instant::now(); - let cpu_start = process_cpu_nanoseconds().expect("read generator CPU clock"); - let pdata = generated_input(start, end, threads, true); - let serialization_start = Instant::now(); - write_otlp(&output, pdata).expect("write generated traffic"); - write_stage_metrics( - &output, - started, - cpu_start, - serialization_start.elapsed().as_nanos(), - ) - .expect("write generator metrics"); - } else if args.first().is_some_and(|a| a == WORKER_ARG) { - if args.len() < 5 { - eprintln!("usage: {WORKER_ARG} ROLE INPUT... OUTPUT TRACE"); - std::process::exit(2); - } - let role = args[1].to_string_lossy().into_owned(); - let paths = args[2..].iter().map(PathBuf::from).collect::>(); - let trace = paths[paths.len() - 1].clone(); - let output = paths[paths.len() - 2].clone(); - let inputs = paths[..paths.len() - 2].to_vec(); - if let Err(e) = run_worker(&role, inputs, output, trace) { - eprintln!("[{role}] {e}"); - std::process::exit(1); - } - } else { - match parent_options(&args).and_then(run_parent) { - Ok(()) => {} - Err(e) => { - eprintln!("{e}"); - std::process::exit(2); - } - } - } -} - -#[cfg(test)] -mod tests { - use super::validate_quantiles; - - /// Scenario: The backend receives correct, incorrect, and incomplete quantile outputs. - /// Guarantees: KLL result validation accepts bounded error and rejects bad or missing values. - #[test] - fn result_validation_requires_correct_p50_and_p99() { - let expected = [(0.5, 100.0), (0.99, 198.0)]; - assert!( - validate_quantiles(&[("0.5".into(), 100.0), ("0.99".into(), 198.0)], expected).is_ok() - ); - assert!( - validate_quantiles(&[("0.5".into(), 12.0), ("0.99".into(), 13.0)], expected).is_err() - ); - assert!(validate_quantiles(&[("0.5".into(), 100.0)], expected).is_err()); - } -} diff --git a/asap-precompute-rs/src/bin/demo/prometheus_export.rs b/asap-precompute-rs/src/bin/demo/prometheus_export.rs deleted file mode 100644 index 567b99d..0000000 --- a/asap-precompute-rs/src/bin/demo/prometheus_export.rs +++ /dev/null @@ -1,202 +0,0 @@ -//! Optional demo-only OTLP/HTTP export. Curl supplies HTTP/TLS without adding -//! an HTTP client dependency to the collector runtime. -use otel_arrow_dfe_pdata::proto::opentelemetry::collector::metrics::v1::{ - ExportMetricsServiceRequest, ExportMetricsServiceResponse, -}; -use otel_arrow_dfe_pdata::proto::opentelemetry::metrics::v1::metric::Data; -use prost_otap::Message; -use std::io::Write; -use std::path::Path; -use std::process::{Command, Stdio}; -use std::time::{SystemTime, UNIX_EPOCH}; - -fn prepare_gauges(bytes: &[u8], now: Option) -> Result, String> { - let mut request = ExportMetricsServiceRequest::decode(bytes).map_err(|e| e.to_string())?; - let mut points = 0; - for resource in &mut request.resource_metrics { - for scope in &mut resource.scope_metrics { - let scope_attributes = scope - .scope - .as_ref() - .map(|s| s.attributes.as_slice()) - .unwrap_or_default(); - for metric in &mut scope.metrics { - let Some(Data::Gauge(gauge)) = &mut metric.data else { - return Err("Prometheus demo export expects only estimated gauges".into()); - }; - for point in &mut gauge.data_points { - // The OTAP codec stores series labels on the scope. Prometheus - // identifies series by point attributes, so materialize them here. - for attribute in scope_attributes { - if !point.attributes.iter().any(|a| a.key == attribute.key) { - point.attributes.push(attribute.clone()); - } - } - // Synthetic windows stay in the trace files. Export is a current - // snapshot of the completed estimate, not historical telemetry. - if let Some(now) = now { - point.start_time_unix_nano = 0; - point.time_unix_nano = now; - } - // Window boundaries are timestamps/metadata, not series - // dimensions: retaining them as labels creates a new - // Prometheus series for every emitted rolling window. - point.attributes.retain(|attribute| { - !matches!( - attribute.key.as_str(), - "sketch.window_start_ms" | "sketch.window_end_ms" - ) - }); - points += 1; - } - } - } - } - if points == 0 { - return Err("no estimated gauge points to export".into()); - } - Ok(request.encode_to_vec()) -} - -pub fn export(path: &Path, endpoint: &str) -> Result<(), String> { - if !(endpoint.starts_with("http://") || endpoint.starts_with("https://")) { - return Err("Prometheus OTLP endpoint must be an http:// or https:// URL".into()); - } - let now = SystemTime::now() - .duration_since(UNIX_EPOCH) - .map_err(|e| e.to_string())? - .as_nanos() as u64; - export_bytes( - &std::fs::read(path).map_err(|e| e.to_string())?, - endpoint, - Some(now), - ) -} - -/// Export estimated OTLP gauges. `None` preserves window-end timestamps for -/// the streaming demo; `Some` publishes a current snapshot for the finite demo. -pub fn export_bytes(bytes: &[u8], endpoint: &str, now: Option) -> Result<(), String> { - if !(endpoint.starts_with("http://") || endpoint.starts_with("https://")) { - return Err("Prometheus OTLP endpoint must be an http:// or https:// URL".into()); - } - let body = prepare_gauges(bytes, now)?; - let mut child = Command::new("curl") - .args([ - "--silent", - "--show-error", - "--fail-with-body", - "--max-time", - "30", - "--header", - "Content-Type: application/x-protobuf", - "--header", - "Accept: application/x-protobuf", - "--data-binary", - "@-", - "--url", - endpoint, - ]) - .stdin(Stdio::piped()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()) - .spawn() - .map_err(|e| format!("start curl for Prometheus OTLP export: {e}"))?; - let write_result = child.stdin.take().expect("piped stdin").write_all(&body); - let output = child.wait_with_output().map_err(|e| e.to_string())?; - if !output.status.success() { - return Err(format!( - "Prometheus OTLP export failed: {} {}", - String::from_utf8_lossy(&output.stderr), - String::from_utf8_lossy(&output.stdout) - )); - } - write_result.map_err(|e| format!("write OTLP request: {e}"))?; - let response = ExportMetricsServiceResponse::decode(output.stdout.as_slice()) - .map_err(|e| format!("invalid Prometheus OTLP response: {e}"))?; - if let Some(partial) = response.partial_success { - if partial.rejected_data_points != 0 || !partial.error_message.is_empty() { - return Err(format!( - "Prometheus OTLP partial success: {} rejected: {}", - partial.rejected_data_points, partial.error_message - )); - } - } - Ok(()) -} - -#[cfg(test)] -mod tests { - use super::*; - use otel_arrow_dfe_pdata::proto::opentelemetry::common::v1::{ - any_value::Value, AnyValue, InstrumentationScope, KeyValue, - }; - use otel_arrow_dfe_pdata::proto::opentelemetry::metrics::v1::{ - Gauge, Metric, NumberDataPoint, ResourceMetrics, ScopeMetrics, - }; - - #[test] - fn export_uses_current_timestamp_and_preserves_estimate() { - let request = ExportMetricsServiceRequest { - resource_metrics: vec![ResourceMetrics { - scope_metrics: vec![ScopeMetrics { - scope: Some(InstrumentationScope { attributes: vec![KeyValue { key: "quantile".into(), value: Some(AnyValue { value: Some(Value::StringValue("0.99".into())) }) }], ..Default::default() }), - metrics: vec![Metric { - name: "request.duration.estimate".into(), - data: Some(Data::Gauge(Gauge { data_points: vec![NumberDataPoint { - attributes: vec![KeyValue { key: "sketch.window_end_ms".into(), value: Some(AnyValue { value: Some(Value::StringValue("2000".into())) }) }], - time_unix_nano: 2_000_000_000, - value: Some(otel_arrow_dfe_pdata::proto::opentelemetry::metrics::v1::number_data_point::Value::AsDouble(198.0)), - ..Default::default() - }] })), - ..Default::default() - }], - ..Default::default() - }], - ..Default::default() - }], - }; - let updated = ExportMetricsServiceRequest::decode( - prepare_gauges(&request.encode_to_vec(), Some(123456)) - .unwrap() - .as_slice(), - ) - .unwrap(); - let metric = &updated.resource_metrics[0].scope_metrics[0].metrics[0]; - let streaming = ExportMetricsServiceRequest::decode( - prepare_gauges(&request.encode_to_vec(), None) - .unwrap() - .as_slice(), - ) - .unwrap(); - let Some(Data::Gauge(streaming_gauge)) = - &streaming.resource_metrics[0].scope_metrics[0].metrics[0].data - else { - panic!("expected streaming gauge") - }; - assert_eq!(streaming_gauge.data_points[0].time_unix_nano, 2_000_000_000); - assert_eq!(metric.name, "request.duration.estimate"); - let Some(Data::Gauge(gauge)) = &metric.data else { - panic!("expected gauge") - }; - assert_eq!(gauge.data_points[0].time_unix_nano, 123456); - let Some(Data::Gauge(original)) = - &request.resource_metrics[0].scope_metrics[0].metrics[0].data - else { - panic!("expected gauge") - }; - assert_eq!(gauge.data_points[0].value, original.data_points[0].value); - assert_eq!( - gauge.data_points[0].attributes, - request.resource_metrics[0].scope_metrics[0] - .scope - .as_ref() - .unwrap() - .attributes - ); - assert!(prepare_gauges( - &ExportMetricsServiceRequest::default().encode_to_vec(), - Some(1) - ) - .is_err()); - } -} diff --git a/asap-precompute-rs/src/control_channel.rs b/asap-precompute-rs/src/control_channel.rs deleted file mode 100644 index 6180f19..0000000 --- a/asap-precompute-rs/src/control_channel.rs +++ /dev/null @@ -1,60 +0,0 @@ -//! [`ControlChannel`] trait. -//! -//! The controller delivers -//! [`crate::config::PrecomputeConfigSet`] plans to runtime adapters -//! through implementations of this trait. Concrete implementations -//! (OpAMP, HTTP-poll, file-watch) land alongside the per-language -//! adapter glue. - -use crate::config::PrecomputeConfigSet; - -/// Delivers [`PrecomputeConfigSet`]s from the controller to -/// adapters at runtime. -pub trait ControlChannel: Send + Sync { - /// Returns a non-`None` [`PrecomputeConfigSet`] if the plan has - /// changed since the last poll. `None` means "no change." - /// - /// Implementations may block briefly on network I/O; callers - /// typically call from a dedicated task / thread. - fn poll(&self) -> Option; - - /// Confirms acceptance of a plan version. - /// - /// OpAMP uses this to report effective config back to the - /// supervisor; HTTP-poll implementations may ignore. - fn ack(&self, plan_version: u64); -} - -#[cfg(test)] -mod tests { - use super::*; - use std::sync::atomic::{AtomicU64, Ordering}; - - /// No-op stub used to prove the trait can be implemented — a - /// minimal fake before the real OpAMP / HTTP-poll impls land. - struct StubChannel { - last_ack: AtomicU64, - } - - impl ControlChannel for StubChannel { - fn poll(&self) -> Option { - None - } - - fn ack(&self, plan_version: u64) { - self.last_ack.store(plan_version, Ordering::Release); - } - } - - #[test] - fn stub_channel_implements_trait() { - let c = StubChannel { - last_ack: AtomicU64::new(0), - }; - // poll returns None for the no-change case. - assert!(c.poll().is_none()); - // ack records the version. - c.ack(7); - assert_eq!(c.last_ack.load(Ordering::Acquire), 7); - } -} diff --git a/asap-precompute-rs/src/otap/decode.rs b/asap-precompute-rs/src/otap/decode.rs deleted file mode 100644 index b1e502b..0000000 --- a/asap-precompute-rs/src/otap/decode.rs +++ /dev/null @@ -1,670 +0,0 @@ -//! `decode_batch`: [`arrow_array::RecordBatch`] → `Vec`. -//! -//! One `Observation` per row in the input batch. The OTAP shape is -//! column-major where OTel's pmetric is a tree, so the codec walks -//! the columns once up front (resolving them to typed Arrow array -//! references), then iterates rows pulling values by index. This is -//! the throughput win. - -use std::sync::Arc; - -use arrow_array::{ - Array, BinaryArray, Float64Array, RecordBatch, StringArray, TimestampNanosecondArray, - UInt32Array, UInt64Array, -}; -use arrow_schema::DataType; -use thiserror::Error; - -use crate::envelope::{Encoding, SketchEnvelope, SketchType}; -use crate::observation::{KeyValue, Observation, ObservationValue}; - -use super::schema::{ - is_reserved_column, ATTR_AGG_ID, ATTR_ENCODING, ATTR_ENVELOPE, ATTR_SCHEMA_VERSION, - ATTR_SKETCH_TYPE, ATTR_WINDOW_END_MS, ATTR_WINDOW_START_MS, COLUMN_METRIC, - COLUMN_TIME_UNIX_NANO, COLUMN_VALUE, -}; - -/// Errors returned by [`decode_batch`]. -/// -/// Granularity choice: each error variant identifies the **column** at -/// fault and the kind of failure (wrong type / missing required field). -/// Variants are intentionally informative-but-not-too-fine: the caller -/// (Phase C plugin) typically logs and drops the whole batch — there -/// is no per-row recovery path because the codec promises atomic -/// decode of the input batch. -#[derive(Debug, Error)] -pub enum OtapDecodeError { - /// A required typed column was present but had an unexpected - /// Arrow `DataType`. Typical cause: an upstream Arrow producer - /// emitting `Int64` where `UInt64` is expected. - #[error( - "otap decode: column {column:?} has unexpected type: expected {expected}, got {actual:?}" - )] - WrongColumnType { - /// Column name. - column: String, - /// Expected Arrow type, in human-readable form. - expected: &'static str, - /// Actual Arrow type observed. - actual: DataType, - }, - - /// An envelope row (one with a non-null `_asap_envelope` cell) - /// carried a `_asap_sketch_type` value the codec didn't recognize. - /// Hard error rather than silent fallback because a misrouted - /// envelope would silently corrupt downstream sketch state. - #[error("otap decode: row {row}: unknown sketch type {value:?}")] - UnknownSketchType { - /// Row index in the input batch. - row: usize, - /// Raw string value observed. - value: String, - }, - - /// An envelope row carried a `_asap_encoding` value the codec - /// didn't recognize. Same reasoning as `UnknownSketchType`. - #[error("otap decode: row {row}: unknown encoding {value:?}")] - UnknownEncoding { - /// Row index in the input batch. - row: usize, - /// Raw string value observed. - value: String, - }, -} - -/// Decode an OTAP `RecordBatch` into a `Vec`. -/// -/// One `Observation` per row. Column resolution is by name. Any row whose -/// `_asap_envelope` cell is non-null emits a -/// [`crate::observation::ObservationValueKind::Envelope`] -/// observation; otherwise the row routes through the scalar -/// [`crate::observation::ObservationValueKind::Float`] path using -/// the `value` column. -/// -/// Resource-attribute scope: at this Phase-B flat-shape layer there -/// is no resource child batch; `Observation::resource_labels` is -/// always empty. Phase C's plugin shell joins the OTAP resource -/// child batch onto the metrics rows before calling `decode_batch`, -/// so resource attrs flow as ordinary `Utf8` columns from the -/// codec's perspective. Adapters that want the OTAP "resource scope" -/// distinction can pre-prefix columns with `resource.` before they -/// hit this codec. -/// -/// Returns `Ok(vec![])` for an empty batch. -pub fn decode_batch(batch: &RecordBatch) -> Result, OtapDecodeError> { - if batch.num_rows() == 0 { - return Ok(Vec::new()); - } - - let columns = ResolvedColumns::resolve(batch)?; - let mut out = Vec::with_capacity(batch.num_rows()); - - for row in 0..batch.num_rows() { - let metric = columns - .metric - .as_ref() - .map(|arr| arr.value(row).to_string()) - .unwrap_or_default(); - let timestamp_ms = columns - .timestamp - .as_ref() - .map(|t| t.timestamp_ms_at(row)) - .unwrap_or(0); - let labels = columns.labels_for_row(row); - - let value = if let Some(env_arr) = &columns.envelope { - if env_arr.is_null(row) { - scalar_value(&columns, row) - } else { - let env = - build_envelope(&columns, row, &metric, &labels, env_arr.value(row).to_vec())?; - ObservationValue::envelope(env) - } - } else { - scalar_value(&columns, row) - }; - - out.push(Observation::new( - timestamp_ms, - metric, - Vec::new(), // resource_labels — see docstring above. - labels, - value, - )); - } - - Ok(out) -} - -/// Pulls scalar value for a row, defaulting to 0.0 if absent or null. -fn scalar_value(columns: &ResolvedColumns<'_>, row: usize) -> ObservationValue { - let v = columns - .value - .as_ref() - .filter(|arr| !arr.is_null(row)) - .map(|arr| arr.value(row)) - .unwrap_or(0.0); - ObservationValue::float(v) -} - -/// Build a [`SketchEnvelope`] for a Strategy-B row. -fn build_envelope( - columns: &ResolvedColumns<'_>, - row: usize, - metric: &str, - labels: &[KeyValue], - payload: Vec, -) -> Result { - let sketch_type = if let Some(arr) = &columns.sketch_type { - if arr.is_null(row) { - SketchType::Unspecified - } else { - parse_sketch_type(row, arr.value(row))? - } - } else { - SketchType::Unspecified - }; - - let encoding = if let Some(arr) = &columns.encoding { - if arr.is_null(row) { - Encoding::Unspecified - } else { - parse_encoding(row, arr.value(row))? - } - } else { - Encoding::Unspecified - }; - - let agg_id = columns - .agg_id - .as_ref() - .filter(|arr| !arr.is_null(row)) - .map(|arr| arr.value(row)) - .unwrap_or(0); - let schema_version = columns - .schema_version - .as_ref() - .filter(|arr| !arr.is_null(row)) - .map(|arr| arr.value(row)) - .unwrap_or(0); - let window_start_ms = columns - .window_start_ms - .as_ref() - .filter(|arr| !arr.is_null(row)) - .map(|arr| arr.value(row)) - .unwrap_or(0); - let window_end_ms = columns - .window_end_ms - .as_ref() - .filter(|arr| !arr.is_null(row)) - .map(|arr| arr.value(row)) - .unwrap_or(0); - - let value = columns - .value - .as_ref() - .filter(|arr| !arr.is_null(row)) - .map(|arr| arr.value(row)) - .unwrap_or(0.0); - - Ok(SketchEnvelope { - schema_version, - sketch_type, - agg_id, - resource_labels: Vec::new(), - labels: labels.to_vec(), - window_start_ms, - window_end_ms, - encoding, - payload, - hash_spec: None, - metric_name: metric.to_string(), - count: 0, - aggregation_temporality: 0, - value, - }) -} - -/// Parses a canonical sketch-type string. `pub(super)` so -/// [`super::dictionary`] can reuse the same parsing (and error -/// variant) for `SCHEMA.sketch_type` rows. -pub(super) fn parse_sketch_type(row: usize, raw: &str) -> Result { - match raw { - "DDSketch" => Ok(SketchType::DDSketch), - "KLLSketch" => Ok(SketchType::KLLSketch), - "HLLSketch" => Ok(SketchType::HLLSketch), - "CountSketch" => Ok(SketchType::CountSketch), - "CountMinSketch" => Ok(SketchType::CountMinSketch), - "CountMinSketchWithHeap" => Ok(SketchType::CountMinSketchWithHeap), - "Unspecified" | "" => Ok(SketchType::Unspecified), - _ => Err(OtapDecodeError::UnknownSketchType { - row, - value: raw.to_string(), - }), - } -} - -/// Parses a canonical encoding string. `pub(super)` so -/// [`super::dictionary`] can reuse the same parsing (and error -/// variant) for `SCHEMA.encoding` rows. -pub(super) fn parse_encoding(row: usize, raw: &str) -> Result { - match raw { - "PROTO_FULL" => Ok(Encoding::ProtoFull), - "PROTO_DELTA" => Ok(Encoding::ProtoDelta), - "MSGPACK" => Ok(Encoding::Msgpack), - "MSGPACK_DELTA" => Ok(Encoding::MsgpackDelta), - "UNSPECIFIED" | "" => Ok(Encoding::Unspecified), - _ => Err(OtapDecodeError::UnknownEncoding { - row, - value: raw.to_string(), - }), - } -} - -/// Pre-resolved typed Arrow array references for the columns we know -/// how to read. Built once per batch in [`ResolvedColumns::resolve`]; -/// per-row decoding only does index lookups against these references. -struct ResolvedColumns<'a> { - timestamp: Option>, - metric: Option<&'a StringArray>, - value: Option<&'a Float64Array>, - envelope: Option<&'a BinaryArray>, - sketch_type: Option<&'a StringArray>, - agg_id: Option<&'a UInt64Array>, - schema_version: Option<&'a UInt32Array>, - window_start_ms: Option<&'a UInt64Array>, - window_end_ms: Option<&'a UInt64Array>, - encoding: Option<&'a StringArray>, - /// Non-reserved `Utf8` columns; treated as per-row labels. - label_columns: Vec<(String, &'a StringArray)>, -} - -/// Wrapper that unifies `UInt64` and `Timestamp(Nanosecond, _)` into a -/// per-row "milliseconds since epoch" reader so [`decode_batch`] -/// doesn't have to branch on type per row. -enum TimestampColumn<'a> { - /// Plain `UInt64` interpreted as nanoseconds since epoch. - UInt64Nanos(&'a UInt64Array), - /// Native `Timestamp(Nanosecond, _)`. - Timestamp(&'a TimestampNanosecondArray), -} - -impl TimestampColumn<'_> { - fn timestamp_ms_at(&self, row: usize) -> u64 { - match self { - Self::UInt64Nanos(arr) => { - if arr.is_null(row) { - 0 - } else { - arr.value(row) / 1_000_000 - } - } - Self::Timestamp(arr) => { - if arr.is_null(row) { - 0 - } else { - let v = arr.value(row); - if v <= 0 { - 0 - } else { - (v as u64) / 1_000_000 - } - } - } - } - } -} - -impl<'a> ResolvedColumns<'a> { - fn resolve(batch: &'a RecordBatch) -> Result { - let schema = batch.schema(); - let mut timestamp = None; - let mut metric = None; - let mut value = None; - let mut envelope = None; - let mut sketch_type = None; - let mut agg_id = None; - let mut schema_version = None; - let mut window_start_ms = None; - let mut window_end_ms = None; - let mut encoding = None; - let mut label_columns: Vec<(String, &StringArray)> = Vec::new(); - - for (i, field) in schema.fields().iter().enumerate() { - let name = field.name(); - let col: &Arc = batch.column(i); - match name.as_str() { - COLUMN_TIME_UNIX_NANO => { - timestamp = Some(resolve_timestamp(name, col)?); - } - COLUMN_METRIC => { - metric = Some(downcast_string(name, col)?); - } - COLUMN_VALUE => { - value = Some(downcast_float64(name, col)?); - } - ATTR_ENVELOPE => { - envelope = Some(downcast_binary(name, col)?); - } - ATTR_SKETCH_TYPE => { - sketch_type = Some(downcast_string(name, col)?); - } - ATTR_AGG_ID => { - agg_id = Some(downcast_u64(name, col)?); - } - ATTR_SCHEMA_VERSION => { - schema_version = Some(downcast_u32(name, col)?); - } - ATTR_WINDOW_START_MS => { - window_start_ms = Some(downcast_u64(name, col)?); - } - ATTR_WINDOW_END_MS => { - window_end_ms = Some(downcast_u64(name, col)?); - } - ATTR_ENCODING => { - encoding = Some(downcast_string(name, col)?); - } - _ if !is_reserved_column(name) && matches!(field.data_type(), DataType::Utf8) => { - let s = downcast_string(name, col)?; - label_columns.push((name.clone(), s)); - } - _ => { - // Non-Utf8, non-reserved columns (e.g. extra - // numeric columns from a multi-field metric batch) - // are ignored — Phase B is value-column-only. - // Future v2 multi-field support reads those here. - } - } - } - - Ok(Self { - timestamp, - metric, - value, - envelope, - sketch_type, - agg_id, - schema_version, - window_start_ms, - window_end_ms, - encoding, - label_columns, - }) - } - - /// Build the labels slice for a single row from all `Utf8` non- - /// reserved columns. Skips null cells (treats them as "label not - /// present for this row" which matches the OTel codec's - /// "absent attribute = no key" behavior). - fn labels_for_row(&self, row: usize) -> Vec { - if self.label_columns.is_empty() { - return Vec::new(); - } - let mut out = Vec::with_capacity(self.label_columns.len()); - for (key, arr) in &self.label_columns { - if arr.is_null(row) { - continue; - } - out.push(KeyValue::new(key.clone(), arr.value(row).to_string())); - } - out - } -} - -fn resolve_timestamp<'a>( - name: &str, - col: &'a Arc, -) -> Result, OtapDecodeError> { - if let Some(arr) = col.as_any().downcast_ref::() { - return Ok(TimestampColumn::UInt64Nanos(arr)); - } - if let Some(arr) = col.as_any().downcast_ref::() { - return Ok(TimestampColumn::Timestamp(arr)); - } - Err(OtapDecodeError::WrongColumnType { - column: name.to_string(), - expected: "UInt64 or Timestamp(Nanosecond)", - actual: col.data_type().clone(), - }) -} - -fn downcast_string<'a>( - name: &str, - col: &'a Arc, -) -> Result<&'a StringArray, OtapDecodeError> { - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapDecodeError::WrongColumnType { - column: name.to_string(), - expected: "Utf8", - actual: col.data_type().clone(), - }) -} - -fn downcast_float64<'a>( - name: &str, - col: &'a Arc, -) -> Result<&'a Float64Array, OtapDecodeError> { - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapDecodeError::WrongColumnType { - column: name.to_string(), - expected: "Float64", - actual: col.data_type().clone(), - }) -} - -fn downcast_binary<'a>( - name: &str, - col: &'a Arc, -) -> Result<&'a BinaryArray, OtapDecodeError> { - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapDecodeError::WrongColumnType { - column: name.to_string(), - expected: "Binary", - actual: col.data_type().clone(), - }) -} - -fn downcast_u64<'a>( - name: &str, - col: &'a Arc, -) -> Result<&'a UInt64Array, OtapDecodeError> { - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapDecodeError::WrongColumnType { - column: name.to_string(), - expected: "UInt64", - actual: col.data_type().clone(), - }) -} - -fn downcast_u32<'a>( - name: &str, - col: &'a Arc, -) -> Result<&'a UInt32Array, OtapDecodeError> { - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapDecodeError::WrongColumnType { - column: name.to_string(), - expected: "UInt32", - actual: col.data_type().clone(), - }) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::observation::ObservationValueKind; - use arrow_array::{Int64Array, RecordBatch}; - use arrow_schema::{Field, Schema}; - use std::sync::Arc; - - fn batch_from(fields: Vec, columns: Vec>) -> RecordBatch { - let schema = Arc::new(Schema::new(fields)); - RecordBatch::try_new(schema, columns).expect("build batch") - } - - #[test] - fn empty_batch_decodes_to_empty_vec() { - let batch = batch_from( - vec![Field::new(COLUMN_METRIC, DataType::Utf8, false)], - vec![Arc::new(StringArray::from(Vec::<&str>::new()))], - ); - assert!(decode_batch(&batch).expect("decode").is_empty()); - } - - #[test] - fn decode_scalar_row() { - let batch = batch_from( - vec![ - Field::new(COLUMN_TIME_UNIX_NANO, DataType::UInt64, false), - Field::new(COLUMN_METRIC, DataType::Utf8, false), - Field::new(COLUMN_VALUE, DataType::Float64, false), - Field::new("region", DataType::Utf8, true), - ], - vec![ - Arc::new(UInt64Array::from(vec![1_000_000_000_u64])), // 1000 ms - Arc::new(StringArray::from(vec!["http_requests"])), - Arc::new(Float64Array::from(vec![2.5])), - Arc::new(StringArray::from(vec![Some("us-east")])), - ], - ); - let obs = decode_batch(&batch).expect("decode"); - assert_eq!(obs.len(), 1); - assert_eq!(obs[0].timestamp_ms, 1000); - assert_eq!(obs[0].metric, "http_requests"); - assert_eq!(obs[0].value.kind, ObservationValueKind::Float); - assert_eq!(obs[0].value.float, 2.5); - assert_eq!(obs[0].labels.len(), 1); - assert_eq!(obs[0].labels[0].key, "region"); - assert_eq!(obs[0].labels[0].value, "us-east"); - } - - #[test] - fn decode_envelope_row_routes_to_envelope_kind() { - let envelope_payload = vec![1_u8, 2, 3, 4]; - let batch = batch_from( - vec![ - Field::new(COLUMN_METRIC, DataType::Utf8, false), - Field::new(ATTR_ENVELOPE, DataType::Binary, true), - Field::new(ATTR_SKETCH_TYPE, DataType::Utf8, true), - Field::new(ATTR_AGG_ID, DataType::UInt64, true), - Field::new(ATTR_SCHEMA_VERSION, DataType::UInt32, true), - Field::new(ATTR_WINDOW_START_MS, DataType::UInt64, true), - Field::new(ATTR_WINDOW_END_MS, DataType::UInt64, true), - Field::new(ATTR_ENCODING, DataType::Utf8, true), - ], - vec![ - Arc::new(StringArray::from(vec!["http_request_duration"])), - Arc::new(BinaryArray::from(vec![envelope_payload.as_slice()])), - Arc::new(StringArray::from(vec![Some("DDSketch")])), - Arc::new(UInt64Array::from(vec![Some(42_u64)])), - Arc::new(UInt32Array::from(vec![Some(1_u32)])), - Arc::new(UInt64Array::from(vec![Some(1_000_u64)])), - Arc::new(UInt64Array::from(vec![Some(2_000_u64)])), - Arc::new(StringArray::from(vec![Some("PROTO_FULL")])), - ], - ); - let obs = decode_batch(&batch).expect("decode"); - assert_eq!(obs.len(), 1); - assert_eq!(obs[0].value.kind, ObservationValueKind::Envelope); - let env = obs[0].value.envelope.as_ref().expect("envelope"); - assert_eq!(env.payload, envelope_payload); - assert_eq!(env.sketch_type, SketchType::DDSketch); - assert_eq!(env.agg_id, 42); - assert_eq!(env.schema_version, 1); - assert_eq!(env.window_start_ms, 1_000); - assert_eq!(env.window_end_ms, 2_000); - assert_eq!(env.encoding, Encoding::ProtoFull); - assert_eq!(env.metric_name, "http_request_duration"); - } - - #[test] - fn decode_rejects_wrong_column_type() { - // `value` arrives as Int64 instead of Float64 — schema mismatch. - let batch = batch_from( - vec![ - Field::new(COLUMN_METRIC, DataType::Utf8, false), - Field::new(COLUMN_VALUE, DataType::Int64, false), - ], - vec![ - Arc::new(StringArray::from(vec!["m"])), - Arc::new(Int64Array::from(vec![1_i64])), - ], - ); - let err = decode_batch(&batch).expect_err("should reject"); - match err { - OtapDecodeError::WrongColumnType { column, .. } => assert_eq!(column, "value"), - other => panic!("unexpected variant: {other:?}"), - } - } - - #[test] - fn decode_rejects_unknown_sketch_type() { - let batch = batch_from( - vec![ - Field::new(ATTR_ENVELOPE, DataType::Binary, false), - Field::new(ATTR_SKETCH_TYPE, DataType::Utf8, false), - ], - vec![ - Arc::new(BinaryArray::from(vec![b"x".as_ref()])), - Arc::new(StringArray::from(vec!["BogoSketch"])), - ], - ); - let err = decode_batch(&batch).expect_err("should reject"); - match err { - OtapDecodeError::UnknownSketchType { row, value } => { - assert_eq!(row, 0); - assert_eq!(value, "BogoSketch"); - } - other => panic!("unexpected variant: {other:?}"), - } - } - - #[test] - fn decode_rejects_unknown_encoding() { - let batch = batch_from( - vec![ - Field::new(ATTR_ENVELOPE, DataType::Binary, false), - Field::new(ATTR_ENCODING, DataType::Utf8, false), - ], - vec![ - Arc::new(BinaryArray::from(vec![b"x".as_ref()])), - Arc::new(StringArray::from(vec!["URL_SAFE_BASE64"])), - ], - ); - let err = decode_batch(&batch).expect_err("should reject"); - match err { - OtapDecodeError::UnknownEncoding { row, value } => { - assert_eq!(row, 0); - assert_eq!(value, "URL_SAFE_BASE64"); - } - other => panic!("unexpected variant: {other:?}"), - } - } - - #[test] - fn null_envelope_cell_falls_back_to_scalar() { - // A row with an `_asap_envelope` column that is null should - // route through the scalar value path, not erroneously emit - // an empty-payload envelope. - let batch = batch_from( - vec![ - Field::new(COLUMN_METRIC, DataType::Utf8, false), - Field::new(COLUMN_VALUE, DataType::Float64, false), - Field::new(ATTR_ENVELOPE, DataType::Binary, true), - ], - vec![ - Arc::new(StringArray::from(vec!["m"])), - Arc::new(Float64Array::from(vec![7.5])), - Arc::new(BinaryArray::from(vec![None as Option<&[u8]>])), - ], - ); - let obs = decode_batch(&batch).expect("decode"); - assert_eq!(obs.len(), 1); - assert_eq!(obs[0].value.kind, ObservationValueKind::Float); - assert_eq!(obs[0].value.float, 7.5); - } -} diff --git a/asap-precompute-rs/src/otap/encode.rs b/asap-precompute-rs/src/otap/encode.rs deleted file mode 100644 index 40215ea..0000000 --- a/asap-precompute-rs/src/otap/encode.rs +++ /dev/null @@ -1,261 +0,0 @@ -//! `encode_batch`: `&[SketchEnvelope]` → [`arrow_array::RecordBatch`]. -//! -//! Strategy-B encoding: one row per envelope, with the envelope -//! payload riding in the -//! well-known `_asap_envelope` Binary column and companion metadata -//! keys (`_asap_sketch_type`, `_asap_agg_id`, `_asap_schema_version`, -//! `_asap_window_start_ms`, `_asap_window_end_ms`, `_asap_encoding`) -//! riding in their typed columns. -//! -//! At the Phase-B layer those carrier keys are projected as flat -//! columns. The Phase-C plugin shell will lift them onto OTAP's -//! per-row attribute child batch (OTAP's strict schema validator -//! rejects extension columns on Logs/Metrics/Traces RecordBatches, -//! so the lift step is required before the batch goes downstream). - -use std::collections::BTreeSet; -use std::sync::Arc; - -use arrow_array::{ - Array, BinaryArray, Float64Array, RecordBatch, StringArray, UInt32Array, UInt64Array, -}; -use arrow_schema::{DataType, Field, Schema}; -use thiserror::Error; - -use crate::envelope::SketchEnvelope; - -use super::schema::{ - ATTR_AGG_ID, ATTR_ENCODING, ATTR_ENVELOPE, ATTR_SCHEMA_VERSION, ATTR_SKETCH_TYPE, - ATTR_WINDOW_END_MS, ATTR_WINDOW_START_MS, COLUMN_METRIC, COLUMN_TIME_UNIX_NANO, COLUMN_VALUE, -}; - -/// Errors returned by [`encode_batch`]. -#[derive(Debug, Error)] -pub enum OtapEncodeError { - /// Constructing the Arrow `RecordBatch` failed. This indicates a - /// codec bug (mismatched array lengths) rather than caller error; - /// kept as an error variant rather than `unwrap` so the plugin - /// shell can surface it via OTAP's error channel rather than - /// crash the host process. - #[error("otap encode: arrow record-batch construction failed: {0}")] - ArrowError(String), -} - -/// Encode a slice of [`SketchEnvelope`]s as a single OTAP `RecordBatch`. -/// -/// Schema: the well-known scalar columns -/// (`time_unix_nano` UInt64, `metric` Utf8, `value` Float64) plus the -/// Strategy-B carrier columns (`_asap_envelope` Binary, -/// `_asap_sketch_type` Utf8, `_asap_agg_id` UInt64, -/// `_asap_schema_version` UInt32, `_asap_window_start_ms` UInt64, -/// `_asap_window_end_ms` UInt64, `_asap_encoding` Utf8) plus one -/// `Utf8` column per distinct label key drawn from the union of -/// every envelope's `labels` slice. -/// -/// `value` carries the estimate-mode scalar (a quantile or cardinality -/// estimate) from `SketchEnvelope::value`, and is 0.0 for sketch-payload -/// envelopes (whose state rides in `_asap_envelope`). Encode -/// does not touch `Observation::resource_labels` because envelopes -/// flatten resource attrs into `labels` already (Strategy-B -/// platforms have no resource scope; per envelope.rs docstring). -/// -/// `time_unix_nano` is set to `window_end_ms * 1_000_000`. -/// -/// Returns an empty batch (zero rows, schema-only) when `envelopes` is -/// empty. -pub fn encode_batch(envelopes: &[SketchEnvelope]) -> Result { - let label_keys = collect_label_keys(envelopes); - - let mut fields: Vec = vec![ - Field::new(COLUMN_TIME_UNIX_NANO, DataType::UInt64, true), - Field::new(COLUMN_METRIC, DataType::Utf8, true), - Field::new(COLUMN_VALUE, DataType::Float64, true), - Field::new(ATTR_ENVELOPE, DataType::Binary, true), - Field::new(ATTR_SKETCH_TYPE, DataType::Utf8, true), - Field::new(ATTR_AGG_ID, DataType::UInt64, true), - Field::new(ATTR_SCHEMA_VERSION, DataType::UInt32, true), - Field::new(ATTR_WINDOW_START_MS, DataType::UInt64, true), - Field::new(ATTR_WINDOW_END_MS, DataType::UInt64, true), - Field::new(ATTR_ENCODING, DataType::Utf8, true), - ]; - for key in &label_keys { - fields.push(Field::new(key, DataType::Utf8, true)); - } - - let mut time_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut metric_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut value_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut envelope_col: Vec>> = Vec::with_capacity(envelopes.len()); - let mut sketch_type_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut agg_id_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut schema_version_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut window_start_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut window_end_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut encoding_col: Vec> = Vec::with_capacity(envelopes.len()); - let mut label_cols: Vec>> = (0..label_keys.len()) - .map(|_| Vec::with_capacity(envelopes.len())) - .collect(); - - for env in envelopes { - time_col.push(Some(env.window_end_ms.saturating_mul(1_000_000))); - metric_col.push(Some(env.metric_name.clone())); - // The estimate-mode `value` (quantile / cardinality); 0.0 for - // sketch-payload envelopes. - value_col.push(Some(env.value)); - envelope_col.push(Some(env.payload.clone())); - sketch_type_col.push(Some(env.sketch_type.name())); - agg_id_col.push(Some(env.agg_id)); - schema_version_col.push(Some(env.schema_version)); - window_start_col.push(Some(env.window_start_ms)); - window_end_col.push(Some(env.window_end_ms)); - encoding_col.push(Some(env.encoding.name())); - - for (col_idx, key) in label_keys.iter().enumerate() { - let value = env.labels.iter().find_map(|kv| { - if &kv.key == key { - Some(kv.value.clone()) - } else { - None - } - }); - label_cols[col_idx].push(value); - } - } - - let mut columns: Vec> = vec![ - Arc::new(UInt64Array::from(time_col)), - Arc::new(StringArray::from(metric_col)), - Arc::new(Float64Array::from(value_col)), - Arc::new(BinaryArray::from_opt_vec( - envelope_col - .iter() - .map(|o| o.as_deref()) - .collect::>(), - )), - Arc::new(StringArray::from(sketch_type_col)), - Arc::new(UInt64Array::from(agg_id_col)), - Arc::new(UInt32Array::from(schema_version_col)), - Arc::new(UInt64Array::from(window_start_col)), - Arc::new(UInt64Array::from(window_end_col)), - Arc::new(StringArray::from(encoding_col)), - ]; - for col in label_cols { - columns.push(Arc::new(StringArray::from(col))); - } - - let schema = Arc::new(Schema::new(fields)); - RecordBatch::try_new(schema, columns).map_err(|e| OtapEncodeError::ArrowError(e.to_string())) -} - -/// Walk every envelope's `labels` slice and return the sorted union of -/// keys. Sorted so the emitted RecordBatch schema is deterministic -/// across runs — important for reproducibility tests and for the -/// cross-host parity tests in Phase E. -fn collect_label_keys(envelopes: &[SketchEnvelope]) -> Vec { - let mut set: BTreeSet = BTreeSet::new(); - for env in envelopes { - for kv in &env.labels { - set.insert(kv.key.clone()); - } - } - set.into_iter().collect() -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::envelope::{Encoding, SketchType}; - use crate::observation::KeyValue; - - fn sample_envelope(payload: &[u8], agg_id: u64) -> SketchEnvelope { - SketchEnvelope { - schema_version: 1, - sketch_type: SketchType::DDSketch, - agg_id, - resource_labels: vec![], - labels: vec![KeyValue::new("region", "us-east")], - window_start_ms: 1_000, - window_end_ms: 2_000, - encoding: Encoding::ProtoFull, - payload: payload.to_vec(), - hash_spec: None, - metric_name: "http_request_duration".into(), - count: 0, - aggregation_temporality: 0, - value: 0.0, - } - } - - #[test] - fn empty_envelopes_produces_empty_batch() { - let batch = encode_batch(&[]).expect("encode"); - assert_eq!(batch.num_rows(), 0); - // Schema still carries the well-known columns. - assert!(batch.schema().column_with_name(ATTR_ENVELOPE).is_some()); - } - - #[test] - fn encode_single_envelope_carries_all_fields() { - let env = sample_envelope(&[0xde, 0xad, 0xbe, 0xef], 7); - let batch = encode_batch(std::slice::from_ref(&env)).expect("encode"); - assert_eq!(batch.num_rows(), 1); - - let env_col = batch - .column_by_name(ATTR_ENVELOPE) - .expect("envelope column") - .as_any() - .downcast_ref::() - .expect("binary"); - assert_eq!(env_col.value(0), &[0xde, 0xad, 0xbe, 0xef]); - - let st = batch - .column_by_name(ATTR_SKETCH_TYPE) - .unwrap() - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(st.value(0), "DDSketch"); - - let id = batch - .column_by_name(ATTR_AGG_ID) - .unwrap() - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(id.value(0), 7); - } - - #[test] - fn label_columns_are_union_sorted() { - let env_a = SketchEnvelope { - labels: vec![KeyValue::new("region", "a"), KeyValue::new("zone", "z1")], - ..sample_envelope(&[1], 1) - }; - let env_b = SketchEnvelope { - labels: vec![KeyValue::new("region", "b"), KeyValue::new("path", "/x")], - ..sample_envelope(&[2], 1) - }; - let batch = encode_batch(&[env_a, env_b]).expect("encode"); - - // The union is {path, region, zone} sorted; reserved columns - // appear before the label columns. - let schema = batch.schema(); - let label_names: Vec<&str> = schema - .fields() - .iter() - .map(|f| f.name().as_str()) - .filter(|n| !super::super::schema::is_reserved_column(n)) - .collect(); - assert_eq!(label_names, vec!["path", "region", "zone"]); - - // Row 0 (env_a) has region=a, zone=z1, path=null. - let path_col = batch - .column_by_name("path") - .unwrap() - .as_any() - .downcast_ref::() - .unwrap(); - assert!(path_col.is_null(0)); - assert_eq!(path_col.value(1), "/x"); - } -} diff --git a/asap-precompute-rs/src/otap/records.rs b/asap-precompute-rs/src/otap/records.rs deleted file mode 100644 index 2f1ecf6..0000000 --- a/asap-precompute-rs/src/otap/records.rs +++ /dev/null @@ -1,997 +0,0 @@ -//! `OtapArrowRecords` family modelling — sibling-batch projection -//! and Strategy-B attribute-lift. -//! -//! The `OtapArrowRecords` family → flat `RecordBatch` projection -//! and Strategy-B attribute-lift on emit are deliverables that Phase B -//! deferred. This module supplies both — without depending on the -//! upstream OTAP submodule (which only mounts at Phase D's build -//! time, per Phase C's scope boundary). -//! -//! # Why a local model -//! -//! The upstream OTAP `OtapArrowRecords` Rust type lives in the -//! `otap-dataflow` workspace (the submodule the Phase D build script -//! patches). At Phase C the plugin lifecycle code is exercised via -//! the `asap-precompute-rs` test harness, **not** an OTAP runtime. -//! The plugin's contract with OTAP is: -//! -//! 1. **On receive:** OTAP hands us an `OtapArrowRecords` (sibling -//! `RecordBatch`es joined by integer ids — resource / scope / -//! metric / per-row attribute). We project that down to a single -//! flat `RecordBatch` whose schema [`super::decode_batch`] knows -//! how to walk: well-known scalar columns plus Strategy-B -//! `_asap_*` columns lifted out of the per-row attribute child -//! batch. -//! 2. **On emit:** [`super::encode_batch`] returns a flat -//! `RecordBatch` with `_asap_*` carrier columns. OTAP's strict -//! schema validator (`crates/pdata/src/schema/payloads.rs::check_match`) -//! rejects extension top-level columns on Logs/Metrics/Traces -//! record batches. -//! Phase C must lift the `_asap_*` columns onto the per-row -//! attribute child batch (`AttributeValueType::Bytes` keyed -//! entries) before passing the batch downstream. -//! -//! Both directions of this transform are **structural**, not -//! algorithmic: the per-row attribute batch is the canonical OTAP -//! Arrow attribute encoding (one row per `(parent_id, key, value)`). -//! Any binding to a real `OtapArrowRecords` Rust type at Phase D -//! becomes a thin wrapper over [`OtapMetricRecords`] below. -//! -//! # Schema -//! -//! Two batches make up the family: -//! -//! - **`metrics`** — well-known scalar columns plus a `parent_id` -//! column. Each row corresponds to one observation; `parent_id` -//! joins the row to its attribute set. -//! - **`attributes`** — three columns: `parent_id` (`UInt32`), -//! `key` (`Utf8`), and one of `bytes` (`Binary`) / `str` -//! (`Utf8`) / `int` (`UInt64`) typed value columns. Multiple rows -//! per `parent_id` represent a multi-attribute set on one parent. -//! This mirrors the upstream OTAP attribute-child-batch shape. -//! -//! Resource and scope child batches are deliberately omitted — Phase C's -//! scope boundary calls out cross-host parity (Phase E) as the place -//! that exercises the full resource scope. The `Observation::resource_labels` -//! field on the runtime side carries the resource attrs once the -//! upstream OTAP wrapper joins them on; here we model only the -//! per-row attribute batch which is the carrier for `_asap_*` keys. -//! -//! # Out of scope (Phase D) -//! -//! - Binding to the upstream `OtapArrowRecords` type itself. -//! - The `linkme` distributed-slice plugin registration. -//! - `effect_handler.send_message` / OTAP's submission API. -//! -//! Phase D wires this projection into the upstream `OtapPdata` shape; -//! the per-row attribute batch's schema we emit here is the same one -//! `crates/pdata/src/schema/payloads.rs::check_match` validates. - -use std::collections::BTreeMap; -use std::sync::Arc; - -use arrow_array::{Array, BinaryArray, RecordBatch, StringArray, UInt32Array, UInt64Array}; -use arrow_schema::{DataType, Field, Schema}; -use thiserror::Error; - -use super::schema::{ - is_reserved_column, ATTR_AGG_ID, ATTR_ENCODING, ATTR_ENVELOPE, ATTR_SCHEMA_VERSION, - ATTR_SKETCH_TYPE, ATTR_WINDOW_END_MS, ATTR_WINDOW_START_MS, -}; - -/// Local model of the upstream OTAP `OtapArrowRecords` type for -/// metrics — a sibling-batch family carrying one metrics batch and -/// one per-row attribute batch joined by integer parent ids. -/// -/// Each record in `metrics` corresponds to a single observation. Its -/// `parent_id` (`UInt32`) joins to zero or more rows in `attributes`, -/// each carrying one key / typed-value pair. -/// -/// The upstream type has additional sibling batches (resource scope, -/// scope, exemplars, …) — this minimal struct captures the two that -/// Phase C cares about. Phase D is the layer that maps to / from the -/// upstream type once the OTAP submodule is wired in. -#[derive(Debug, Clone)] -pub struct OtapMetricRecords { - /// Metrics RecordBatch: well-known scalar columns plus - /// `parent_id` (`UInt32`). - pub metrics: RecordBatch, - /// Per-row attribute child RecordBatch: `parent_id` (`UInt32`), - /// `key` (`Utf8`), and at least one typed-value column from the - /// canonical OTAP set: `bytes` (`Binary`), `str` (`Utf8`), or - /// `int` (`UInt64`). Rows are not required to be sorted by - /// `parent_id`; the projection joins by id-equality. - pub attributes: RecordBatch, -} - -/// Failure modes for [`flatten`] / [`lift`]. -#[derive(Debug, Error)] -pub enum OtapRecordsError { - /// Either of the input batches is missing a required column. - #[error("otap records: batch {batch:?} missing required column {column:?}")] - MissingColumn { - /// Which sibling batch is missing the column. - batch: &'static str, - /// Column name. - column: &'static str, - }, - /// Required column has the wrong Arrow `DataType`. - #[error( - "otap records: column {column:?} on batch {batch:?} has wrong type: expected {expected}, got {actual:?}" - )] - WrongColumnType { - /// Sibling batch. - batch: &'static str, - /// Column name. - column: &'static str, - /// Expected Arrow type, in human-readable form. - expected: &'static str, - /// Actual Arrow type observed. - actual: DataType, - }, - /// Constructing one of the output batches failed. Indicates a - /// codec bug rather than caller error — kept as an error variant - /// so the plugin shell surfaces it via OTAP's error channel - /// rather than crashing the host process. - #[error("otap records: arrow record-batch construction failed: {0}")] - ArrowError(String), -} - -/// Well-known column on the per-row attribute child batch (parent -/// join key). -pub const ATTR_BATCH_PARENT_ID: &str = "parent_id"; - -/// Well-known column on the per-row attribute child batch (key). -pub const ATTR_BATCH_KEY: &str = "key"; - -/// Well-known typed-value column on the per-row attribute child -/// batch carrying `Binary` values. -pub const ATTR_BATCH_BYTES: &str = "bytes"; - -/// Well-known typed-value column on the per-row attribute child -/// batch carrying `Utf8` values. -pub const ATTR_BATCH_STR: &str = "str"; - -/// Well-known typed-value column on the per-row attribute child -/// batch carrying `UInt64` values (also covers `UInt32`-shaped -/// schema-version values when up-cast). -pub const ATTR_BATCH_INT: &str = "int"; - -/// Project an [`OtapMetricRecords`] family down to a flat -/// `RecordBatch` whose schema matches what [`super::decode_batch`] -/// expects: well-known scalar columns (`time_unix_nano`, `metric`, -/// `value`) plus Strategy-B `_asap_*` columns lifted from the -/// per-row attribute child batch, plus any other `Utf8`-typed -/// attribute keys promoted to per-row label columns. -/// -/// Resource attrs are not yet joined here — Phase C's runtime -/// `Observation::resource_labels` is sourced separately by the -/// upstream OTAP wrapper; see the docstring on [`OtapMetricRecords`]. -/// -/// Returns a flat `RecordBatch` ready to feed into -/// [`super::decode_batch`]. The schema is the union of: -/// -/// - Every column already on `records.metrics` **except** -/// `parent_id` (consumed by the join and dropped from the flat -/// shape). -/// - Every Strategy-B `_asap_*` attribute lifted to a top-level -/// typed column (`_asap_envelope` Binary, `_asap_sketch_type` Utf8, -/// `_asap_agg_id` UInt64, `_asap_schema_version` UInt32, -/// `_asap_window_start_ms` UInt64, `_asap_window_end_ms` UInt64, -/// `_asap_encoding` Utf8). -/// - Every other `str`-valued attribute promoted to a `Utf8` label -/// column named after the attribute key. -/// -/// Where a row in `records.metrics` has no matching attribute row -/// the corresponding cell is null. -pub fn flatten(records: &OtapMetricRecords) -> Result { - let n_rows = records.metrics.num_rows(); - let parent_ids = require_uint32(&records.metrics, "metrics", ATTR_BATCH_PARENT_ID)?.clone(); - - // Index attribute rows by parent_id → vec<(key, typed value)>. - let attr_index = build_attr_index(&records.attributes)?; - - // Walk all attribute keys to discover the union of label keys - // (every `str` value whose key is non-reserved) so we know the - // schema of the flat batch up front. - let mut label_keys: BTreeMap = BTreeMap::new(); - for entries in attr_index.values() { - for (key, val) in entries { - if !is_reserved_column(key) { - if let AttrValue::Str(_) = val { - label_keys.insert(key.clone(), ()); - } - } - } - } - let label_keys: Vec = label_keys.into_keys().collect(); - - // Build per-row Strategy-B carriers + label columns in one pass. - let mut envelope_col: Vec>> = Vec::with_capacity(n_rows); - let mut sketch_type_col: Vec> = Vec::with_capacity(n_rows); - let mut agg_id_col: Vec> = Vec::with_capacity(n_rows); - let mut schema_version_col: Vec> = Vec::with_capacity(n_rows); - let mut window_start_col: Vec> = Vec::with_capacity(n_rows); - let mut window_end_col: Vec> = Vec::with_capacity(n_rows); - let mut encoding_col: Vec> = Vec::with_capacity(n_rows); - let mut label_cols: Vec>> = (0..label_keys.len()) - .map(|_| Vec::with_capacity(n_rows)) - .collect(); - - for row in 0..n_rows { - if parent_ids.is_null(row) { - envelope_col.push(None); - sketch_type_col.push(None); - agg_id_col.push(None); - schema_version_col.push(None); - window_start_col.push(None); - window_end_col.push(None); - encoding_col.push(None); - for col in &mut label_cols { - col.push(None); - } - continue; - } - let pid = parent_ids.value(row); - let entries = attr_index.get(&pid); - let mut envelope: Option> = None; - let mut sketch_type: Option = None; - let mut agg_id: Option = None; - let mut schema_version: Option = None; - let mut window_start: Option = None; - let mut window_end: Option = None; - let mut encoding: Option = None; - let mut row_labels: Vec> = vec![None; label_keys.len()]; - - if let Some(entries) = entries { - for (key, val) in entries { - match (key.as_str(), val) { - (ATTR_ENVELOPE, AttrValue::Bytes(b)) => envelope = Some(b.clone()), - (ATTR_SKETCH_TYPE, AttrValue::Str(s)) => sketch_type = Some(s.clone()), - (ATTR_AGG_ID, AttrValue::Int(v)) => agg_id = Some(*v), - (ATTR_SCHEMA_VERSION, AttrValue::Int(v)) => { - schema_version = Some(*v as u32); - } - (ATTR_WINDOW_START_MS, AttrValue::Int(v)) => window_start = Some(*v), - (ATTR_WINDOW_END_MS, AttrValue::Int(v)) => window_end = Some(*v), - (ATTR_ENCODING, AttrValue::Str(s)) => encoding = Some(s.clone()), - (k, AttrValue::Str(s)) if !is_reserved_column(k) => { - if let Some(idx) = label_keys.iter().position(|x| x == k) { - row_labels[idx] = Some(s.clone()); - } - } - _ => { - // Other attribute shapes (e.g. resource attrs - // typed as Int) are not yet mapped to label - // columns; Phase E (cross-host parity) is - // where the full resource-attr coverage lands. - } - } - } - } - envelope_col.push(envelope); - sketch_type_col.push(sketch_type); - agg_id_col.push(agg_id); - schema_version_col.push(schema_version); - window_start_col.push(window_start); - window_end_col.push(window_end); - encoding_col.push(encoding); - for (idx, val) in row_labels.into_iter().enumerate() { - label_cols[idx].push(val); - } - } - - // Carry forward every non-`parent_id`, non-Strategy-B column - // already on `records.metrics`. Phase B's `decode_batch` walks - // them by name, so keep their original schema field intact. - let mut fields: Vec = Vec::new(); - let mut columns: Vec> = Vec::new(); - for (i, field) in records.metrics.schema().fields().iter().enumerate() { - if field.name() == ATTR_BATCH_PARENT_ID { - continue; - } - // Skip any Strategy-B columns that may already be on the - // metrics batch — the attribute-lift path on emit puts them - // on the attribute batch, but a producer might double-encode - // them. Phase C prefers attribute values when they exist. - if is_strategy_b_attr(field.name()) { - continue; - } - fields.push(field.as_ref().clone()); - columns.push(records.metrics.column(i).clone()); - } - fields.push(Field::new(ATTR_ENVELOPE, DataType::Binary, true)); - columns.push(Arc::new(BinaryArray::from_opt_vec( - envelope_col.iter().map(|o| o.as_deref()).collect(), - ))); - fields.push(Field::new(ATTR_SKETCH_TYPE, DataType::Utf8, true)); - columns.push(Arc::new(StringArray::from(sketch_type_col))); - fields.push(Field::new(ATTR_AGG_ID, DataType::UInt64, true)); - columns.push(Arc::new(UInt64Array::from(agg_id_col))); - fields.push(Field::new(ATTR_SCHEMA_VERSION, DataType::UInt32, true)); - columns.push(Arc::new(UInt32Array::from(schema_version_col))); - fields.push(Field::new(ATTR_WINDOW_START_MS, DataType::UInt64, true)); - columns.push(Arc::new(UInt64Array::from(window_start_col))); - fields.push(Field::new(ATTR_WINDOW_END_MS, DataType::UInt64, true)); - columns.push(Arc::new(UInt64Array::from(window_end_col))); - fields.push(Field::new(ATTR_ENCODING, DataType::Utf8, true)); - columns.push(Arc::new(StringArray::from(encoding_col))); - for (key, col) in label_keys.iter().zip(label_cols) { - fields.push(Field::new(key, DataType::Utf8, true)); - columns.push(Arc::new(StringArray::from(col))); - } - - let schema = Arc::new(Schema::new(fields)); - RecordBatch::try_new(schema, columns).map_err(|e| OtapRecordsError::ArrowError(e.to_string())) -} - -/// Lift Strategy-B `_asap_*` carrier columns from a flat -/// `RecordBatch` onto the per-row attribute child batch, returning -/// an [`OtapMetricRecords`] family ready for OTAP downstream emit. -/// -/// This is the inverse of [`flatten`]: where [`flatten`] lowers -/// attribute rows into top-level columns for the codec to consume, -/// `lift` raises top-level Strategy-B columns into attribute rows so -/// the resulting `RecordBatch` family passes OTAP's schema validator -/// (`crates/pdata/src/schema/payloads.rs::check_match` rejects any -/// extension column on Logs/Metrics/Traces RecordBatches). -/// -/// Behaviour: -/// -/// - Strategy-B `_asap_*` columns are removed from the metrics -/// batch and re-emitted as one `(parent_id, key, value)` row per -/// non-null cell on the attribute batch (`bytes` for -/// `_asap_envelope`; `str` for `_asap_sketch_type` / `_asap_encoding`; -/// `int` for the numeric carriers). -/// - A monotonic `parent_id` (`UInt32`, `[0, num_rows)`) is added -/// to the metrics batch so each row joins back to its attributes. -/// - Other top-level `Utf8` columns (label columns from -/// [`super::encode_batch`]'s union) are also lifted to attribute -/// rows with `str` typed values, since OTAP's metrics schema -/// validator rejects them as extension columns too. Reserved -/// well-known scalar columns (`time_unix_nano`, `metric`, `value`) -/// stay on the metrics batch. -pub fn lift(flat: &RecordBatch) -> Result { - let n_rows = flat.num_rows(); - let parent_ids: Vec = (0..n_rows as u32).collect(); - - // Resolve the Strategy-B carrier columns up front so per-row - // walking is index-only. - let envelope = match flat.column_by_name(ATTR_ENVELOPE) { - Some(c) => Some(downcast_binary(c, "metrics", ATTR_ENVELOPE)?.clone()), - None => None, - }; - let sketch_type = optional_string(flat, ATTR_SKETCH_TYPE)?; - let agg_id = optional_u64(flat, ATTR_AGG_ID)?; - let schema_version = optional_u32(flat, ATTR_SCHEMA_VERSION)?; - let window_start = optional_u64(flat, ATTR_WINDOW_START_MS)?; - let window_end = optional_u64(flat, ATTR_WINDOW_END_MS)?; - let encoding = optional_string(flat, ATTR_ENCODING)?; - - // Collect non-reserved Utf8 columns as label-attribute lifts. - let label_arrays: Vec<(String, StringArray)> = flat - .schema() - .fields() - .iter() - .enumerate() - .filter_map(|(i, f)| { - let name = f.name(); - if is_reserved_column(name) { - return None; - } - if !matches!(f.data_type(), DataType::Utf8) { - return None; - } - let arr = flat - .column(i) - .as_any() - .downcast_ref::()? - .clone(); - Some((name.clone(), arr)) - }) - .collect(); - - // Build attribute rows, one per non-null `_asap_*` cell + label. - let mut attr_parent: Vec = Vec::new(); - let mut attr_key: Vec = Vec::new(); - let mut attr_bytes: Vec>> = Vec::new(); - let mut attr_str: Vec> = Vec::new(); - let mut attr_int: Vec> = Vec::new(); - - for (row, &pid) in parent_ids.iter().enumerate().take(n_rows) { - if let Some(arr) = &envelope { - if !arr.is_null(row) { - attr_parent.push(pid); - attr_key.push(ATTR_ENVELOPE.to_string()); - attr_bytes.push(Some(arr.value(row).to_vec())); - attr_str.push(None); - attr_int.push(None); - } - } - if let Some(arr) = &sketch_type { - if !arr.is_null(row) { - attr_parent.push(pid); - attr_key.push(ATTR_SKETCH_TYPE.to_string()); - attr_bytes.push(None); - attr_str.push(Some(arr.value(row).to_string())); - attr_int.push(None); - } - } - if let Some(arr) = &agg_id { - if !arr.is_null(row) { - attr_parent.push(pid); - attr_key.push(ATTR_AGG_ID.to_string()); - attr_bytes.push(None); - attr_str.push(None); - attr_int.push(Some(arr.value(row))); - } - } - if let Some(arr) = &schema_version { - if !arr.is_null(row) { - attr_parent.push(pid); - attr_key.push(ATTR_SCHEMA_VERSION.to_string()); - attr_bytes.push(None); - attr_str.push(None); - attr_int.push(Some(arr.value(row) as u64)); - } - } - if let Some(arr) = &window_start { - if !arr.is_null(row) { - attr_parent.push(pid); - attr_key.push(ATTR_WINDOW_START_MS.to_string()); - attr_bytes.push(None); - attr_str.push(None); - attr_int.push(Some(arr.value(row))); - } - } - if let Some(arr) = &window_end { - if !arr.is_null(row) { - attr_parent.push(pid); - attr_key.push(ATTR_WINDOW_END_MS.to_string()); - attr_bytes.push(None); - attr_str.push(None); - attr_int.push(Some(arr.value(row))); - } - } - if let Some(arr) = &encoding { - if !arr.is_null(row) { - attr_parent.push(pid); - attr_key.push(ATTR_ENCODING.to_string()); - attr_bytes.push(None); - attr_str.push(Some(arr.value(row).to_string())); - attr_int.push(None); - } - } - for (key, arr) in &label_arrays { - if arr.is_null(row) { - continue; - } - attr_parent.push(pid); - attr_key.push(key.clone()); - attr_bytes.push(None); - attr_str.push(Some(arr.value(row).to_string())); - attr_int.push(None); - } - } - - let attributes_schema = Arc::new(Schema::new(vec![ - Field::new(ATTR_BATCH_PARENT_ID, DataType::UInt32, false), - Field::new(ATTR_BATCH_KEY, DataType::Utf8, false), - Field::new(ATTR_BATCH_BYTES, DataType::Binary, true), - Field::new(ATTR_BATCH_STR, DataType::Utf8, true), - Field::new(ATTR_BATCH_INT, DataType::UInt64, true), - ])); - let attributes = RecordBatch::try_new( - attributes_schema, - vec![ - Arc::new(UInt32Array::from(attr_parent)), - Arc::new(StringArray::from(attr_key)), - Arc::new(BinaryArray::from_opt_vec( - attr_bytes.iter().map(|o| o.as_deref()).collect(), - )), - Arc::new(StringArray::from(attr_str)), - Arc::new(UInt64Array::from(attr_int)), - ], - ) - .map_err(|e| OtapRecordsError::ArrowError(e.to_string()))?; - - // Build the metrics-side batch: scalar columns + parent_id, no - // Strategy-B columns, no lifted label columns. - let mut fields: Vec = Vec::new(); - let mut columns: Vec> = Vec::new(); - for (i, field) in flat.schema().fields().iter().enumerate() { - let name = field.name(); - if is_strategy_b_attr(name) { - continue; - } - if !is_reserved_column(name) - && matches!(field.data_type(), DataType::Utf8) - && label_arrays.iter().any(|(k, _)| k == name) - { - // Label columns lifted to attributes — drop from metrics. - continue; - } - fields.push(field.as_ref().clone()); - columns.push(flat.column(i).clone()); - } - fields.push(Field::new(ATTR_BATCH_PARENT_ID, DataType::UInt32, false)); - columns.push(Arc::new(UInt32Array::from(parent_ids))); - let metrics_schema = Arc::new(Schema::new(fields)); - let metrics = RecordBatch::try_new(metrics_schema, columns) - .map_err(|e| OtapRecordsError::ArrowError(e.to_string()))?; - - Ok(OtapMetricRecords { - metrics, - attributes, - }) -} - -/// Returns whether `name` is a Strategy-B carrier key (i.e. a -/// `_asap_*` column the lift step removes from the top-level -/// metrics batch). -fn is_strategy_b_attr(name: &str) -> bool { - matches!( - name, - ATTR_ENVELOPE - | ATTR_SKETCH_TYPE - | ATTR_AGG_ID - | ATTR_SCHEMA_VERSION - | ATTR_WINDOW_START_MS - | ATTR_WINDOW_END_MS - | ATTR_ENCODING - ) -} - -/// Internal typed-value carrier for a single attribute cell. -#[derive(Debug, Clone)] -enum AttrValue { - Bytes(Vec), - Str(String), - Int(u64), -} - -fn build_attr_index( - attributes: &RecordBatch, -) -> Result>, OtapRecordsError> { - let parent_ids = require_uint32(attributes, "attributes", ATTR_BATCH_PARENT_ID)?; - let keys = require_string(attributes, "attributes", ATTR_BATCH_KEY)?; - let bytes = optional_binary(attributes, ATTR_BATCH_BYTES)?; - let strs = optional_string(attributes, ATTR_BATCH_STR)?; - let ints = optional_u64(attributes, ATTR_BATCH_INT)?; - - let mut out: BTreeMap> = BTreeMap::new(); - for row in 0..attributes.num_rows() { - if parent_ids.is_null(row) || keys.is_null(row) { - continue; - } - let pid = parent_ids.value(row); - let key = keys.value(row).to_string(); - let val = if let Some(arr) = &bytes { - if !arr.is_null(row) { - AttrValue::Bytes(arr.value(row).to_vec()) - } else if let Some(arr) = &strs { - if !arr.is_null(row) { - AttrValue::Str(arr.value(row).to_string()) - } else if let Some(arr) = &ints { - if !arr.is_null(row) { - AttrValue::Int(arr.value(row)) - } else { - continue; - } - } else { - continue; - } - } else if let Some(arr) = &ints { - if !arr.is_null(row) { - AttrValue::Int(arr.value(row)) - } else { - continue; - } - } else { - continue; - } - } else if let Some(arr) = &strs { - if !arr.is_null(row) { - AttrValue::Str(arr.value(row).to_string()) - } else if let Some(arr) = &ints { - if !arr.is_null(row) { - AttrValue::Int(arr.value(row)) - } else { - continue; - } - } else { - continue; - } - } else if let Some(arr) = &ints { - if !arr.is_null(row) { - AttrValue::Int(arr.value(row)) - } else { - continue; - } - } else { - continue; - }; - out.entry(pid).or_default().push((key, val)); - } - Ok(out) -} - -fn require_uint32<'a>( - batch: &'a RecordBatch, - label: &'static str, - column: &'static str, -) -> Result<&'a UInt32Array, OtapRecordsError> { - let col = batch - .column_by_name(column) - .ok_or(OtapRecordsError::MissingColumn { - batch: label, - column, - })?; - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapRecordsError::WrongColumnType { - batch: label, - column, - expected: "UInt32", - actual: col.data_type().clone(), - }) -} - -fn require_string<'a>( - batch: &'a RecordBatch, - label: &'static str, - column: &'static str, -) -> Result<&'a StringArray, OtapRecordsError> { - let col = batch - .column_by_name(column) - .ok_or(OtapRecordsError::MissingColumn { - batch: label, - column, - })?; - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapRecordsError::WrongColumnType { - batch: label, - column, - expected: "Utf8", - actual: col.data_type().clone(), - }) -} - -fn optional_binary<'a>( - batch: &'a RecordBatch, - column: &'static str, -) -> Result, OtapRecordsError> { - Ok(match batch.column_by_name(column) { - Some(col) => Some(downcast_binary(col, "attributes", column)?), - None => None, - }) -} - -fn optional_string( - batch: &RecordBatch, - column: &'static str, -) -> Result, OtapRecordsError> { - Ok(match batch.column_by_name(column) { - Some(col) => Some( - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapRecordsError::WrongColumnType { - batch: "attributes", - column, - expected: "Utf8", - actual: col.data_type().clone(), - })? - .clone(), - ), - None => None, - }) -} - -fn optional_u64( - batch: &RecordBatch, - column: &'static str, -) -> Result, OtapRecordsError> { - Ok(match batch.column_by_name(column) { - Some(col) => Some( - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapRecordsError::WrongColumnType { - batch: "attributes", - column, - expected: "UInt64", - actual: col.data_type().clone(), - })? - .clone(), - ), - None => None, - }) -} - -fn optional_u32( - batch: &RecordBatch, - column: &'static str, -) -> Result, OtapRecordsError> { - Ok(match batch.column_by_name(column) { - Some(col) => Some( - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapRecordsError::WrongColumnType { - batch: "attributes", - column, - expected: "UInt32", - actual: col.data_type().clone(), - })? - .clone(), - ), - None => None, - }) -} - -fn downcast_binary<'a>( - col: &'a Arc, - label: &'static str, - column: &'static str, -) -> Result<&'a BinaryArray, OtapRecordsError> { - col.as_any() - .downcast_ref::() - .ok_or_else(|| OtapRecordsError::WrongColumnType { - batch: label, - column, - expected: "Binary", - actual: col.data_type().clone(), - }) -} - -#[cfg(test)] -mod tests { - use super::super::schema::{COLUMN_METRIC, COLUMN_TIME_UNIX_NANO, COLUMN_VALUE}; - use super::*; - use arrow_array::Float64Array; - use arrow_schema::{Field, Schema}; - - /// Build a minimal `OtapMetricRecords` family carrying one - /// envelope-bearing row with all Strategy-B carriers set on the - /// attribute batch. - fn fixture_records() -> OtapMetricRecords { - let metrics_schema = Arc::new(Schema::new(vec![ - Field::new(COLUMN_TIME_UNIX_NANO, DataType::UInt64, false), - Field::new(COLUMN_METRIC, DataType::Utf8, false), - Field::new(COLUMN_VALUE, DataType::Float64, true), - Field::new(ATTR_BATCH_PARENT_ID, DataType::UInt32, false), - ])); - let metrics = RecordBatch::try_new( - metrics_schema, - vec![ - Arc::new(UInt64Array::from(vec![1_000_000_u64])), // 1ms - Arc::new(StringArray::from(vec!["http_latency"])), - Arc::new(Float64Array::from(vec![Some(2.5)])), - Arc::new(UInt32Array::from(vec![0_u32])), - ], - ) - .expect("build metrics"); - - let attributes_schema = Arc::new(Schema::new(vec![ - Field::new(ATTR_BATCH_PARENT_ID, DataType::UInt32, false), - Field::new(ATTR_BATCH_KEY, DataType::Utf8, false), - Field::new(ATTR_BATCH_BYTES, DataType::Binary, true), - Field::new(ATTR_BATCH_STR, DataType::Utf8, true), - Field::new(ATTR_BATCH_INT, DataType::UInt64, true), - ])); - let attr_parent: Vec = vec![0, 0, 0, 0, 0, 0, 0, 0]; - let attr_key: Vec<&str> = vec![ - ATTR_ENVELOPE, - ATTR_SKETCH_TYPE, - ATTR_AGG_ID, - ATTR_SCHEMA_VERSION, - ATTR_WINDOW_START_MS, - ATTR_WINDOW_END_MS, - ATTR_ENCODING, - "region", // arbitrary label attribute - ]; - let payload: &[u8] = &[0xde, 0xad, 0xbe, 0xef]; - let attr_bytes: Vec> = - vec![Some(payload), None, None, None, None, None, None, None]; - let attr_str: Vec> = vec![ - None, - Some("DDSketch"), - None, - None, - None, - None, - Some("PROTO_FULL"), - Some("us-east"), - ]; - let attr_int: Vec> = vec![ - None, - None, - Some(42), - Some(1), - Some(1_000), - Some(2_000), - None, - None, - ]; - let attributes = RecordBatch::try_new( - attributes_schema, - vec![ - Arc::new(UInt32Array::from(attr_parent)), - Arc::new(StringArray::from(attr_key)), - Arc::new(BinaryArray::from_opt_vec(attr_bytes)), - Arc::new(StringArray::from(attr_str)), - Arc::new(UInt64Array::from(attr_int)), - ], - ) - .expect("build attributes"); - OtapMetricRecords { - metrics, - attributes, - } - } - - #[test] - fn flatten_lifts_strategy_b_attrs_to_top_level_columns() { - let records = fixture_records(); - let flat = flatten(&records).expect("flatten"); - assert_eq!(flat.num_rows(), 1); - - // Strategy-B columns were promoted from attributes to top-level. - let env = flat - .column_by_name(ATTR_ENVELOPE) - .expect("envelope column") - .as_any() - .downcast_ref::() - .expect("Binary"); - assert_eq!(env.value(0), &[0xde, 0xad, 0xbe, 0xef]); - - let sk = flat - .column_by_name(ATTR_SKETCH_TYPE) - .unwrap() - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(sk.value(0), "DDSketch"); - - let agg = flat - .column_by_name(ATTR_AGG_ID) - .unwrap() - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(agg.value(0), 42); - - let region = flat - .column_by_name("region") - .unwrap() - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(region.value(0), "us-east"); - - // parent_id was consumed by the join. - assert!(flat.column_by_name(ATTR_BATCH_PARENT_ID).is_none()); - } - - #[test] - fn flatten_then_decode_round_trips_envelope_payload() { - let records = fixture_records(); - let flat = flatten(&records).expect("flatten"); - let observations = super::super::decode_batch(&flat).expect("decode"); - assert_eq!(observations.len(), 1); - let env = observations[0] - .value - .envelope - .as_ref() - .expect("envelope routed through KindEnvelope"); - assert_eq!(env.payload, vec![0xde, 0xad, 0xbe, 0xef]); - assert_eq!(env.agg_id, 42); - assert_eq!(env.sketch_type, crate::envelope::SketchType::DDSketch); - } - - #[test] - fn lift_then_flatten_round_trips() { - // Start with what `encode_batch` would produce (a flat batch - // with `_asap_*` top-level columns). - let env = crate::envelope::SketchEnvelope { - schema_version: 1, - sketch_type: crate::envelope::SketchType::DDSketch, - agg_id: 7, - resource_labels: vec![], - labels: vec![crate::observation::KeyValue::new("path", "/api")], - window_start_ms: 1_000, - window_end_ms: 2_000, - encoding: crate::envelope::Encoding::ProtoFull, - payload: vec![1, 2, 3, 4], - hash_spec: None, - metric_name: "m".into(), - count: 0, - aggregation_temporality: 0, - value: 0.0, - }; - let flat = super::super::encode_batch(std::slice::from_ref(&env)).expect("encode"); - let lifted = lift(&flat).expect("lift"); - - // Lift removes Strategy-B columns from the metrics batch; the - // top-level metrics schema must NOT carry `_asap_*` columns - // (this is what passes OTAP's strict validator). - for name in [ - ATTR_ENVELOPE, - ATTR_SKETCH_TYPE, - ATTR_AGG_ID, - ATTR_SCHEMA_VERSION, - ATTR_WINDOW_START_MS, - ATTR_WINDOW_END_MS, - ATTR_ENCODING, - "path", // also lifted - ] { - assert!( - lifted.metrics.column_by_name(name).is_none(), - "metrics batch must not carry top-level {name}" - ); - } - // parent_id is on the metrics side. - assert!(lifted - .metrics - .column_by_name(ATTR_BATCH_PARENT_ID) - .is_some()); - - // Round-trip the lifted family back through flatten and - // confirm the envelope payload survives intact. - let re_flat = flatten(&lifted).expect("flatten"); - let observations = super::super::decode_batch(&re_flat).expect("decode"); - assert_eq!(observations.len(), 1); - let decoded = observations[0] - .value - .envelope - .as_ref() - .expect("envelope kind preserved"); - assert_eq!(decoded.payload, env.payload); - assert_eq!(decoded.sketch_type, env.sketch_type); - assert_eq!(decoded.agg_id, env.agg_id); - // Labels survive via the `path` attribute lift. - assert_eq!(decoded.labels, env.labels); - } - - #[test] - fn flatten_preserves_scalar_columns_when_no_envelope_attr() { - // Build a metrics batch with only scalar columns + parent_id; - // attributes batch is empty. Decode should observe a Float row. - let metrics_schema = Arc::new(Schema::new(vec![ - Field::new(COLUMN_TIME_UNIX_NANO, DataType::UInt64, false), - Field::new(COLUMN_METRIC, DataType::Utf8, false), - Field::new(COLUMN_VALUE, DataType::Float64, false), - Field::new(ATTR_BATCH_PARENT_ID, DataType::UInt32, false), - ])); - let metrics = RecordBatch::try_new( - metrics_schema, - vec![ - Arc::new(UInt64Array::from(vec![1_000_000_u64])), - Arc::new(StringArray::from(vec!["m"])), - Arc::new(Float64Array::from(vec![2.71_f64])), - Arc::new(UInt32Array::from(vec![0_u32])), - ], - ) - .expect("metrics"); - let attributes_schema = Arc::new(Schema::new(vec![ - Field::new(ATTR_BATCH_PARENT_ID, DataType::UInt32, false), - Field::new(ATTR_BATCH_KEY, DataType::Utf8, false), - Field::new(ATTR_BATCH_BYTES, DataType::Binary, true), - Field::new(ATTR_BATCH_STR, DataType::Utf8, true), - Field::new(ATTR_BATCH_INT, DataType::UInt64, true), - ])); - let attributes = RecordBatch::new_empty(attributes_schema); - let records = OtapMetricRecords { - metrics, - attributes, - }; - - let flat = flatten(&records).expect("flatten"); - let obs = super::super::decode_batch(&flat).expect("decode"); - assert_eq!(obs.len(), 1); - assert_eq!( - obs[0].value.kind, - crate::observation::ObservationValueKind::Float - ); - assert_eq!(obs[0].value.float, 2.71_f64); - } -} diff --git a/asap-precompute-rs/tests/api_surface.rs b/asap-precompute-rs/tests/api_surface.rs deleted file mode 100644 index 4baec23..0000000 --- a/asap-precompute-rs/tests/api_surface.rs +++ /dev/null @@ -1,260 +0,0 @@ -//! Type-level smoke tests for the bootstrap public API. -//! -//! These tests don't exercise the runtime (which is -//! `unimplemented!()` here). They lock in the trait surface, the -//! constructor shapes, and the serde round-trip behavior so -//! subsequent changes notice if anything drifts. - -use asap_precompute_rs::adapter::Adapter; -use asap_precompute_rs::control_channel::ControlChannel; -use asap_precompute_rs::precompute::{ - BoxedObserver, FrequencyEntry, Precompute, PrecomputeImpl, SketchObserver, -}; -use asap_precompute_rs::{ - AggregationMode, Encoding, KeyValue, LabelMatcher, MatchOp, Observation, ObservationValue, - ObservationValueKind, OnOverflow, PrecomputeConfig, PrecomputeConfigSet, SketchEnvelope, - SketchType, -}; - -// --------------------------------------------------------------- -// Constructors - -#[test] -fn observation_new_constructs_with_all_four_value_kinds() { - let labels = vec![KeyValue::new("path", "/api")]; - - let o_float = Observation::new( - 1_000, - "metric", - vec![], - labels.clone(), - ObservationValue::float(1.5), - ); - assert_eq!(o_float.value.kind, ObservationValueKind::Float); - - let o_hash = Observation::new( - 1_000, - "metric", - vec![], - labels.clone(), - ObservationValue::hash(0xDEADBEEF), - ); - assert_eq!(o_hash.value.kind, ObservationValueKind::Hash); - - let o_bytes = Observation::new( - 1_000, - "metric", - vec![], - labels.clone(), - ObservationValue::bytes(b"opaque-key".to_vec()), - ); - assert_eq!(o_bytes.value.kind, ObservationValueKind::Bytes); - - let env = SketchEnvelope { - schema_version: 1, - sketch_type: SketchType::DDSketch, - encoding: Encoding::ProtoFull, - ..Default::default() - }; - let o_env = Observation::new( - 1_000, - "metric", - vec![], - labels, - ObservationValue::envelope(env), - ); - assert_eq!(o_env.value.kind, ObservationValueKind::Envelope); -} - -#[test] -fn precompute_config_default_has_sensible_defaults() { - let cfg = PrecomputeConfig::default(); - assert_eq!(cfg.agg_id, 0); - assert_eq!(cfg.sketch_type, SketchType::Unspecified); - assert_eq!(cfg.mode, AggregationMode::Tumbling); - assert_eq!(cfg.on_overflow, OnOverflow::Drop); - assert_eq!(cfg.encoding, Encoding::Unspecified); - assert!(!cfg.delta_transmission); - assert!(!cfg.global_aggregation); - assert!(!cfg.omit_resource_attrs); - assert!(!cfg.emit_window_stats); - assert_eq!(cfg.matchers.len(), 0); - assert_eq!(cfg.aggregate_by.len(), 0); -} - -#[test] -fn label_matcher_default_op_returns_equal() { - assert_eq!(LabelMatcher::default_op(), MatchOp::Equal); - let m = LabelMatcher::default(); - assert_eq!(m.op, MatchOp::Equal); -} - -// --------------------------------------------------------------- -// SketchEnvelope serde round-trip - -#[test] -fn sketch_envelope_round_trips_through_serde_json() { - let env = SketchEnvelope { - schema_version: 1, - sketch_type: SketchType::CountMinSketch, - agg_id: 1234, - resource_labels: vec![KeyValue::new("region", "us-west-2")], - labels: vec![ - KeyValue::new("status", "200"), - KeyValue::new("path", "/api"), - ], - window_start_ms: 1_700_000_000_000, - window_end_ms: 1_700_000_060_000, - encoding: Encoding::ProtoDelta, - payload: vec![0x01, 0x02, 0x03, 0x04, 0x05], - hash_spec: None, - metric_name: "http_request_count".into(), - count: 4096, - aggregation_temporality: 1, - value: 0.0, - }; - let json = serde_json::to_string(&env).expect("serialize"); - let decoded: SketchEnvelope = serde_json::from_str(&json).expect("deserialize"); - assert_eq!(env, decoded); -} - -#[test] -fn precompute_config_set_round_trips_through_serde_json() { - let cs = PrecomputeConfigSet { - version: 17, - configs: vec![ - PrecomputeConfig { - agg_id: 1, - sketch_type: SketchType::DDSketch, - ..Default::default() - }, - PrecomputeConfig { - agg_id: 2, - sketch_type: SketchType::HLLSketch, - ..Default::default() - }, - ], - }; - let json = serde_json::to_string(&cs).expect("serialize"); - let decoded: PrecomputeConfigSet = serde_json::from_str(&json).expect("deserialize"); - assert_eq!(cs, decoded); - assert_eq!(decoded.find_by_agg_id(2).map(|c| c.agg_id), Some(2)); -} - -// --------------------------------------------------------------- -// ControlChannel trait can be implemented by a stub - -struct NoOpChannel; - -impl ControlChannel for NoOpChannel { - fn poll(&self) -> Option { - None - } - fn ack(&self, _plan_version: u64) {} -} - -#[test] -fn control_channel_trait_implementable_by_stub() { - let c: Box = Box::new(NoOpChannel); - assert!(c.poll().is_none()); - c.ack(99); -} - -// --------------------------------------------------------------- -// Adapter trait can be implemented by a stub - -struct NoOpAdapter; - -impl Adapter for NoOpAdapter { - type Event = (); - - fn decode( - &self, - _ev: Self::Event, - ) -> Result, asap_precompute_rs::precompute::PrecomputeError> { - Ok(vec![]) - } - - fn encode( - &self, - _envelopes: &[SketchEnvelope], - ) -> Result { - Ok(()) - } - - fn schedule_tick( - &self, - _period: std::time::Duration, - _cb: Box, - ) -> asap_precompute_rs::adapter::CancelTick { - Box::new(|| {}) - } - - fn emit_telemetry(&self, _stats: &asap_precompute_rs::precompute::StatsSnapshot) {} -} - -#[test] -fn adapter_trait_implementable_by_stub() { - let a = NoOpAdapter; - let envs: Vec = vec![]; - assert!(a.encode(&envs).is_ok()); - assert!(a.decode(()).is_ok()); -} - -// --------------------------------------------------------------- -// SketchObserver trait can be implemented by a stub (proves the -// trait is dyn-safe and accepts the documented value shapes). - -struct DropObserver; - -impl SketchObserver for DropObserver { - fn observe( - &self, - _sketch: &mut dyn asap_precompute_rs::Sketch, - _obs: &Observation, - ) -> Result<(), asap_precompute_rs::precompute::PrecomputeError> { - Ok(()) - } -} - -#[test] -fn sketch_observer_trait_implementable_by_stub() { - let _: BoxedObserver = Box::new(DropObserver); -} - -// --------------------------------------------------------------- -// PrecomputeImpl constructs without runtime wiring. - -#[test] -fn precompute_impl_constructs_with_no_config() { - let p = PrecomputeImpl::new(None, None, None); - assert_eq!(p.sketch_type(), SketchType::Unspecified); - assert!(!p.is_closed()); - p.shutdown().expect("shutdown succeeds"); - assert!(p.is_closed()); -} - -#[test] -fn precompute_impl_update_config_swaps_active() { - let p = PrecomputeImpl::new(None, None, None); - let cs = PrecomputeConfigSet { - version: 1, - configs: vec![PrecomputeConfig { - agg_id: 7, - sketch_type: SketchType::DDSketch, - ..Default::default() - }], - }; - p.update_config(&cs); - // stats() returns the empty snapshot — runtime fields - // remain zero until the runtime is wired up. - let s = p.stats(); - assert_eq!(s.input_observations, 0); -} - -#[test] -fn frequency_entry_default_is_empty() { - let e = FrequencyEntry::default(); - assert!(e.key.is_empty()); - assert_eq!(e.count, 0.0); -} diff --git a/asap-precompute-rs/tests/otap_codec.rs b/asap-precompute-rs/tests/otap_codec.rs deleted file mode 100644 index 1a2eaef..0000000 --- a/asap-precompute-rs/tests/otap_codec.rs +++ /dev/null @@ -1,237 +0,0 @@ -//! Integration tests for the OTAP-Rust codec. -//! -//! These exercise the codec end-to-end as a stable public surface, -//! complementing the unit tests in `src/otap/{decode,encode,schema,plugin}.rs`. -//! -//! Coverage map: -//! - decode-then-encode round trip preserves payload bytes per row. -//! - encode-then-decode round trip preserves envelope bytes. -//! - per-sketch-type (DDSketch / KLL / HLL / CountSketch / CountMinSketch) -//! envelopes round-trip with their `payload` field intact — the -//! codec is sketch-agnostic, but the test asserts that promise. - -#![cfg(feature = "otap")] - -use std::sync::Arc; - -use arrow_array::{ - Array, BinaryArray, Float64Array, RecordBatch, StringArray, UInt32Array, UInt64Array, -}; -use arrow_schema::{DataType, Field, Schema}; - -use asap_precompute_rs::envelope::{Encoding, SketchEnvelope, SketchType}; -use asap_precompute_rs::observation::{KeyValue, ObservationValueKind}; -use asap_precompute_rs::otap::{ - decode_batch, encode_batch, ATTR_AGG_ID, ATTR_ENCODING, ATTR_ENVELOPE, ATTR_SCHEMA_VERSION, - ATTR_SKETCH_TYPE, ATTR_WINDOW_END_MS, ATTR_WINDOW_START_MS, COLUMN_METRIC, - COLUMN_TIME_UNIX_NANO, COLUMN_VALUE, -}; - -fn envelope_for(sketch_type: SketchType, payload: &[u8]) -> SketchEnvelope { - SketchEnvelope { - schema_version: 1, - sketch_type, - agg_id: 1234, - resource_labels: vec![], - labels: vec![ - KeyValue::new("path", "/api"), - KeyValue::new("region", "us-east"), - ], - window_start_ms: 1_000, - window_end_ms: 2_000, - encoding: Encoding::ProtoFull, - payload: payload.to_vec(), - hash_spec: None, - metric_name: format!("metric_{}", sketch_type.name()), - count: 0, - aggregation_temporality: 0, - value: 0.0, - } -} - -/// Build a hand-crafted Strategy-B RecordBatch carrying envelope rows -/// for every sketch type in `entries`. Used to assert the -/// decode → encode → decode round trip preserves payload bytes per row. -fn batch_from_envelope_entries(entries: &[(SketchType, &[u8])]) -> RecordBatch { - let metric_col: Vec<&str> = entries - .iter() - .map(|(t, _)| match t { - SketchType::DDSketch => "metric_DDSketch", - SketchType::KLLSketch => "metric_KLLSketch", - SketchType::HLLSketch => "metric_HLLSketch", - SketchType::CountSketch => "metric_CountSketch", - SketchType::CountMinSketch => "metric_CountMinSketch", - SketchType::CountMinSketchWithHeap => "metric_CountMinSketchWithHeap", - SketchType::Unspecified => "metric_Unspecified", - }) - .collect(); - let envelope_col: Vec<&[u8]> = entries.iter().map(|(_, p)| *p).collect(); - let sketch_type_col: Vec<&str> = entries.iter().map(|(t, _)| t.name()).collect(); - let agg_id_col: Vec = entries.iter().map(|_| 1234_u64).collect(); - let schema_version_col: Vec = entries.iter().map(|_| 1_u32).collect(); - let window_start_col: Vec = entries.iter().map(|_| 1_000_u64).collect(); - let window_end_col: Vec = entries.iter().map(|_| 2_000_u64).collect(); - let encoding_col: Vec<&str> = entries.iter().map(|_| "PROTO_FULL").collect(); - - let schema = Arc::new(Schema::new(vec![ - Field::new(COLUMN_METRIC, DataType::Utf8, false), - Field::new(ATTR_ENVELOPE, DataType::Binary, false), - Field::new(ATTR_SKETCH_TYPE, DataType::Utf8, false), - Field::new(ATTR_AGG_ID, DataType::UInt64, false), - Field::new(ATTR_SCHEMA_VERSION, DataType::UInt32, false), - Field::new(ATTR_WINDOW_START_MS, DataType::UInt64, false), - Field::new(ATTR_WINDOW_END_MS, DataType::UInt64, false), - Field::new(ATTR_ENCODING, DataType::Utf8, false), - ])); - RecordBatch::try_new( - schema, - vec![ - Arc::new(StringArray::from(metric_col)), - Arc::new(BinaryArray::from(envelope_col)), - Arc::new(StringArray::from(sketch_type_col)), - Arc::new(UInt64Array::from(agg_id_col)), - Arc::new(UInt32Array::from(schema_version_col)), - Arc::new(UInt64Array::from(window_start_col)), - Arc::new(UInt64Array::from(window_end_col)), - Arc::new(StringArray::from(encoding_col)), - ], - ) - .expect("build batch") -} - -#[test] -fn decode_then_encode_preserves_payload_bytes_per_row() { - // Mix of distinct payloads across rows to verify the codec - // doesn't accidentally hoist the first row's bytes onto every - // subsequent row. - let p0 = vec![0xde, 0xad, 0xbe, 0xef]; - let p1 = vec![0x01, 0x02, 0x03]; - let p2 = (0..=255_u8).collect::>(); - let batch_in = batch_from_envelope_entries(&[ - (SketchType::DDSketch, &p0), - (SketchType::KLLSketch, &p1), - (SketchType::HLLSketch, &p2), - ]); - - let observations = decode_batch(&batch_in).expect("decode"); - assert_eq!(observations.len(), 3); - - let envelopes: Vec = observations - .into_iter() - .map(|obs| { - assert_eq!(obs.value.kind, ObservationValueKind::Envelope); - *obs.value - .envelope - .expect("decoded row should be envelope-kind") - }) - .collect(); - - let batch_out = encode_batch(&envelopes).expect("encode"); - assert_eq!(batch_out.num_rows(), 3); - - let env_col = batch_out - .column_by_name(ATTR_ENVELOPE) - .expect("envelope column on encoded batch") - .as_any() - .downcast_ref::() - .expect("Binary type"); - assert_eq!(env_col.value(0), p0.as_slice()); - assert_eq!(env_col.value(1), p1.as_slice()); - assert_eq!(env_col.value(2), p2.as_slice()); -} - -#[test] -fn encode_then_decode_preserves_envelope_bytes() { - let envelopes = vec![ - envelope_for(SketchType::DDSketch, &[1, 1, 1, 1]), - envelope_for(SketchType::KLLSketch, &[2, 2, 2]), - envelope_for( - SketchType::CountMinSketch, - &(0..128_u8).collect::>(), - ), - ]; - - let batch = encode_batch(&envelopes).expect("encode"); - let observations = decode_batch(&batch).expect("decode"); - assert_eq!(observations.len(), envelopes.len()); - - for (obs, original) in observations.iter().zip(envelopes.iter()) { - assert_eq!(obs.value.kind, ObservationValueKind::Envelope); - let decoded = obs.value.envelope.as_ref().expect("envelope present"); - assert_eq!(decoded.payload, original.payload); - assert_eq!(decoded.sketch_type, original.sketch_type); - assert_eq!(decoded.agg_id, original.agg_id); - assert_eq!(decoded.schema_version, original.schema_version); - assert_eq!(decoded.window_start_ms, original.window_start_ms); - assert_eq!(decoded.window_end_ms, original.window_end_ms); - assert_eq!(decoded.encoding, original.encoding); - assert_eq!(decoded.metric_name, original.metric_name); - // Labels round-trip through the union-of-keys label column - // construction. - assert_eq!(decoded.labels, original.labels); - } -} - -#[test] -fn per_sketch_type_round_trip_smoke() { - // The codec is sketch-agnostic — it carries opaque envelope - // bytes — but all five sketch types must round-trip through - // encode/decode without payload corruption. Treat each as a - // smoke test. - for (sketch_type, payload) in [ - (SketchType::DDSketch, vec![0xaa; 32]), - (SketchType::KLLSketch, vec![0xbb; 64]), - (SketchType::HLLSketch, vec![0xcc; 16]), - (SketchType::CountSketch, vec![0xdd; 128]), - (SketchType::CountMinSketch, vec![0xee; 256]), - (SketchType::CountMinSketchWithHeap, vec![0xef; 96]), - ] { - let env = envelope_for(sketch_type, &payload); - let batch = encode_batch(std::slice::from_ref(&env)).expect("encode"); - let obs = decode_batch(&batch).expect("decode"); - assert_eq!(obs.len(), 1); - assert_eq!(obs[0].value.kind, ObservationValueKind::Envelope); - let decoded = obs[0].value.envelope.as_ref().expect("envelope"); - assert_eq!( - decoded.payload, payload, - "payload corruption for sketch type {:?}", - sketch_type - ); - assert_eq!(decoded.sketch_type, sketch_type); - } -} - -#[test] -fn scalar_then_back_round_trip_via_observation_does_not_corrupt_metric_name() { - // Scalar (non-envelope) decode path: the codec emits a Float - // observation. Encode is envelope-direction only by design (encode - // is the egress path — ingest is decode-only). This test pins the - // asymmetry: encoding a Float observation back to OTAP is NOT in the - // codec's surface because the runtime never asks the codec to encode - // raw scalars — only `Precompute::tick` envelopes flow through encode. - let schema = Arc::new(Schema::new(vec![ - Field::new(COLUMN_TIME_UNIX_NANO, DataType::UInt64, false), - Field::new(COLUMN_METRIC, DataType::Utf8, false), - Field::new(COLUMN_VALUE, DataType::Float64, false), - Field::new("k1", DataType::Utf8, true), - ])); - let batch = RecordBatch::try_new( - schema, - vec![ - Arc::new(UInt64Array::from(vec![1_000_000_u64; 3])), // 1ms - Arc::new(StringArray::from(vec!["m1", "m2", "m3"])), - Arc::new(Float64Array::from(vec![1.0_f64, 2.0, 3.0])), - Arc::new(StringArray::from(vec![Some("v"), None, Some("z")])), - ], - ) - .expect("build batch"); - - let obs = decode_batch(&batch).expect("decode"); - assert_eq!(obs.len(), 3); - assert_eq!(obs[0].metric, "m1"); - assert_eq!(obs[0].value.kind, ObservationValueKind::Float); - assert_eq!(obs[0].value.float, 1.0); - assert_eq!(obs[0].labels, vec![KeyValue::new("k1", "v")]); - // Row 1 had a null label cell; decode skips it. - assert!(obs[1].labels.is_empty()); -} diff --git a/asap-precompute-rs/tests/otap_multiprocess_e2e.rs b/asap-precompute-rs/tests/otap_multiprocess_e2e.rs deleted file mode 100644 index b37d306..0000000 --- a/asap-precompute-rs/tests/otap_multiprocess_e2e.rs +++ /dev/null @@ -1,244 +0,0 @@ -#![cfg(feature = "otap-engine")] - -use asap_precompute_rs::otap::codec::decode_pdata_to_observations; -use asap_sketchlib::sketches::KLL; -use otel_arrow_dfe_config::SignalType; -use otel_arrow_dfe_otap::pdata::OtapPdata; -use otel_arrow_dfe_pdata::OtlpProtoBytes; -use serde_json::Value; -use std::collections::HashSet; -use std::fs; -use std::path::{Path, PathBuf}; -use std::process::Command; -use std::time::{SystemTime, UNIX_EPOCH}; - -const POINTS_PER_SOURCE: u64 = 25_000; - -fn unique_dir() -> PathBuf { - let id = SystemTime::now() - .duration_since(UNIX_EPOCH) - .expect("clock after epoch") - .as_nanos(); - std::env::temp_dir().join(format!("asap-otap-e2e-{}-{id}", std::process::id())) -} - -fn read_observations(path: &Path) -> Vec { - let bytes = fs::read(path).expect("read OTLP artifact"); - let pdata = OtapPdata::new_todo_context( - OtlpProtoBytes::new_from_bytes(SignalType::Metrics, bytes).into(), - ); - decode_pdata_to_observations(pdata) - .expect("decode OTLP artifact") - .observations -} - -fn run_scenario(scenario: &str, points: u64) -> (PathBuf, Value) { - let dir = unique_dir().join(scenario); - fs::create_dir_all(&dir).expect("create scenario directory"); - let manifest_path = dir.join("result.json"); - let output = Command::new(env!("CARGO_BIN_EXE_asap-otap-demo")) - .args([ - "--scenario", - scenario, - "--output-dir", - dir.to_str().unwrap(), - "--result-manifest", - manifest_path.to_str().unwrap(), - "--points-per-source", - &points.to_string(), - ]) - .output() - .expect("run scenario"); - assert!( - output.status.success(), - "{scenario} failed\nstdout:\n{}\nstderr:\n{}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ); - let manifest = serde_json::from_slice(&fs::read(manifest_path).expect("manifest")) - .expect("valid manifest"); - (dir, manifest) -} - -/// Scenario: Raw baseline sends two sources through branch, merge, and final RuntimePipelines. -/// Guarantees: All raw observations reach the backend and every stage has a distinct PID. -#[test] -fn raw_baseline_uses_four_processes_and_preserves_every_signal() { - let (dir, manifest) = run_scenario("raw", 100); - assert!(manifest["pipeline_elapsed_nanoseconds"].as_u64().unwrap() > 0); - assert_eq!(read_observations(&dir.join("out.otlp")).len(), 200); - let processors = manifest["processors"].as_array().unwrap(); - assert_eq!(processors.len(), 4); - assert_eq!( - processors - .iter() - .map(|processor| processor["pid"].as_u64().unwrap()) - .collect::>() - .len(), - 4 - ); -} - -/// Scenario: Exact baseline sorts two source runs, merges them, and estimates p50/p99. -/// Guarantees: The backend receives exact order-statistic results from four distinct processes. -#[test] -fn exact_baseline_uses_four_processes_and_returns_exact_quantiles() { - let (dir, manifest) = run_scenario("exact", 100); - let output = read_observations(&dir.join("out.otlp")); - let mut values = output - .iter() - .map(|observation| observation.value.float) - .collect::>(); - values.sort_by(f64::total_cmp); - assert_eq!(values, vec![101.0, 198.0]); - assert_eq!( - manifest["processors"] - .as_array() - .unwrap() - .iter() - .map(|processor| processor["pid"].as_u64().unwrap()) - .collect::>() - .len(), - 4 - ); -} - -#[test] -fn semantic_traffic_uses_generator_processes_and_records_stage_resources() { - let dir = unique_dir(); - fs::create_dir_all(&dir).unwrap(); - let manifest_path = dir.join("result.json"); - let output = Command::new(env!("CARGO_BIN_EXE_asap-otap-demo")) - .args([ - "--scenario", - "kll", - "--traffic", - "semantic", - "--generator-threads", - "2", - "--points-per-source", - "1000", - "--output-dir", - dir.to_str().unwrap(), - "--result-manifest", - manifest_path.to_str().unwrap(), - ]) - .output() - .unwrap(); - assert!( - output.status.success(), - "{}", - String::from_utf8_lossy(&output.stderr) - ); - let manifest: Value = serde_json::from_slice(&fs::read(manifest_path).unwrap()).unwrap(); - assert!(manifest["pipeline_elapsed_nanoseconds"].as_u64().unwrap() > 0); - assert!(manifest["validation_elapsed_nanoseconds"].as_u64().unwrap() > 0); - for stage in ["a", "b", "sa", "sb", "merged", "out"] { - let metric: Value = - serde_json::from_slice(&fs::read(dir.join(format!("{stage}.metrics.json"))).unwrap()) - .unwrap(); - assert!(metric["elapsed_nanoseconds"].as_u64().unwrap() > 0); - assert!(metric["cpu_nanoseconds"].as_u64().unwrap() > 0); - assert!(metric["output_bytes"].as_u64().unwrap() > 0); - } -} - -/// Scenario: KLL demo processes two sources through create, merge, and estimate workers. -/// Guarantees: Four distinct processes preserve ASAPv1 sketches and emit bounded-error quantiles. -#[test] -fn four_process_pipeline_preserves_asapv1_data_and_estimates_quantiles() { - let dir = unique_dir(); - fs::create_dir(&dir).expect("create E2E directory"); - let manifest_path = dir.join("result.json"); - let output = Command::new(env!("CARGO_BIN_EXE_asap-otap-demo")) - .args([ - "--output-dir", - dir.to_str().unwrap(), - "--result-manifest", - manifest_path.to_str().unwrap(), - "--points-per-source", - &POINTS_PER_SOURCE.to_string(), - ]) - .output() - .expect("run multi-process demo"); - assert!( - output.status.success(), - "demo failed\nstdout:\n{}\nstderr:\n{}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ); - - let manifest: Value = - serde_json::from_slice(&fs::read(&manifest_path).expect("result manifest exists")) - .expect("valid result manifest"); - assert_eq!(manifest["points_per_source"], POINTS_PER_SOURCE); - let parent_pid = manifest["parent_pid"].as_u64().expect("parent pid"); - let processors = manifest["processors"].as_array().expect("processor runs"); - assert_eq!(processors.len(), 4); - let roles = processors - .iter() - .map(|p| p["role"].as_str().unwrap()) - .collect::>(); - assert_eq!( - roles, - HashSet::from(["create_a", "create_b", "merge", "estimate"]) - ); - let pids = processors - .iter() - .map(|p| p["pid"].as_u64().unwrap()) - .collect::>(); - assert_eq!(pids.len(), 4, "each DF processor must have its own PID"); - assert!(!pids.contains(&parent_pid)); - - for name in ["sa.otlp", "sb.otlp", "merged.otlp"] { - let observations = read_observations(&dir.join(name)); - assert_eq!(observations.len(), 1, "{name} contains one sketch"); - let observation = &observations[0]; - let sketch = observation - .value - .envelope - .as_ref() - .expect("sketch envelope"); - assert!(sketch.payload.starts_with(b"ASAPv1"), "{name} uses ASAPv1"); - KLL::::deserialize_from_bytes(&sketch.payload) - .unwrap_or_else(|error| panic!("{name} contains a valid ASAPv1 KLL: {error}")); - assert_eq!( - observation - .resource_labels - .iter() - .find(|kv| kv.key == "service.name") - .map(|kv| kv.value.as_str()), - Some("checkout") - ); - } - - let estimates = read_observations(&dir.join("out.otlp")); - assert_eq!(estimates.len(), 2); - for (quantile, expected) in [("0.5", 25_000.0), ("0.99", 49_500.0)] { - let value = estimates - .iter() - .find(|o| { - o.labels - .iter() - .any(|kv| kv.key == "quantile" && kv.value == quantile) - }) - .expect("quantile output"); - assert_eq!(value.metric, "request.duration.estimate"); - assert!((value.value.float - expected).abs() <= expected * 0.05); - assert!(value - .resource_labels - .iter() - .any(|kv| kv.key == "service.name" && kv.value == "checkout")); - } - - for (trace, data_type) in [ - ("create-a.debug.log", "DataType: Summary"), - ("create-b.debug.log", "DataType: Summary"), - ("merge.debug.log", "DataType: Summary"), - ("estimate.debug.log", "DataType: Gauge"), - ] { - let trace = fs::read_to_string(dir.join(trace)).expect("official debug trace"); - assert!(trace.contains(data_type)); - assert!(trace.contains("service.name: checkout")); - } -} diff --git a/benchmarks/Dockerfile b/benchmarks/Dockerfile deleted file mode 100644 index cf32c8a..0000000 --- a/benchmarks/Dockerfile +++ /dev/null @@ -1,9 +0,0 @@ -FROM ubuntu:24.04 -RUN apt-get update && apt-get install -y --no-install-recommends python3 util-linux && rm -rf /var/lib/apt/lists/* - -COPY asap-otap-demo /usr/local/bin/asap-otap-demo -COPY run-pipeline-component.sh /usr/local/bin/run-pipeline-component -COPY summarize-stages.py /usr/local/benchmarks/summarize-stages.py - -ENV ASAP_DEMO_BIN=/usr/local/bin/asap-otap-demo -ENTRYPOINT ["/usr/local/bin/run-pipeline-component"] diff --git a/benchmarks/PROFILING.md b/benchmarks/PROFILING.md index 48ccf51..5938adc 100644 --- a/benchmarks/PROFILING.md +++ b/benchmarks/PROFILING.md @@ -30,7 +30,7 @@ used by other threads cannot inflate a computation scope. Categories are: | --- | --- | | `computation` | Numeric buffering, Exact sorting/merge/quantile selection; KLL update/merge/quantile query | | `pdata_codec` | ASAP observation/envelope construction and conversion, including upstream OTAP/Arrow/protobuf conversion invoked by those adapters | -| `sketch_codec` | Sketch serialization/deserialization using the production KLL wrapper/Msgpack codec | +| `sketch_codec` | Sketch serialization/deserialization using the production KLL wrapper/ASAPv1 codec | | `processor_bookkeeping` | Remaining synchronous processor work: benchmark labels/validation, window maps, object destruction, pass-through output allocation | | `outside_scopes` | Process CPU minus the above: native OTLP receiver/exporter, runtime, channels, native transport conversion, allocator work outside the processor, control listener and probe overhead | diff --git a/benchmarks/README.md b/benchmarks/README.md index 1b56df3..3059375 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,162 +1,16 @@ # ASAP sketch performance benchmarks -**The default nightly evaluation now uses persistent OTLP/HTTP streaming across -seven containers, real data-interface RX/TX counters, bounded backpressure, and -backend-validated throughput. See [STREAMING.md](STREAMING.md) for the current -setup, bandwidth profiles, commands, and artifacts.** - -The remainder of this document describes the historical file-backed batch -baseline. Run it with `./benchmarks/run-file-nightly.sh`; its throughput figures -are not directly comparable to the streaming evaluation. - -The nightly benchmark runs with OTAP's official Python `pipeline_perf_test` -orchestrator. It deploys raw, exact-quantile, and ASAP KLL scenarios as -identically configured Docker containers, observes each cgroup with the -upstream `docker_component` monitor, and emits upstream process reports. -Criterion also runs isolated KLL stage measurements in the nightly workflow. - -The logical topology is `traffic generator → ASAP sketch SUT → simulated -backend`. The generator uses reproducible SplitMix64 durations with a long tail -and OpenTelemetry `http.server.request.duration` semantic attributes. The local microbenchmark has -three SUT workloads: transmitting every raw value, collecting and sorting every -raw value for exact p50/p99, and creating then merging two self-describing -ASAPv1 KLL shards for approximate p50/p99. The official-orchestrated nightly -runs the same three workloads with real OTAP `RuntimePipeline` workers. The -backend checks raw cardinality, exact sort results, and KLL's 5% error bound. - -Run the official-style nightly suite locally with Docker and Python 3.13 (the -runner fetches the pinned OTAP orchestrator revision on first use): +The nightly evaluation uses persistent OTLP/HTTP streaming across seven +containers, real data-interface RX/TX counters, bounded backpressure, and +backend-validated throughput. See [STREAMING.md](STREAMING.md) for the setup, +bandwidth profiles, commands, and artifacts, [CAPACITY_DESIGN.md](CAPACITY_DESIGN.md) +for the capacity definition, and [PROFILING.md](PROFILING.md) for CPU +profiling. ```sh -./benchmarks/run-file-nightly.sh -``` - -## Compared topologies - -All scenarios generate the same reproducible semantic-convention-shaped -`http.server.request.duration` values. Nightly gives each of two sources one -native `OtapPdata` batch of 262,144 observations. The local Criterion workload -uses 4,096-observation batches. Both timed boundaries end after the simulated -backend decodes the scenario's output pdata. - -```text -OTAP raw pass-through - -source A -> pass-through A --\ - -> pass-through merge -> pass-through final -> backend -source B -> pass-through B --/ -``` - -This is the no-computation control. It measures the cost of retaining and -transmitting every raw value; it intentionally has a larger output than the two -quantile scenarios. - -```text -OTAP exact quantile - -source A -> sort A -> encoded sorted run A --\ - -> merge sorted runs -source B -> sort B -> encoded sorted run B --/ | - v - exact p50/p99 -> backend -``` - -The exact path retains all N values until sorting completes. Its result must -equal the reference p50/p99 exactly. - -```text -ASAP KLL quantile - -source A -> KLL create A -> ASAPv1 KLL A --\ - -> KLL merge -> merged ASAPv1 KLL -source B -> KLL create B -> ASAPv1 KLL B --/ | - v - KLL p50/p99 -> backend +./benchmarks/run-nightly.sh ``` -The KLL path uses two deterministic `k=400` creators. Merge operates on the -self-describing ASAPv1 representation supplied by `asap_sketchlib`; its p50/p99 -must remain within 5% of the exact result. - -In nightly, every topology has two generator processes, two branch workers, one merge worker, and one -final/estimate worker. Every worker is a separate OS process containing a real -OTAP `RuntimePipeline`; the generators run outside OTAP. Two available CPUs -form a configurable traffic-generator pool. Branch A, branch B, merge, and -estimate each receive one dedicated CPU with `taskset`/Docker `--cpuset-cpus`. -The validating backend receives all remaining host CPUs because it is benchmark -infrastructure. No component has a Docker memory limit; actual RSS is reported. -Each process writes elapsed time, CPU use, peak RSS, output bytes, and OTLP -file serialization time. The upstream Docker monitor includes the entire process -tree in container CPU and RSS measurements. All scenarios use the same input count and -the same warm-up and observation intervals. - -The local Criterion microbenchmark records throughput and latency for three pipelines with identical -native pdata ingress and backend encode/decode boundaries: a no-computation OTAP -raw pass-through, an OTAP exact-quantile processor that sorts all raw values, -and the two-creator ASAP KLL path. Both quantile pipelines emit the same p50/p99 -Gauge shape. Before timing, the backend validates values, metric labels, -resource labels, exact p50/p99 results, and the KLL 5% accuracy bound. - -The nightly workflow compiles first, outside measurement. During its observation -window, OTAP's orchestrator samples CPU and RSS from the Docker cgroup. Each -component writes completed input count, elapsed time, and signals/s below -`benchmark-results/{raw,exact,kll}/throughput.env`. The rate uses only completed -batches; `run_wall_nanoseconds` records the full run duration. An interrupted -partial batch does not dilute the rate. The orchestrator writes three process -reports under `benchmark-results/`. They are uploaded by CI. -The per-role `stages.json` and `stages.jsonl` files report generator, create, -merge, and estimate resources separately. `kll-isolated-stages.txt` contains -Criterion throughput for KLL creation, merge, p50/p99 estimation, and ASAPv1 -serialization. - -## Nightly configuration - -| Setting | Current value | -| --- | --- | -| Scenarios | raw, exact, KLL | -| Sources | 2 processes, 262,144 observations each | -| Generator | 2 threads per source; SplitMix64 long-tail durations, seed `0x9e3779b97f4a7c15` | -| KLL | `k=400`, Msgpack ASAPv1, p50/p99 | -| CPU placement | configurable generator pool; one dedicated CPU for each downstream component | -| Observation | 5 s warm-up, 30 s monitored window | -| Debug output | disabled in nightly measurements; enabled in the interactive demo | -| Output | config.json, throughput.env, stages.json, stages.jsonl, process reports, Criterion text | - -Stage process rates include OTAP startup and file IO. `serialization_seconds` -measures final OTLP file conversion and write. Criterion's `serialize_asapv1` -result isolates sketch framing. Stages run sequentially after generation, so -these rates describe stage work rather than simultaneous streaming capacity. - -## Input scale sweep - -The nightly runner also repeats each raw, exact, and KLL pipeline twice at -32,768, 65,536, 262,144, and 524,288 observations **per source**. It uses the -same semantic values, generator threads, CPU placement, and disabled debug -output at every size. Run it locally after building `asap-otap-demo`: - -```sh -python3 benchmarks/run-scale-sweep.py -``` - -Choose sizes and repetitions with `--points 65536 262144 524288` and -`--repetitions 3`. Results appear under `benchmark-results/scale-sweep/`: - -- `throughput.svg` plots raw, exact, and KLL signals/s plus KLL speedup ratios. -- `{raw,exact,kll}-resources.svg` plots CPU seconds and peak RSS for each of the - two traffic generators and four processor roles. -- `summary.csv` and `resources.csv` contain median measurements; `runs.json` - retains every individual run, and `config.json` records CPU sets and sizes. - -Each scale point measures one complete batch, including traffic generation, -OTAP startup, file boundaries, and backend decoding. Reference sorting and -quantile validation run after the timed pipeline; raw skips reference sorting. -`runs.json` also records process wall time and validation time. The charts describe -this file-backed demo's scaling. Peak RSS is per process and should not be -summed as a simultaneous container peak because merge and estimate run after -the creators finish. -The `branch_a` and `branch_b` roles pass values through in raw, sort values in -exact, and create KLL sketches in KLL. - ## Viewing nightly results today There is not yet a published ASAP dashboard. Results are available as GitHub @@ -165,22 +19,12 @@ Actions artifacts: 1. Open the repository's **Actions** tab and select **ASAP sketch benchmark**. 2. Open a successful scheduled, manual, or pull-request run. 3. Download the `asap-sketch-benchmark-` artifact from **Artifacts**. -4. Compare `raw/throughput.env`, `exact/throughput.env`, and - `kll/throughput.env` for completed input throughput and serialized boundary - bytes. Read `otap-raw-resources.md`, `otap-exact-quantile-resources.md`, and - `asap-kll-resources.md` for the upstream orchestrator's CPU and memory - summaries. +4. Read `streaming/summary.json`, `streaming/capacity.json`, and the + `throughput.svg` / `resources.svg` charts. -Artifacts expire after 30 days. Pull-request runs are useful for reviewing a -specific change; the scheduled run at 02:30 UTC is the consistent source for -day-over-day comparisons after this workflow is merged to the default branch. -GitHub-hosted runners are shared infrastructure, so small differences between -runs should not be treated as regressions without repeated measurements. - -The serialized byte rates describe the current file-backed processor boundary. -They are not network-bandwidth measurements: the processors exchange files -inside one container, and Docker network RX/TX is therefore expected to remain -near zero. +Artifacts expire after 30 days. GitHub-hosted runners are shared +infrastructure, so small differences between runs should not be treated as +regressions without repeated measurements. ## Future dashboard publication @@ -208,27 +52,6 @@ scheduled runs establish normal GitHub-hosted-runner variance. If stable cross-project numbers are required later, the same summary format can be fed by an isolated runner or an upstream OTAP shared-server contribution. -The local comparison answers two separate questions. Raw pass-through measures the -cost of retaining and transmitting all observations, so its output cardinality -is intentionally much larger. Exact sort and KLL both emit two Gauge values and -are directly comparable for this fixed batch workload. At these input sizes, -OTAP pdata decode and harness costs can dominate both quantile implementations; -larger streams and bounded-memory tests are needed to expose their different -scaling behavior. Peak RSS is for the whole Criterion process, including input, -OTAP/Arrow dependencies, and the harness—not just the algorithm state. - -This repository consumes the official framework rather than copying its Python -implementation. It pins the same OTAP revision as the Rust dependencies. It does -not yet run on OpenTelemetry's shared bare-metal host. The file-backed OTLP -boundary makes CPU, RSS, throughput, and artifact bytes comparable, but artifact -bytes are not a network-bandwidth measurement. A future upstream contribution -should replace it with the standard network traffic generator and backend nodes. - -For performance debugging, bench profiles retain symbols (`strip = "none"`). -Criterion identifies regressions locally. A follow-up can use `samply`, or the -OTAP admin server's `/api/v1/debug/pprof/profile` and `/heap` endpoints after the -processor is hosted by the upstream engine binary. - ## Upstream references - [Nightly dashboard](https://open-telemetry.github.io/otel-arrow/benchmarks/nightly/) @@ -236,6 +59,5 @@ processor is hosted by the upstream engine binary. - [Comparison dashboard](https://open-telemetry.github.io/otel-arrow/compare/) and [comparison suites](https://github.com/open-telemetry/otel-arrow/tree/main/tools/pipeline_perf_test/test_suites/comparison_dashboard) - [Traffic generator](https://github.com/open-telemetry/otel-arrow/blob/main/rust/otap-dataflow/crates/dev-nodes/src/receivers/traffic_generator/README.md) -- [Admin debug API](https://github.com/open-telemetry/otel-arrow/blob/main/rust/otap-dataflow/crates/admin/README.md#debug), - [Samply](https://github.com/mstange/samply), and - [Criterion](https://github.com/bheisler/criterion.rs) +- [Admin debug API](https://github.com/open-telemetry/otel-arrow/blob/main/rust/otap-dataflow/crates/admin/README.md#debug) and + [Samply](https://github.com/mstange/samply) diff --git a/benchmarks/STREAMING.md b/benchmarks/STREAMING.md index 0abd87f..40adba3 100644 --- a/benchmarks/STREAMING.md +++ b/benchmarks/STREAMING.md @@ -1,8 +1,6 @@ # Sustained OTLP network benchmark -`./benchmarks/run-nightly.sh` now runs persistent streaming, not repeated file-backed demos. -The historical batch experiment is still available as `./benchmarks/run-file-nightly.sh`; -its numbers must not be mixed with this benchmark's results. +`./benchmarks/run-nightly.sh` runs persistent streaming. For exclusive CPU cost attribution and worker call-stack sampling, see [PROFILING.md](PROFILING.md). CPU profiling uses drained accounting boundaries @@ -41,7 +39,7 @@ wall-clock rotation, late-event handling, or window-watermark policies. - Exact retains and sorts each branch's window, sends sorted runs in bounded batches, merges the two runs, then returns exact p50/p99. - KLL updates a `k=400` sketch incrementally per input batch, sends one ASAPv1 - Msgpack sketch per source/window, merges them, then returns p50/p99. + sketch per source/window, merges them, then returns p50/p99. - Backend validates every quantile result. Exact must match; KLL tolerance is `max(abs(reference) * 0.05, 0.001)`. This is value error, not rank error. - Top-k scenarios use 64-bit keys drawn from Zipf(s=1.1) over @@ -136,7 +134,7 @@ small sampling skew; each rate uses its own elapsed interval. ## Running and artifacts -Full suite (builds release binary and Docker image; also runs isolated KLL Criterion): +Full suite (builds release binary and Docker image): ```sh ./benchmarks/run-nightly.sh @@ -178,5 +176,4 @@ makes estimate also answer over the last N completed windows; the default of CI runs the full 189-run matrix, checks actual network traffic, completed windows, zero loss after drain, and resource artifacts. It uploads results even on failure. -The old PR's 2.43× was measured with the file-backed batch harness and is not a -result of this streaming implementation. New speedups must be measured, not assumed. +New speedups must be measured, not assumed. diff --git a/benchmarks/nightly/asap-sketch.yaml b/benchmarks/nightly/asap-sketch.yaml deleted file mode 100644 index 13bb4fc..0000000 --- a/benchmarks/nightly/asap-sketch.yaml +++ /dev/null @@ -1,65 +0,0 @@ -name: Nightly - aligned OTAP quantile pipelines - -# Every scenario uses the same image, two 262,144-signal sources, four -# RuntimePipeline worker processes, 30-second observation window, cgroup limits, -# file-backed OTLP boundary, and correctness-checking backend. -components: - otap-raw: - deployment: - docker: - image: asap-quantile-benchmark:local - network: asap-benchmark - volumes: ['./benchmark-results:/results'] - environment: - - ASAP_BENCH_SCENARIO=raw - - ASAP_BENCH_POINTS_PER_SOURCE=262144 - - ASAP_BENCH_TRAFFIC=semantic - - ASAP_BENCH_GENERATOR_THREADS=2 - - ASAP_BENCH_DISABLE_DEBUG=1 - - ASAP_BENCH_RESULT_DIR=/results/raw - monitoring: - docker_component: { allocated_cores: 4 } - otap-exact-quantile: - deployment: - docker: - image: asap-quantile-benchmark:local - network: asap-benchmark - volumes: ['./benchmark-results:/results'] - environment: - - ASAP_BENCH_SCENARIO=exact - - ASAP_BENCH_POINTS_PER_SOURCE=262144 - - ASAP_BENCH_TRAFFIC=semantic - - ASAP_BENCH_GENERATOR_THREADS=2 - - ASAP_BENCH_DISABLE_DEBUG=1 - - ASAP_BENCH_RESULT_DIR=/results/exact - monitoring: - docker_component: { allocated_cores: 4 } - asap-kll: - deployment: - docker: - image: asap-quantile-benchmark:local - network: asap-benchmark - volumes: ['./benchmark-results:/results'] - environment: - - ASAP_BENCH_SCENARIO=kll - - ASAP_BENCH_POINTS_PER_SOURCE=262144 - - ASAP_BENCH_TRAFFIC=semantic - - ASAP_BENCH_GENERATOR_THREADS=2 - - ASAP_BENCH_DISABLE_DEBUG=1 - - ASAP_BENCH_RESULT_DIR=/results/kll - monitoring: - docker_component: { allocated_cores: 4 } - -tests: - - name: OTAP-RAW - from_template: - path: benchmarks/nightly/scenario-steps.yaml.j2 - variables: { component: otap-raw, report_name: otap-raw-resources } - - name: OTAP-EXACT-QUANTILE - from_template: - path: benchmarks/nightly/scenario-steps.yaml.j2 - variables: { component: otap-exact-quantile, report_name: otap-exact-quantile-resources } - - name: ASAP-KLL - from_template: - path: benchmarks/nightly/scenario-steps.yaml.j2 - variables: { component: asap-kll, report_name: asap-kll-resources } diff --git a/benchmarks/nightly/scenario-steps.yaml.j2 b/benchmarks/nightly/scenario-steps.yaml.j2 deleted file mode 100644 index d2fb631..0000000 --- a/benchmarks/nightly/scenario-steps.yaml.j2 +++ /dev/null @@ -1,47 +0,0 @@ -steps: - - name: Deploy {{component}} - action: - component_action: { phase: deploy, target: "{{component}}" } - - name: Start monitoring {{component}} - action: - component_action: { phase: start_monitoring, target: "{{component}}" } - - name: Warm up - action: - wait: { delay_seconds: 5 } - - name: Observe load - action: - wait: { delay_seconds: 30 } - hooks: - run: - pre: - - record_event: { name: "{{component}}_observation_start" } - post: - - record_event: { name: "{{component}}_observation_stop" } - - name: Stop monitoring {{component}} - action: - component_action: { phase: stop_monitoring, target: "{{component}}" } - - name: Destroy {{component}} - action: - component_action: { phase: destroy, target: "{{component}}" } - - name: Write {{component}} report - action: - no_op: {} - hooks: - run: - post: - - process_report: - name: "{{component}} container resources" - components: ["{{component}}"] - between_events: - start: { name: "{{component}}_observation_start" } - end: { name: "{{component}}_observation_stop" } - include_sections: { component_summary: true, component_detail: true } - output: - - format: { template: {} } - destination: { console: {} } - - format: { template: {} } - destination: - file: - directory: benchmark-results - name: "{{report_name}}" - extension: .md diff --git a/benchmarks/run-file-nightly.sh b/benchmarks/run-file-nightly.sh deleted file mode 100755 index e98747d..0000000 --- a/benchmarks/run-file-nightly.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) -otap_rev=${OTAP_BENCH_REV:-3e85c3460361446ebfce99e9f35fffd2dd5ab740} -framework_dir=${OTAP_PIPELINE_PERF_DIR:-"$repo_root/.cache/otel-arrow-$otap_rev"} -python_deps=${OTAP_PIPELINE_PERF_PYTHON_DEPS:-"$repo_root/.cache/pipeline-perf-python"} -python_bin=${PYTHON:-python3} -result_root="$repo_root/benchmark-results" - -if ! "$python_bin" -c 'import sys; raise SystemExit(sys.version_info < (3, 13))'; then - echo "OTAP's pinned benchmark dependencies require Python 3.13 or newer." >&2 - echo "Set PYTHON to a compatible interpreter." >&2 - exit 2 -fi - -# Docker creates a missing bind-mount source as root. Create it as the caller so -# the host-side orchestrator can write process reports alongside container data. -mkdir -p "$result_root" - -cargo build --manifest-path "$repo_root/asap-precompute-rs/Cargo.toml" \ - --release --features otap-engine --bin asap-otap-demo -cargo bench --manifest-path "$repo_root/asap-precompute-rs/Cargo.toml" \ - --features otap-engine --bench asap_sketch_pipeline -- \ - kll_isolated_stages --sample-size 10 --warm-up-time 1 --measurement-time 1 \ - | tee "$result_root/kll-isolated-stages.txt" -docker_context="$repo_root/.cache/asap-benchmark-image" -mkdir -p "$docker_context" -cp "$repo_root/asap-precompute-rs/target/release/asap-otap-demo" "$docker_context/asap-otap-demo" -strip --strip-debug "$docker_context/asap-otap-demo" -cp "$repo_root/benchmarks/run-pipeline-component.sh" "$repo_root/benchmarks/summarize-stages.py" "$docker_context/" -docker build -f "$repo_root/benchmarks/Dockerfile" \ - -t asap-quantile-benchmark:local "$docker_context" - -if [[ ! -f "$framework_dir/tools/pipeline_perf_test/orchestrator/run_orchestrator.py" ]]; then - mkdir -p "$(dirname "$framework_dir")" - git clone --filter=blob:none --no-checkout \ - https://github.com/open-telemetry/otel-arrow.git "$framework_dir" - git -C "$framework_dir" checkout "$otap_rev" -fi - -if ! PYTHONPATH="$python_deps${PYTHONPATH:+:$PYTHONPATH}" "$python_bin" -c \ - 'import docker, duckdb, pandas, pyarrow, pydantic, yaml' 2>/dev/null; then - "$python_bin" -m pip install --quiet --target "$python_deps" \ - -r "$framework_dir/tools/pipeline_perf_test/orchestrator/requirements.txt" -fi - -cd "$repo_root" -PYTHONPATH="$python_deps${PYTHONPATH:+:$PYTHONPATH}" \ -"$python_bin" "$framework_dir/tools/pipeline_perf_test/orchestrator/run_orchestrator.py" \ - --config "$repo_root/benchmarks/nightly/asap-sketch.yaml" - -"$python_bin" "$repo_root/benchmarks/run-scale-sweep.py" \ - --binary "$docker_context/asap-otap-demo" \ - --output "$result_root/scale-sweep" diff --git a/benchmarks/run-nightly.sh b/benchmarks/run-nightly.sh index cff4d32..314fcb2 100755 --- a/benchmarks/run-nightly.sh +++ b/benchmarks/run-nightly.sh @@ -8,10 +8,6 @@ read -r -a docker_command <<< "${DOCKER_COMMAND:-docker}" mkdir -p benchmark-results cargo build --manifest-path asap-precompute-rs/Cargo.toml \ --release --features stream-benchmark --bin asap-stream-bench -cargo bench --manifest-path asap-precompute-rs/Cargo.toml \ - --features otap-engine --bench asap_sketch_pipeline -- \ - kll_isolated_stages --sample-size 10 --warm-up-time 1 --measurement-time 1 \ - | tee benchmark-results/kll-isolated-stages.txt context=$(mktemp -d) trap 'rm -rf "$context"' EXIT cp "$cargo_target_dir/release/asap-stream-bench" "$context/" diff --git a/benchmarks/run-pipeline-component.sh b/benchmarks/run-pipeline-component.sh deleted file mode 100755 index 5a979a7..0000000 --- a/benchmarks/run-pipeline-component.sh +++ /dev/null @@ -1,100 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) -result_dir=${ASAP_BENCH_RESULT_DIR:-"$repo_root/benchmark-results/kll-runtime-pipeline"} -points_per_source=${ASAP_BENCH_POINTS_PER_SOURCE:-65536} -scenario=${ASAP_BENCH_SCENARIO:-kll} -generator_threads=${ASAP_BENCH_GENERATOR_THREADS:-2} -traffic=${ASAP_BENCH_TRAFFIC:-semantic} -demo_bin=${ASAP_DEMO_BIN:-"$repo_root/asap-precompute-rs/target/release/asap-otap-demo"} - -test -x "$demo_bin" -mkdir -p "$result_dir" -chmod 0777 "$result_dir" -stage_log="$result_dir/stages.jsonl" -: > "$stage_log" -if [[ -z "${ASAP_GENERATOR_CORES:-}" || -z "${ASAP_PROCESSOR_CORES:-}" ]]; then - core_sets=$(python3 -c 'import os,sys; cores=sorted(os.sched_getaffinity(0))[:4]; sys.exit("benchmark needs at least two available CPUs") if len(cores)<2 else None; split=max(1,len(cores)//2); print(",".join(map(str,cores[:split]))); print(",".join(map(str,cores[split:])))') - export ASAP_GENERATOR_CORES=${ASAP_GENERATOR_CORES:-$(sed -n '1p' <<< "$core_sets")} - export ASAP_PROCESSOR_CORES=${ASAP_PROCESSOR_CORES:-$(sed -n '2p' <<< "$core_sets")} -fi -python3 -c 'import json,sys; json.dump(dict(scenario=sys.argv[1], points_per_source=int(sys.argv[2]), traffic=sys.argv[3], generator_threads=int(sys.argv[4]), generator_cores=sys.argv[5], processor_cores=sys.argv[6], debug_disabled=sys.argv[7]=="1"), open(sys.argv[8], "w"), indent=2)' \ - "$scenario" "$points_per_source" "$traffic" "$generator_threads" "$ASAP_GENERATOR_CORES" "$ASAP_PROCESSOR_CORES" "${ASAP_BENCH_DISABLE_DEBUG:-0}" "$result_dir/config.json" - -iteration=0 -completed_iterations=0 -signals=0 -transport_bytes=0 -started_ns=$(date +%s%N) -last_completed_ns=$started_ns -stop_requested=0 -current_child= - -write_summary() { - local finished_ns elapsed_ns - finished_ns=$(date +%s%N) - elapsed_ns=$((last_completed_ns - started_ns)) - { - echo "iterations=$completed_iterations" - echo "input_signals=$signals" - echo "serialized_transport_bytes=$transport_bytes" - echo "elapsed_nanoseconds=$elapsed_ns" - echo "run_wall_nanoseconds=$((finished_ns - started_ns))" - if (( elapsed_ns > 0 )); then - echo "signals_per_second=$((signals * 1000000000 / elapsed_ns))" - echo "serialized_bytes_per_second=$((transport_bytes * 1000000000 / elapsed_ns))" - else - echo "signals_per_second=0" - echo "serialized_bytes_per_second=0" - fi - } > "$result_dir/throughput.env" - python3 "$repo_root/benchmarks/summarize-stages.py" "$stage_log" "$result_dir/stages.json" -} - -request_stop() { - stop_requested=1 - if [[ -n "$current_child" ]]; then - kill -TERM "$current_child" 2>/dev/null || true - fi -} -trap request_stop TERM INT -trap write_summary EXIT - -while (( stop_requested == 0 )); do - iteration=$((iteration + 1)) - run_dir="$result_dir/current" - mkdir -p "$run_dir" - chmod 0777 "$run_dir" - if [[ "$iteration" == 1 && "${ASAP_BENCH_DISABLE_DEBUG:-0}" == 1 ]]; then - python3 -c 'from pathlib import Path; import sys; [path.unlink() for path in Path(sys.argv[1]).glob("*.debug.log")]' "$run_dir" - fi - "$demo_bin" \ - --scenario "$scenario" \ - --output-dir "$run_dir" \ - --result-manifest "$run_dir/result.json" \ - --points-per-source "$points_per_source" \ - --traffic "$traffic" \ - --generator-threads "$generator_threads" & - current_child=$! - if ! wait "$current_child"; then - if (( stop_requested != 0 )); then - break - fi - exit 1 - fi - current_child= - for stage in a b sa sb merged out; do - test -s "$run_dir/$stage.metrics.json" - python3 -c 'import json,sys; data=json.load(open(sys.argv[1])); data.update(stage=sys.argv[2], iteration=int(sys.argv[3]), scenario=sys.argv[4], points_per_source=int(sys.argv[5])); print(json.dumps(data))' \ - "$run_dir/$stage.metrics.json" "$stage" "$iteration" "$scenario" "$points_per_source" >> "$stage_log" - done - completed_iterations=$((completed_iterations + 1)) - signals=$((signals + points_per_source * 2)) - for artifact in a.otlp b.otlp sa.otlp sb.otlp merged.otlp out.otlp; do - artifact_bytes=$(wc -c < "$run_dir/$artifact") - transport_bytes=$((transport_bytes + artifact_bytes)) - done - last_completed_ns=$(date +%s%N) - write_summary -done diff --git a/benchmarks/run-scale-sweep.py b/benchmarks/run-scale-sweep.py deleted file mode 100644 index 486b632..0000000 --- a/benchmarks/run-scale-sweep.py +++ /dev/null @@ -1,272 +0,0 @@ -#!/usr/bin/env python3 -"""Compare raw, exact, and KLL at increasing traffic volumes.""" - -import argparse -import csv -import datetime -import json -import os -import statistics -import subprocess -import time -from pathlib import Path - - -SCENARIOS = ("raw", "exact", "kll") -COLORS = {"raw": "#64748b", "exact": "#dc2626", "kll": "#2563eb"} -ROLES = ("traffic_a", "traffic_b", "branch_a", "branch_b", "merge", "estimate") -ROLE_COLORS = {"traffic_a": "#0891b2", "traffic_b": "#06b6d4", "branch_a": "#2563eb", - "branch_b": "#818cf8", "merge": "#dc2626", "estimate": "#9333ea"} - - -def cpu_sets(): - cores = sorted(os.sched_getaffinity(0))[:4] - if len(cores) < 2: - raise RuntimeError("the scale sweep needs at least two available CPUs") - split = max(1, len(cores) // 2) - return ",".join(map(str, cores[:split])), ",".join(map(str, cores[split:])) - - -def measure(binary, output, points, scenario, repeat, generator_cores, processor_cores): - run_dir = output / "runs" / str(points) / scenario / str(repeat) - run_dir.mkdir(parents=True, exist_ok=True) - environment = os.environ.copy() - environment.update({ - "ASAP_GENERATOR_CORES": generator_cores, - "ASAP_PROCESSOR_CORES": processor_cores, - "ASAP_BENCH_DISABLE_DEBUG": "1", - }) - command = [str(binary), "--scenario", scenario, "--traffic", "semantic", - "--generator-threads", "2", "--points-per-source", str(points), - "--output-dir", str(run_dir), "--result-manifest", str(run_dir / "result.json")] - start = time.monotonic_ns() - completed = subprocess.run(command, env=environment, capture_output=True, text=True, - timeout=300, check=False) - process_wall_seconds = (time.monotonic_ns() - start) / 1e9 - (run_dir / "stdout.txt").write_text(completed.stdout) - (run_dir / "stderr.txt").write_text(completed.stderr) - if completed.returncode: - raise RuntimeError(f"{scenario}, {points} per source, repetition {repeat}: " - f"exit {completed.returncode}; {completed.stderr[-2000:]}") - manifest = json.loads((run_dir / "result.json").read_text()) - elapsed = manifest["pipeline_elapsed_nanoseconds"] / 1e9 - if elapsed <= 0 or elapsed > process_wall_seconds: - raise RuntimeError(f"invalid pipeline duration: {elapsed} seconds") - stages = {} - for name, role in (("a", "traffic_a"), ("b", "traffic_b"), ("sa", "branch_a"), - ("sb", "branch_b"), ("merged", "merge"), ("out", "estimate")): - stages[role] = json.loads((run_dir / f"{name}.metrics.json").read_text()) - # Boundary sizes remain in the stage metrics; keep the sweep artifacts compact. - for artifact in run_dir.glob("*.otlp"): - artifact.unlink() - return { - "points_per_source": points, - "input_signals": points * 2, - "scenario": scenario, - "repeat": repeat, - "elapsed_seconds": elapsed, - "process_wall_seconds": process_wall_seconds, - "validation_seconds": manifest["validation_elapsed_nanoseconds"] / 1e9, - "signals_per_second": points * 2 / elapsed, - "stages": stages, - } - - -def aggregate(runs): - groups = {} - for run in runs: - groups.setdefault((run["points_per_source"], run["scenario"]), []).append(run) - rows = [] - for (points, scenario), samples in sorted(groups.items()): - rates = [sample["signals_per_second"] for sample in samples] - row = { - "points_per_source": points, - "input_signals": points * 2, - "scenario": scenario, - "repetitions": len(samples), - "median_signals_per_second": statistics.median(rates), - "min_signals_per_second": min(rates), - "max_signals_per_second": max(rates), - "median_elapsed_seconds": statistics.median(sample["elapsed_seconds"] for sample in samples), - } - rows.append(row) - by_points = {} - for row in rows: - by_points.setdefault(row["points_per_source"], {})[row["scenario"]] = row - for row in rows: - series = by_points[row["points_per_source"]] - if row["scenario"] == "kll": - row["kll_over_raw"] = row["median_signals_per_second"] / series["raw"]["median_signals_per_second"] - row["kll_over_exact"] = row["median_signals_per_second"] / series["exact"]["median_signals_per_second"] - else: - row["kll_over_raw"] = "" - row["kll_over_exact"] = "" - return rows - - -def aggregate_resources(runs): - groups = {} - for run in runs: - for role, metrics in run["stages"].items(): - groups.setdefault((run["points_per_source"], run["scenario"], role), []).append(metrics) - rows = [] - for (points, scenario, role), samples in sorted(groups.items()): - rows.append({ - "points_per_source": points, - "input_signals": points * 2, - "scenario": scenario, - "component": role, - "repetitions": len(samples), - "median_cpu_seconds": statistics.median(sample["cpu_nanoseconds"] / 1e9 for sample in samples), - "median_peak_rss_mib": statistics.median(sample["peak_rss_kib"] / 1024 for sample in samples), - "median_elapsed_seconds": statistics.median(sample["elapsed_nanoseconds"] / 1e9 for sample in samples), - "median_serialization_seconds": statistics.median(sample["serialization_nanoseconds"] / 1e9 for sample in samples), - }) - return rows - - -def draw_resources(rows, scenario, path): - selected = [row for row in rows if row["scenario"] == scenario] - points = sorted({row["points_per_source"] for row in selected}) - by_key = {(row["points_per_source"], row["component"]): row for row in selected} - width, height = 980, 690 - left, right = 90, 920 - top1, bottom1 = 80, 325 - top2, bottom2 = 405, 625 - max_cpu = max(row["median_cpu_seconds"] for row in selected) * 1.12 - max_rss = max(row["median_peak_rss_mib"] for row in selected) * 1.12 - x_at = lambda index: left + (right - left) * index / max(1, len(points) - 1) - svg = [f'', - '', - '', - f'{scenario.upper()} resource usage by component', - 'Median of repeated full pipeline runs; CPU seconds and process peak RSS'] - for top, bottom, maximum, unit in ((top1, bottom1, max_cpu, "CPU seconds"), - (top2, bottom2, max_rss, "Peak RSS MiB")): - svg.append(f'') - svg.append(f'') - svg.append(f'{unit}') - for tick in range(5): - y = bottom - (bottom - top) * tick / 4 - svg.append(f'') - svg.append(f'{maximum*tick/4:.1f}') - for role in ROLES: - color = ROLE_COLORS[role] - for top, bottom, maximum, field in ((top1, bottom1, max_cpu, "median_cpu_seconds"), - (top2, bottom2, max_rss, "median_peak_rss_mib")): - coords = [(x_at(i), bottom - by_key[(p, role)][field] / maximum * (bottom - top)) - for i, p in enumerate(points)] - svg.append(f'') - for x, y in coords: - svg.append(f'') - for i, role in enumerate(ROLES): - x = left + i * 138 - svg.append(f'') - svg.append(f'{role}') - for i, points_per_source in enumerate(points): - svg.append(f'{points_per_source*2:,}') - svg.append(f'Total input signals per run') - svg.append('') - path.write_text("\n".join(svg)) - - -def draw_svg(rows, path): - points = sorted({row["points_per_source"] for row in rows}) - by_key = {(row["points_per_source"], row["scenario"]): row for row in rows} - max_rate = max(row["median_signals_per_second"] for row in rows) * 1.12 - max_ratio = max(max(by_key[(p, "kll")]["kll_over_raw"], by_key[(p, "kll")]["kll_over_exact"]) for p in points) * 1.12 - width, height = 980, 690 - left, right = 90, 920 - top1, bottom1 = 75, 345 - top2, bottom2 = 420, 625 - x_at = lambda index: left + (right - left) * index / max(1, len(points) - 1) - y_rate = lambda value: bottom1 - value / max_rate * (bottom1 - top1) - y_ratio = lambda value: bottom2 - value / max_ratio * (bottom2 - top2) - svg = [f'', - '', - '', - 'Throughput versus traffic volume', - 'Two sources; deterministic semantic data; median of repeated full pipeline runs'] - for top, bottom, maximum, unit in ((top1, bottom1, max_rate, "signals/s"), (top2, bottom2, max_ratio, "KLL speedup")): - svg.append(f'') - svg.append(f'') - svg.append(f'{unit}') - for tick in range(5): - value = maximum * tick / 4 - y = bottom - (bottom - top) * tick / 4 - svg.append(f'') - label = f"{value/1000:.0f}k" if unit == "signals/s" else f"{value:.1f}x" - svg.append(f'{label}') - for index, p in enumerate(points): - x = x_at(index) - svg.append(f'{2*p:,}') - svg.append(f'Total input signals per run') - for scenario in SCENARIOS: - coords = [(x_at(i), y_rate(by_key[(p, scenario)]["median_signals_per_second"])) for i, p in enumerate(points)] - svg.append(f'') - for x, y in coords: - svg.append(f'') - for field, color, label in (("kll_over_raw", "#0f766e", "KLL / raw"), - ("kll_over_exact", "#7c3aed", "KLL / exact")): - coords = [(x_at(i), y_ratio(by_key[(p, "kll")][field])) for i, p in enumerate(points)] - svg.append(f'') - for x, y in coords: - svg.append(f'') - for i, (label, color) in enumerate([("raw", COLORS["raw"]), ("exact", COLORS["exact"]), ("KLL", COLORS["kll"]), - ("KLL / raw", "#0f766e"), ("KLL / exact", "#7c3aed")]): - x = left + i * 160 - svg.append(f'') - svg.append(f'{label}') - svg.append('') - path.write_text("\n".join(svg)) - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--binary", type=Path, default=Path("asap-precompute-rs/target/release/asap-otap-demo")) - parser.add_argument("--output", type=Path, default=Path("benchmark-results/scale-sweep")) - parser.add_argument("--points", type=int, nargs="+", default=[32768, 65536, 262144, 524288]) - parser.add_argument("--repetitions", type=int, default=2) - args = parser.parse_args() - if args.repetitions < 1 or any(points < 1 for points in args.points): - parser.error("repetitions and points must be positive") - binary = args.binary.resolve() - if not binary.is_file(): - parser.error(f"binary does not exist: {binary}") - args.output.mkdir(parents=True, exist_ok=True) - generator_cores, processor_cores = cpu_sets() - configuration = {"points_per_source": sorted(set(args.points)), "repetitions": args.repetitions, - "generator_cores": generator_cores, "processor_cores": processor_cores, - "generator_threads_per_source": 2, "traffic": "semantic", "debug_disabled": True, - "traffic_seed": "0x9e3779b97f4a7c15", - "started_utc": datetime.datetime.now(datetime.timezone.utc).isoformat(), - "binary": str(binary)} - (args.output / "config.json").write_text(json.dumps(configuration, indent=2) + "\n") - runs = [] - for points in configuration["points_per_source"]: - for repeat in range(1, args.repetitions + 1): - for scenario in SCENARIOS: - run = measure(binary, args.output, points, scenario, repeat, generator_cores, processor_cores) - runs.append(run) - (args.output / "runs.json").write_text(json.dumps(runs, indent=2) + "\n") - print(f'{points*2:>9} signals {scenario:>5} #{repeat}: {run["signals_per_second"]:,.0f} signals/s', flush=True) - rows = aggregate(runs) - resources = aggregate_resources(runs) - (args.output / "summary.json").write_text(json.dumps(rows, indent=2) + "\n") - with (args.output / "summary.csv").open("w", newline="") as destination: - writer = csv.DictWriter(destination, fieldnames=list(rows[0])) - writer.writeheader() - writer.writerows(rows) - draw_svg(rows, args.output / "throughput.svg") - (args.output / "resources.json").write_text(json.dumps(resources, indent=2) + "\n") - with (args.output / "resources.csv").open("w", newline="") as destination: - writer = csv.DictWriter(destination, fieldnames=list(resources[0])) - writer.writeheader() - writer.writerows(resources) - for scenario in SCENARIOS: - draw_resources(resources, scenario, args.output / f"{scenario}-resources.svg") - - -if __name__ == "__main__": - main() diff --git a/benchmarks/summarize-stages.py b/benchmarks/summarize-stages.py deleted file mode 100644 index 744b077..0000000 --- a/benchmarks/summarize-stages.py +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/bin/env python3 -"""Summarize per-process metrics from completed benchmark iterations.""" - -import json -import sys -from collections import defaultdict - - -def summarize(records): - groups = defaultdict(list) - for record in records: - groups[record["stage"]].append(record) - result = {"schema_version": 2, "stages": {}} - for stage, items in sorted(groups.items()): - elapsed = sum(item["elapsed_nanoseconds"] for item in items) / 1e9 - if stage in ("a", "b", "sa", "sb"): - unit = "signals" - inputs = sum(item["points_per_source"] for item in items) - else: - unit = "sketches" if items[0]["scenario"] == "kll" else "batches" - inputs = len(items) * (2 if stage == "merged" else 1) - equivalent_signals = sum(item["points_per_source"] for item in items) - if stage in ("merged", "out"): - equivalent_signals *= 2 - result["stages"][stage] = { - "role": {"a": "traffic_a", "b": "traffic_b", "sa": "create_a", "sb": "create_b", "merged": "merge", "out": "estimate"}[stage], - "iterations": len(items), - "input_unit": unit, - "input_count": inputs, - "input_per_second": round(inputs / elapsed, 2) if elapsed else 0, - "equivalent_signals_per_second": round(equivalent_signals / elapsed, 2) if elapsed else 0, - "process_seconds": round(elapsed, 6), - "cpu_seconds": round(sum(item["cpu_nanoseconds"] for item in items) / 1e9, 6), - "peak_rss_kib": max(item["peak_rss_kib"] for item in items), - "serialization_seconds": round(sum(item["serialization_nanoseconds"] for item in items) / 1e9, 6), - "output_bytes": sum(item["output_bytes"] for item in items), - } - return result - - -if __name__ == "__main__": - with open(sys.argv[1], encoding="utf-8") as source: - records = [json.loads(line) for line in source if line.strip()] - with open(sys.argv[2], "w", encoding="utf-8") as destination: - json.dump(summarize(records), destination, indent=2) - destination.write("\n") diff --git a/demos/kll-grafana/compose.yaml b/demos/kll-grafana/compose.yaml index 52246e0..c9ec5e4 100644 --- a/demos/kll-grafana/compose.yaml +++ b/demos/kll-grafana/compose.yaml @@ -1,13 +1,11 @@ # Host networking keeps Prometheus, Grafana and the demo metrics endpoint on -# localhost only (Linux). Prometheus scrapes run-streaming.py --demo on :9465 -# and still accepts OTLP pushes from the finite asap-otap-demo export. +# localhost only (Linux). Prometheus scrapes run-streaming.py --demo on :9465. services: prometheus: image: prom/prometheus:v3.5.0 network_mode: host command: - --config.file=/etc/prometheus/prometheus.yml - - --web.enable-otlp-receiver - --web.listen-address=127.0.0.1:${PROMETHEUS_PORT:-9090} volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro diff --git a/demos/kll-grafana/grafana/dashboards/kll.json b/demos/kll-grafana/grafana/dashboards/kll.json deleted file mode 100644 index a78a6da..0000000 --- a/demos/kll-grafana/grafana/dashboards/kll.json +++ /dev/null @@ -1,81 +0,0 @@ -{ - "uid": "asap-kll", - "title": "ASAP KLL estimates", - "schemaVersion": 39, - "version": 1, - "refresh": "5s", - "time": { - "from": "now-15m", - "to": "now" - }, - "panels": [ - { - "id": 1, - "type": "stat", - "title": "Latest KLL estimates", - "datasource": { - "type": "prometheus", - "uid": "asap-prometheus" - }, - "gridPos": { - "h": 8, - "w": 24, - "x": 0, - "y": 0 - }, - "targets": [ - { - "refId": "A", - "expr": "request_duration_estimate{quantile=~\"0.5|0.99\"}", - "legendFormat": "quantile {{quantile}}", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "decimals": 3 - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - } - } - }, - { - "id": 2, - "type": "timeseries", - "title": "KLL estimates over time", - "datasource": { - "type": "prometheus", - "uid": "asap-prometheus" - }, - "gridPos": { - "h": 8, - "w": 24, - "x": 0, - "y": 8 - }, - "targets": [ - { - "refId": "A", - "expr": "request_duration_estimate{quantile=~\"0.5|0.99\"}", - "legendFormat": "quantile {{quantile}}", - "instant": false - } - ], - "fieldConfig": { - "defaults": { - "decimals": 3 - }, - "overrides": [] - }, - "options": {} - } - ] -} diff --git a/demos/kll-grafana/verify.py b/demos/kll-grafana/verify.py deleted file mode 100644 index fead757..0000000 --- a/demos/kll-grafana/verify.py +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env python3 -"""Run the real demo and verify its estimates through Prometheus and Grafana.""" -import argparse -import json -import re -import subprocess -import time -import urllib.parse -import urllib.request - - -def get(base, path): - with urllib.request.urlopen(base.rstrip('/') + path, timeout=10) as response: - return json.load(response) - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('binary', help='path to the built asap-otap-demo binary') - parser.add_argument('--prometheus', default='http://localhost:9090') - parser.add_argument('--grafana', default='http://localhost:3000') - args = parser.parse_args() - output = subprocess.check_output([ - args.binary, '--prometheus-otlp-endpoint', - args.prometheus.rstrip('/') + '/api/v1/otlp/v1/metrics', - ], text=True) - print(output, end='') - expected = {q: float(v) for q, v in re.findall( - r'result metric=request.duration.estimate quantile=(\S+) value=(\S+)', output)} - assert set(expected) == {'0.5', '0.99'}, expected - query = '/api/v1/query?' + urllib.parse.urlencode({'query': 'request_duration_estimate{quantile=~"0.5|0.99"}'}) - for base, path in [ - (args.prometheus, query), - (args.grafana, '/api/datasources/proxy/uid/asap-prometheus' + query), - ]: - for attempt in range(30): - response = get(base, path) - assert response['status'] == 'success', response - actual = {row['metric']['quantile']: float(row['value'][1]) - for row in response['data']['result']} - if set(actual) == set(expected): - break - time.sleep(1) - assert set(actual) == set(expected), actual - for quantile, value in expected.items(): - assert abs(actual[quantile] - value) <= 0.001, (actual, expected) - dashboard = get(args.grafana, '/api/dashboards/uid/asap-kll')['dashboard'] - assert dashboard['panels'][0]['targets'][0]['expr'].startswith('request_duration_estimate') - print('Verified p50/p99 in Prometheus, Grafana datasource queries, and provisioned dashboard.') - - -if __name__ == '__main__': - main() diff --git a/docs/demo-guide.md b/docs/demo-guide.md deleted file mode 100644 index ef48d22..0000000 --- a/docs/demo-guide.md +++ /dev/null @@ -1,153 +0,0 @@ -# ASAPCollector multi-process OTAP demo - -The runnable demo executes four `urn:asap:processor:asap_sketches` dataflow -processors in four child OS processes: two creators, one merger, and one -estimator. Each child owns a real OTAP `RuntimePipeline`. - -```mermaid -flowchart LR - A["create A
OS process"] -->|"OTLP Metrics
Summary + ASAPv1"| M["merge
OS process"] - B["create B
OS process"] -->|"OTLP Metrics
Summary + ASAPv1"| M - M -->|"OTLP Metrics
Summary + ASAPv1"| E["estimate
OS process"] - E -->|"OTLP Metrics
Gauge p50/p99"| O[output] -``` - -Process boundaries contain standard protobuf -`ExportMetricsServiceRequest` messages. Inside each worker the message is -native `OtapPdata`. Sketch state is an OTLP Summary data point whose -`sketch.envelope` bytes use asap_sketchlib's self-describing format and begin -with the `ASAPv1` magic bytes. The removed private sketch-stream batch format -is not used. - -## Run - -```sh -cd asap-precompute-rs -cargo run --bin asap-otap-demo --features otap-engine -``` - -The parent prints each child PID and only the final p50/p99 values. It does not -dump intermediate payloads to stdout. - -## Official OTAP debugging output - -Every worker inserts OTAP's official `urn:otel:processor:debug` after the ASAP -processor: - -```yaml -debug: - type: "urn:otel:processor:debug" - config: - verbosity: detailed - mode: batch - signals: [metrics] - output: "/tmp/.../merge.debug.log" -``` - -This processor forwards the original pdata unchanged and writes OTAP's logical -OTLP view to a per-worker file. Detailed mode shows metric/data-point types, -Summary fields, and `sketch.*` attributes. The demo prints the trace directory -at completion. OTAP internal telemetry is useful for node throughput and -latency, but it does not replace payload/type inspection; the validation -exporter is intended for assertions rather than interactive tracing. - -The official debug processor describes the logical OTLP view. It does not dump -the physical Arrow child schemas. The official console exporter can render -native OTAP and OTLP payloads, but it is a terminal sink, while the debug -processor is appropriate here because records must continue to the next -process boundary. - -## Automated coverage - -The existing integration scenario still verifies the four-processor graph in -one runtime: - -```sh -cargo test --features otap-engine --test otap_pipeline_e2e -``` - -The runnable binary additionally asserts that the final output contains p50 -and p99 after the four child processes finish successfully. - -For a continuous stream with one pinned process per processor, rolling -quantiles, and a live KLL vs exact comparison, see -[the multi-process streaming KLL demo](./kll-windows.md). - -## Write KLL estimates to Prometheus and view them in Grafana - -The optional export sends the validated estimator output as OTLP/HTTP protobuf -straight to Prometheus. Prometheus stores the gauges and Grafana queries them. -It requires `curl` (7.76 or newer) on the machine running the demo, plus Docker -Compose for the bundled backend. No additional collector is required. - -From the repository root: - -```sh -docker compose -f demos/kll-grafana/compose.yaml up -d -cd asap-precompute-rs -cargo run --bin asap-otap-demo --features otap-engine -- \ - --prometheus-otlp-endpoint http://localhost:9090/api/v1/otlp/v1/metrics -``` - -Open [the KLL dashboard](http://localhost:3000/d/asap-kll) (anonymous viewer -access is enabled for this local demo). The dashboard provisions its Prometheus -datasource automatically and displays p50/p99 using: - -```promql -request_duration_estimate{quantile=~"0.5|0.99"} -``` - -The [Prometheus OTLP receiver](https://prometheus.io/docs/guides/opentelemetry/) -is enabled with `--web.enable-otlp-receiver`. Its default translation converts -`request.duration.estimate` to `request_duration_estimate` and retains the -`quantile` label. [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) -connects that datasource to the dashboard. - -This is a finite demo: one run writes one sample per quantile. Repeat the -command to add samples to the time series. Instant queries show the latest -sample for Prometheus's default five-minute lookback; the history panel shows -previous samples within the selected time range. Values use the demo input's -units (the sequential input is synthetic). The outbound copy uses the current -wall-clock timestamp, so the synthetic 1970-era window remains available in -`out.otlp` and debug traces while the dashboard shows the completed estimate -now. Original gauge values, resource attributes, and series labels are preserved; -window-boundary metadata is excluded from Prometheus labels to keep series stable. -Series labels stored on OTAP scopes (including `quantile`) are also copied to -gauge point attributes, which Prometheus uses to identify separate series. -Only the `kll` scenario supports this option. HTTP errors, invalid OTLP -responses, and partial rejection fail the command; no automatic retry can -silently duplicate a partially accepted write. - -To check the entire path after building the binary, run from the repository root: - -```sh -python3 demos/kll-grafana/verify.py asap-precompute-rs/target/debug/asap-otap-demo -``` - -The check compares the printed estimates with Prometheus results and Grafana's -queries through its configured datasource, and verifies the dashboard exists. -Set `PROMETHEUS_PORT` and `GRAFANA_PORT` when starting Compose if the default -ports are occupied, and supply matching `--prometheus` / `--grafana` URLs to -the check. Services bind to localhost and data persists in Compose volumes. -Stop them with `docker compose -f demos/kll-grafana/compose.yaml down`; add `-v` -when you want to remove the demo's stored data. - -## Before and after example - -Without the export option, the demo prints estimates and retains local OTLP -files and debug traces. It does not write samples to Prometheus: - -![Console output without Prometheus export](./images/kll-before-console.png) - -This image renders captured console output from a real run; process launch -lines and trace directory paths are omitted for readability. - -With `--prometheus-otlp-endpoint`, the same estimated values are stored as -separate quantile series and appear in the provisioned Grafana dashboard: - -![Grafana dashboard after exporting KLL estimates](./images/kll-after-grafana.png) - -The screenshot is from the running Grafana demo. This run produced p50 = 100 -and p99 = 198 for the synthetic sequential input. Prometheus and Grafana -queries returned those same values. The history panel uses Prometheus range -queries; displayed steps can repeat the latest sample between demo runs. diff --git a/docs/kll-windows.md b/docs/kll-windows.md index ad70005..f53a8d7 100644 --- a/docs/kll-windows.md +++ b/docs/kll-windows.md @@ -105,9 +105,7 @@ KLL answers stayed within the backend's 5% tolerance in every window The Compose stack uses host networking so Prometheus, Grafana and the metrics endpoint listen only on `127.0.0.1` (ports 9090, 3000 and 9465; -`PROMETHEUS_PORT` and `GRAFANA_PORT` change the first two). The finite -`asap-otap-demo` export described in the [demo guide](./demo-guide.md) still -pushes to the same Prometheus over OTLP. +`PROMETHEUS_PORT` and `GRAFANA_PORT` change the first two). ## Configure an OTAP rolling estimator @@ -118,7 +116,6 @@ ASAPv1 KLL panes of `window_slide` width: type: urn:asap:processor:asap_sketches config: sketch_type: kll - encoding: Msgpack window_size: 300s window_slide: 5s output_metric_name: request.duration.rolling_estimate @@ -136,8 +133,8 @@ Omitting `window_slide` retains the original tumbling runtime behavior. Pane boundaries align to Unix time and are half-open `[start, end)`. The lookback must be an integer multiple of the pane width, and both durations -must be whole milliseconds. Sliding is currently KLL-only with full Msgpack -encoding, no delta transmission, and no late scalar updates to closed panes. +must be whole milliseconds. Sliding is currently KLL-only, with no late +scalar updates to closed panes. The first results use the observations available during warmup; a complete lookback becomes available after the process has run for that duration. @@ -156,12 +153,11 @@ upstream panes, plus reusable aligned range queries: ```rust use std::time::Duration; -use asap_precompute_rs::{Encoding, PrecomputeConfig, SketchType}; +use asap_precompute_rs::{PrecomputeConfig, SketchType}; use asap_precompute_rs::kll_windows::KllWindowStore; let config = PrecomputeConfig { sketch_type: SketchType::KLLSketch, - encoding: Encoding::Msgpack, quantiles: vec![0.5, 0.99], ..Default::default() }; From 143d4a5f2d47c7ca0cc4dc131de17661034f1799 Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Mon, 12 Oct 2026 18:08:22 +0000 Subject: [PATCH 2/2] Serialize every sketch as ASAPv1; let every sketch type transmit Every wrapper now wraps sketchlib's native sketch (DDSketch, ErtlMLE HyperLogLog, CountMin, Count, KLL, CMSHeap) and snapshot() returns its self-describing ASAPv1 envelope. Sketch::merge_snapshot decodes one and merges it; delta transmission (compute_delta_against, apply_delta*, DeltaResult, SnapshotCache, delta_transmission/delta_threshold), the Encoding enum, the proto path, hash_spec and KLL's replay history are gone. The OTAP codec validates sketch.envelope against the declared sketch type on every decode path, including Gauge data points. The processor accepts transmit_sketch for every sketch type and dispatches countminsketch_heap (rows/columns/k), whose estimate mode emits one Gauge per heavy hitter with key and rank labels. controller_url, agent_id and encoding are removed from the plugin config. HLL is fixed at sketchlib's precision-14 Ertl-MLE HyperLogLog, so the hll precision parameter is gone. CMS/CountSketch rows are clamped to MATRIX_MAX_ROWS; the power-of-two column rounding is no longer needed. --- asap-precompute-rs/Cargo.lock | 185 ------ asap-precompute-rs/Cargo.toml | 25 +- asap-precompute-rs/README.md | 42 +- .../plugins/asap_sketches/README.md | 19 +- .../plugins/asap_sketches/sample.toml | 22 +- .../src/bin/stream_bench/model.rs | 13 +- .../src/bin/stream_bench/topk.rs | 3 +- asap-precompute-rs/src/config.rs | 72 +-- asap-precompute-rs/src/envelope.rs | 172 +----- asap-precompute-rs/src/kll_windows.rs | 35 +- asap-precompute-rs/src/lib.rs | 19 +- asap-precompute-rs/src/observation.rs | 15 +- asap-precompute-rs/src/otap/codec.rs | 278 +++------ asap-precompute-rs/src/otap/config.rs | 191 ++---- asap-precompute-rs/src/otap/mod.rs | 22 +- asap-precompute-rs/src/otap/processor.rs | 32 +- asap-precompute-rs/src/otap/schema.rs | 165 +----- asap-precompute-rs/src/precompute.rs | 159 +---- asap-precompute-rs/src/sketches/cms.rs | 470 +++------------ asap-precompute-rs/src/sketches/cms_heap.rs | 87 ++- .../src/sketches/countsketch.rs | 456 +++------------ asap-precompute-rs/src/sketches/ddsketch.rs | 345 ++--------- asap-precompute-rs/src/sketches/hll.rs | 349 ++--------- asap-precompute-rs/src/sketches/kll.rs | 292 ++-------- asap-precompute-rs/src/sketches/mod.rs | 40 +- asap-precompute-rs/src/snapshot_cache.rs | 546 ------------------ asap-precompute-rs/src/window.rs | 39 +- asap-precompute-rs/tests/kll_windows.rs | 7 +- asap-precompute-rs/tests/otap_pipeline_e2e.rs | 8 +- asap-precompute-rs/tests/runtime.rs | 341 ++--------- docs/data_model.md | 36 +- docs/df-processor-topology.md | 59 +- 32 files changed, 768 insertions(+), 3776 deletions(-) delete mode 100644 asap-precompute-rs/src/snapshot_cache.rs diff --git a/asap-precompute-rs/Cargo.lock b/asap-precompute-rs/Cargo.lock index 79e6252..6d1167a 100644 --- a/asap-precompute-rs/Cargo.lock +++ b/asap-precompute-rs/Cargo.lock @@ -78,18 +78,6 @@ dependencies = [ "libc", ] -[[package]] -name = "anes" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299" - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - [[package]] name = "anyhow" version = "1.0.103" @@ -358,7 +346,6 @@ dependencies = [ "arrow-schema", "asap_sketchlib", "async-trait", - "criterion", "humantime-serde", "libc", "linkme", @@ -370,9 +357,6 @@ dependencies = [ "otel-arrow-dfe-pdata-views", "otel-arrow-dfe-state", "otel-arrow-dfe-telemetry", - "proptest", - "prost 0.13.5", - "prost 0.14.4", "serde", "serde_json", "thiserror 1.0.69", @@ -797,12 +781,6 @@ dependencies = [ "unicode-normalization", ] -[[package]] -name = "cast" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5" - [[package]] name = "cc" version = "1.2.66" @@ -899,31 +877,6 @@ dependencies = [ "inout", ] -[[package]] -name = "clap" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" -dependencies = [ - "clap_builder", -] - -[[package]] -name = "clap_builder" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" -dependencies = [ - "anstyle", - "clap_lex", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - [[package]] name = "clru" version = "0.6.3" @@ -1115,39 +1068,6 @@ dependencies = [ "cfg-if", ] -[[package]] -name = "criterion" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1c047a62b0cc3e145fa84415a3191f628e980b194c2755aa12300a4e6cbd928" -dependencies = [ - "anes", - "cast", - "ciborium", - "clap", - "criterion-plot", - "itertools 0.13.0", - "num-traits", - "oorandom", - "plotters", - "rayon", - "regex", - "serde", - "serde_json", - "tinytemplate", - "walkdir", -] - -[[package]] -name = "criterion-plot" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b1bcc0dc7dfae599d84ad0b1a55f80cde8af3725da8313b528da95ef783e338" -dependencies = [ - "cast", - "itertools 0.13.0", -] - [[package]] name = "crossbeam-channel" version = "0.5.17" @@ -4563,12 +4483,6 @@ version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" -[[package]] -name = "oorandom" -version = "11.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e" - [[package]] name = "openssl" version = "0.10.81" @@ -5275,34 +5189,6 @@ version = "0.3.34" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548" -[[package]] -name = "plotters" -version = "0.3.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5aeb6f403d7a4911efb1e33402027fc44f29b5bf6def3effcc22d7bb75f2b747" -dependencies = [ - "num-traits", - "plotters-backend", - "plotters-svg", - "wasm-bindgen", - "web-sys", -] - -[[package]] -name = "plotters-backend" -version = "0.3.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df42e13c12958a16b3f7f4386b9ab1f3e7933914ecea48da7139435263a4172a" - -[[package]] -name = "plotters-svg" -version = "0.3.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "51bae2ac328883f7acdfea3d66a7c35751187f870bc81f94563733a154d7a670" -dependencies = [ - "plotters-backend", -] - [[package]] name = "portable-atomic" version = "1.15.0" @@ -5397,25 +5283,6 @@ dependencies = [ "parking_lot", ] -[[package]] -name = "proptest" -version = "1.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744" -dependencies = [ - "bit-set", - "bit-vec", - "bitflags 2.13.0", - "num-traits", - "rand 0.9.4", - "rand_chacha 0.9.0", - "rand_xorshift", - "regex-syntax", - "rusty-fork", - "tempfile", - "unarray", -] - [[package]] name = "prost" version = "0.13.5" @@ -5532,12 +5399,6 @@ dependencies = [ "pulldown-cmark", ] -[[package]] -name = "quick-error" -version = "1.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0" - [[package]] name = "quote" version = "1.0.46" @@ -5641,15 +5502,6 @@ version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" -[[package]] -name = "rand_xorshift" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a" -dependencies = [ - "rand_core 0.9.5", -] - [[package]] name = "rayon" version = "1.12.0" @@ -6022,18 +5874,6 @@ version = "1.0.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" -[[package]] -name = "rusty-fork" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cc6bf79ff24e648f6da1f8d1f011e9cac26491b619e6b9280f2b47f1774e6ee2" -dependencies = [ - "fnv", - "quick-error", - "tempfile", - "wait-timeout", -] - [[package]] name = "ryu" version = "1.0.23" @@ -6719,16 +6559,6 @@ dependencies = [ "zerovec", ] -[[package]] -name = "tinytemplate" -version = "1.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "be4d6b5f19ff7664e8c98d03e2139cb510db9b0a60b55f8e8709b689d939b6bc" -dependencies = [ - "serde", - "serde_json", -] - [[package]] name = "tinyvec" version = "1.12.0" @@ -7075,12 +6905,6 @@ dependencies = [ "arrayvec", ] -[[package]] -name = "unarray" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94" - [[package]] name = "unicase" version = "2.9.0" @@ -7261,15 +7085,6 @@ version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5c3082ca00d5a5ef149bb8b555a72ae84c9c59f7250f013ac822ac2e49b19c64" -[[package]] -name = "wait-timeout" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11" -dependencies = [ - "libc", -] - [[package]] name = "walkdir" version = "2.5.0" diff --git a/asap-precompute-rs/Cargo.toml b/asap-precompute-rs/Cargo.toml index 302ff29..274a537 100644 --- a/asap-precompute-rs/Cargo.toml +++ b/asap-precompute-rs/Cargo.toml @@ -9,9 +9,6 @@ license = "MIT" [dependencies] asap_sketchlib = { git = "https://github.com/ProjectASAP/asap_sketchlib", rev = "3fc3442c3275d374f79ba522f473de83912c6345" } serde = { version = "1", features = ["derive"] } -prost = "0.13" -# OTAP's generated protobuf types implement the 0.14 Message trait. -prost-otap = { package = "prost", version = "0.14", optional = true } thiserror = "1" # Apache Arrow is only pulled in by the OTAP codec. Consumers on the @@ -54,16 +51,13 @@ libc = { version = "0.2", optional = true } tokio = { version = "1", features = ["sync", "rt", "time", "net"], optional = true } [dev-dependencies] -proptest = "1" serde_json = "1" # The full Tokio runtime is required by the live OTAP pipeline integration test. tokio = { version = "1", features = ["full"] } -criterion = { version = "0.7", features = ["html_reports"] } [features] default = [] -# Enables the OTAP codec + plugin lifecycle at `crate::otap` -# (`decode_batch` / `encode_batch` over `arrow::RecordBatch`). No dependency -# on real OTAP Dataflow crates — this is the standalone-buildable half. +# Enables `crate::otap` (plugin config + carrier attribute keys). No +# dependency on real OTAP Dataflow crates. otap = ["dep:arrow-array", "dep:arrow-schema"] # Enables `otap::processor`/`otap::codec` — the real # `local::Processor` node (`AsapSketchesProcessor`) and its @@ -75,7 +69,6 @@ otap = ["dep:arrow-array", "dep:arrow-schema"] # folded into `otap`. otap-engine = [ "otap", - "dep:prost-otap", "dep:otel-arrow-dfe-engine", "dep:otel-arrow-dfe-otap", "dep:otel-arrow-dfe-config", @@ -99,21 +92,7 @@ stream-benchmark = [ "otel-arrow-dfe-core-nodes/dev-tools", ] -[[bin]] -name = "asap-otap-demo" -path = "src/bin/asap_otap_demo.rs" -required-features = ["otap-engine"] - [[bin]] name = "asap-stream-bench" path = "src/bin/asap_stream_bench.rs" required-features = ["stream-benchmark"] - -[[bench]] -name = "asap_sketch_pipeline" -harness = false -required-features = ["otap-engine"] - -[profile.bench] -debug = true -strip = "none" diff --git a/asap-precompute-rs/README.md b/asap-precompute-rs/README.md index 54dfc4c..fd0fe97 100644 --- a/asap-precompute-rs/README.md +++ b/asap-precompute-rs/README.md @@ -1,14 +1,15 @@ # asap-precompute-rs `asap-precompute-rs` is ASAP's host-neutral edge precompute runtime. -It owns the windowing, snapshot caching, and delta-encoding logic for -the Rust edge runtime, and — under the `otap` feature — the OTAP -Dataflow integration layer (Arrow codec + native OTAP processor). +It owns the windowing logic for the Rust edge runtime, and — under the +`otap-engine` feature — the OTAP Dataflow processor and its native +`OtapPdata` codec. -Per-platform Adapter implementations (the Layer-4 shims) translate their -host's native event into an `Observation`, hand it to a `Precompute`, -and translate the runtime's emitted `SketchEnvelope` back to the host's -native event. +A host translates its native event into an `Observation`, hands it to a +`Precompute`, and translates each emitted `SketchEnvelope` back to its +own format. A sketch envelope's payload is the sketch's full state as the +self-describing ASAPv1 envelope from `asap_sketchlib`; there are no +deltas. ## Module map @@ -17,24 +18,22 @@ native event. | Rust module | Responsibility | | --- | --- | | `observation` | Input event model (`Observation`, `ObservationValue`). | -| `envelope` | Output model (`SketchEnvelope`, `SketchType`, `Encoding`). | +| `envelope` | Output model (`SketchEnvelope`, `SketchType`). | | `precompute` | Core runtime: `observe` / `tick` / `drain` / `update_config`. | | `window` | Tumbling-window state and rotation. | -| `snapshot_cache` | Delta encoding — full vs delta snapshot decisions. | +| `kll_windows` | Retained KLL panes and sliding-window estimates. | | `matchers` | Label matching / series-key construction. | | `config` | `PrecomputeConfig`, `SketchParams`, `WindowSpec`. | -| `sketches/*` | Sketch wrappers over `asap_sketchlib` (DDSketch, KLL, HLL, CountSketch, CountMin-Sketch). | -| `adapter` | Host-adapter trait surface. | -| `control_channel` | Controller plan delivery (`ControlChannel`). | +| `sketches/*` | Sketch wrappers over `asap_sketchlib` (DDSketch, KLL, HLL, CountSketch, Count-Min, Count-Min with heap). | -### OTAP Dataflow integration (`src/otap/`, `otap` feature) +### OTAP Dataflow integration (`src/otap/`) | Rust module | Responsibility | | --- | --- | -| `otap::config` | `PluginConfig` + `resolve()` — 5-sketch `sketch_type` dispatch. | -| `otap::decode` / `otap::encode` | Arrow `RecordBatch` ↔ `Observation` / `SketchEnvelope` codec. | -| `otap::records` | `OtapMetricRecords` + `flatten` / `lift` (sibling-batch ↔ flat-batch projection, Strategy-B attribute-lift). | -| `otap::schema` | Well-known column names + `_asap_*` Strategy-B carrier keys. | +| `otap::config` | `PluginConfig` + `resolve()` — `sketch_type` dispatch. | +| `otap::schema` | `sketch.*` carrier attribute keys. | +| `otap::codec` | `SketchEnvelope` ↔ `OtapPdata` (`otap-engine`). | +| `otap::processor` | The `urn:asap:processor:asap_sketches` processor (`otap-engine`). | ## Dependency @@ -44,14 +43,11 @@ public dependency. ## Validation ```sh -cargo build -cargo build --features otap -cargo test --features otap -cargo clippy --all-targets --features otap -- -D warnings cargo fmt --check +cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings +cargo test --features otap-engine,stream-benchmark ``` The default build compiles the row-oriented runtime only (no -Arrow/Tokio); the `otap` feature enables the OTAP codec. The test suite -covers the runtime plus the OTAP codec (`tests/otap_codec.rs`) and the +Arrow/Tokio). The test suite covers the runtime, the OTAP codec, and the real dataflow processor (`tests/otap_pipeline_e2e.rs`). diff --git a/asap-precompute-rs/plugins/asap_sketches/README.md b/asap-precompute-rs/plugins/asap_sketches/README.md index de0aafe..ecc4935 100644 --- a/asap-precompute-rs/plugins/asap_sketches/README.md +++ b/asap-precompute-rs/plugins/asap_sketches/README.md @@ -18,8 +18,7 @@ builds the whole thing. Design notes: -- This is a single unified plugin rather than five per-sketch - plugins. +- This is a single unified plugin rather than one plugin per sketch. - Registered under the URN `urn:asap:processor:asap_sketches` (`processor.rs`'s `ASAP_SKETCHES_PROCESSOR_FACTORY`). - The configuration shape is documented by @@ -37,26 +36,26 @@ Field reference: | Field | Required | Description | |---|---|---| -| `sketch_type` | yes | One of `ddsketch` / `kll` / `hll` / `countsketch` / `countminsketch`. Case-insensitive. | +| `sketch_type` | yes | One of `ddsketch` / `kll` / `hll` / `countsketch` / `countminsketch` / `countminsketch_heap`. Case-insensitive. | | `window_size` | yes | Duration string (e.g. `"10s"`). Tumbling window rotation period. | | `output_metric_name` | yes | Metric name stamped onto every emitted envelope. | | `agg_id` | optional | Controller-plan join key. Defaults to `0`. | | `sketch_params.relative_accuracy` | ddsketch | DDSketch alpha (`0 < α < 1`). Default `0.01`. | | `sketch_params.k` | kll | KLL buffer size. Default `200`. | | `sketch_params.seed` | kll | Optional deterministic compaction seed. Default unseeded. | -| `sketch_params.precision` | hll | HLL register-count exponent (4-18). Default `12`. | | `sketch_params.width` | countsketch, cms | Sketch column count. Default `2048`. | | `sketch_params.depth` | countsketch, cms | Sketch row count. Default `4`. | +| `sketch_params.rows` / `columns` | countminsketch_heap | CMS dimensions. Default `4` / `2048`. | +| `sketch_params.k` | countminsketch_heap | Heavy-hitter heap size. Default `10`. | | `default_key` | optional | CountSketch fallback key when an observation has neither bytes nor labels. Defaults to `output_metric_name`. | | `omit_resource_attrs` | optional | Exclude resource attributes from series identity. Default `false`. | | `global_aggregation` | optional | Collapse all observations into one series. Default `false`. | | `emit_window_stats` | optional | Add `sample_count` and `window_duration_seconds` labels. Default `false`. | -| `encoding` | optional | Sketch transport uses KLL `Msgpack`/`MsgpackDelta`, the canonical self-describing ASAPv1 format. Default `Msgpack`. Other sketch/encoding combinations require estimate mode. | -| `delta_transmission` | optional | Emit deltas after the initial full snapshot. Default `false`. | -| `transmit_sketch` | optional | `true` emits canonical KLL ASAPv1 state; `false` permits supported scalar estimates from other algorithms. Default `true`. | +| `transmit_sketch` | optional | `true` emits each window's full sketch as the self-describing ASAPv1 envelope from `asap_sketchlib` (every sketch type; there are no deltas). `false` emits scalar estimates: quantiles (DDSketch/KLL), a cardinality (HLL), or the top-k keys with `key` and `rank` labels (`countminsketch_heap`). Default `true`. | | `quantiles` | estimate mode | DDSketch/KLL quantiles to emit, such as `[0.5, 0.99]`. | -| `controller_url` | optional | Legacy bootstrap field, predates the engine's own OpAMP controller extension (which now owns `instance_uid`/`endpoint`/`agent_description` centrally) — likely a deprecation candidate rather than something to keep pushing new values into. | -| `agent_id` | optional | Same caveat as `controller_url`. | + +HLL uses sketchlib's fixed precision-14 HyperLogLog, so it has no +`sketch_params`. ## Lifecycle @@ -66,7 +65,7 @@ calls — no separate Tokio-task lifecycle needed: 1. **`Message::PData`** — `codec::decode_pdata_to_observations()` turns the real `OtapPdata` directly into `Vec` - (content-routed: a data point carrying `_asap_envelope` merges as a + (content-routed: a data point carrying `sketch.envelope` merges as a pre-aggregated sketch via `Precompute::observe_envelope`; everything else is a plain scalar sample). 2. **`NodeControlMsg::TimerTick`** (regular flush) / **`Shutdown`** diff --git a/asap-precompute-rs/plugins/asap_sketches/sample.toml b/asap-precompute-rs/plugins/asap_sketches/sample.toml index 03fc2d9..e45b59d 100644 --- a/asap-precompute-rs/plugins/asap_sketches/sample.toml +++ b/asap-precompute-rs/plugins/asap_sketches/sample.toml @@ -16,8 +16,9 @@ plugin = "asap_sketches" ## "ddsketch" (quantiles, relative-error) ## "kll" (quantiles, rank-error) ## "hll" (cardinality) - ## "countsketch" (frequency / top-k) + ## "countsketch" (frequency) ## "countminsketch" (frequency, biased upward) + ## "countminsketch_heap" (frequency plus top-k heavy hitters) sketch_type = "kll" ## Window size. Rust duration string, parsed via humantime; @@ -36,31 +37,22 @@ plugin = "asap_sketches" ## Output and series controls. Creation and merge processors keep ## transmit_sketch=true. An estimation processor sets it to false and - ## supplies quantiles (DDSketch/KLL) instead. + ## supplies quantiles (DDSketch/KLL) instead. Sketches travel as the + ## self-describing ASAPv1 envelope from asap_sketchlib, always full state. transmit_sketch = true # quantiles = [0.5, 0.99] - encoding = "Msgpack" # canonical ASAPv1 self-describing envelope - # delta_transmission = false # default_key = "fallback" # CountSketch only # omit_resource_attrs = false # global_aggregation = false # emit_window_stats = false - ## Legacy bootstrap fields — predate the engine's own OpAMP - ## controller extension, which now owns instance identification and - ## remote-config delivery centrally. Sketch-runtime params - ## (sketch_type, window, sketch_params) in this block are push-overridable - ## via NodeControlMsg::Config from that path; these two fields are - ## likely deprecation candidates rather than something to configure - ## going forward. - # controller_url = "http://controller:8080" - # agent_id = "otap-host-01" - ## Sketch-specific tuning knobs. Only the keys relevant to ## `sketch_type` are read; others are ignored. [pipelines.metrics.processors.config.sketch_params] k = 200 # kll — buffer size # relative_accuracy = 0.01 # ddsketch estimate mode — relative error - # precision = 12 # hll — register count exponent (4-18) # width = 2048 # countsketch / countminsketch — width # depth = 4 # countsketch / countminsketch — depth + # rows = 4 # countminsketch_heap — rows + # columns = 2048 # countminsketch_heap — columns + # k = 10 # countminsketch_heap — heap size (default 10) diff --git a/asap-precompute-rs/src/bin/stream_bench/model.rs b/asap-precompute-rs/src/bin/stream_bench/model.rs index b125d14..104aa3e 100644 --- a/asap-precompute-rs/src/bin/stream_bench/model.rs +++ b/asap-precompute-rs/src/bin/stream_bench/model.rs @@ -1,7 +1,7 @@ //! Count-aligned windows for a fair streaming comparison. No wall-clock flush races. use super::profile::{self, Category}; use asap_precompute_rs::{ - envelope::{Encoding, SketchEnvelope, SketchType}, + envelope::{SketchEnvelope, SketchType}, observation::{KeyValue, Observation}, otap::codec::{decode_pdata_to_observations, encode_envelopes_to_pdata}, precompute::{QuantileSketch, Sketch}, @@ -70,9 +70,7 @@ impl Meta { ], window_start_ms: 1000, window_end_ms: 2000, - encoding: Encoding::Unspecified, payload: vec![], - hash_spec: None, metric_name: "http.server.request.duration".into(), count: 0, aggregation_temporality: 0, @@ -86,13 +84,12 @@ pub fn values_pdata(values: &[f64], meta: Meta) -> Result { .map_err(|e| e.to_string()) } fn new_sketch(seed: u64) -> KLLWrapper { - KLLWrapper::new(400, Some(seed)).with_wire_encoding(Encoding::Msgpack) + KLLWrapper::new(400, Some(seed)) } fn sketch_pdata(sketch: &KLLWrapper, count: usize, meta: Meta) -> Result { let _cpu = profile::scope(Category::PdataCodec); let mut env = meta.envelope(0.0); env.sketch_type = SketchType::KLLSketch; - env.encoding = Encoding::Msgpack; env.payload = { let _cpu = profile::scope(Category::SketchCodec); sketch.snapshot().map_err(|e| e.to_string())? @@ -106,11 +103,7 @@ fn decode_sketch(o: &Observation, count: usize) -> Result { if env.count != count as u64 { return Err("sketch count mismatch".into()); } - let mut sketch = new_sketch(99); - sketch - .apply_delta_encoded(&env.payload, env.encoding) - .map_err(|e| e.to_string())?; - Ok(sketch) + KLLWrapper::from_snapshot(&env.payload).map_err(|e| e.to_string()) } const QUANTILES: [&str; 2] = ["0.5", "0.99"]; /// Tumbling quantiles, plus rolling quantiles when the estimate retains windows. diff --git a/asap-precompute-rs/src/bin/stream_bench/topk.rs b/asap-precompute-rs/src/bin/stream_bench/topk.rs index 3f5f138..ab66f1a 100644 --- a/asap-precompute-rs/src/bin/stream_bench/topk.rs +++ b/asap-precompute-rs/src/bin/stream_bench/topk.rs @@ -6,7 +6,7 @@ use super::model::{Meta, NormalizedBatch, MAX_PENDING}; use super::profile::{self, Category}; use asap_precompute_rs::{ - envelope::{Encoding, SketchEnvelope, SketchType}, + envelope::{SketchEnvelope, SketchType}, observation::{KeyValue, Observation}, otap::codec::{decode_pdata_to_observations, encode_envelopes_to_pdata}, precompute::FrequencySketch, @@ -108,7 +108,6 @@ fn sketch_pdata(sketch: &CMSHeapWrapper, count: usize, meta: Meta) -> Result; /// Returns the parameter value or `default` if not present. @@ -156,10 +155,8 @@ pub fn sketch_size_string(sketch_type: SketchType, params: &SketchParams) -> Opt .get("k") .and_then(|v| as_wire_u64(*v)) .map(|v| v.to_string()), - SketchType::HLLSketch => params - .get("precision") - .and_then(|v| as_wire_u64(*v)) - .map(|v| v.to_string()), + // HLLWrapper is sketchlib's fixed precision-14 HyperLogLog. + SketchType::HLLSketch => Some("14".to_string()), SketchType::CountSketch => match (params.get("width"), params.get("depth")) { (Some(w), Some(d)) => match (as_wire_u64(*w), as_wire_u64(*d)) { (Some(w), Some(d)) => Some(format!("{w} x {d}")), @@ -188,9 +185,7 @@ pub fn sketch_size_string(sketch_type: SketchType, params: &SketchParams) -> Opt /// carrying a fractional part (e.g. a unit mixup like `k = 0.5`) all /// indicate a misconfigured param. Silently truncating (`as u64`) /// would report a fabricated, wrong `sketch_size` on the wire instead -/// (`k = 0.5` truncating to `"0"`); better to omit the field, mirroring -/// `resolve_hash_seed`'s "omit rather than fabricate" stance for a -/// malformed input. +/// (`k = 0.5` truncating to `"0"`); better to omit the field. fn as_wire_u64(v: f64) -> Option { if !v.is_finite() || v < 0.0 || v.fract() != 0.0 { return None; @@ -222,26 +217,6 @@ pub struct PrecomputeConfig { /// Chooses between sketch-on-the-wire and quantile-summary-on- /// the-wire output. pub transmit_sketch: bool, - /// Enables delta encoding against the cached outbound snapshot. - pub delta_transmission: bool, - /// Caps the absolute delta size; if the computed delta exceeds - /// `delta_threshold * full-state size`, the runtime emits the - /// full state instead. - /// - /// Mirrors today's `ddsketch::ComputeDelta` threshold semantics. - /// The unit is sketch-specific (`u64` for DDSketch/CMS bucket - /// counts; `f64` for CountSketch L2 cells, stored here as - /// rounded-up `u64` — adapters convert as needed). - pub delta_threshold: u64, - /// Overrides the wire encoding of emitted envelopes. - /// - /// Default zero value (`Unspecified`, treated as `ProtoFull`) is - /// used for non-delta transmissions. When `delta_transmission` is - /// true, the runtime emits `ProtoDelta` frames after the initial - /// `ProtoFull` snapshot. `Msgpack` is supported for legacy paths - /// (HLL / CountSketch / CMS today carry an explicit `encoding` - /// knob in their existing config). - pub encoding: Encoding, /// Output mode when `transmit_sketch` is false. For /// quantile-sketch types (DDSketch / KLL), each quantile in this /// list becomes a separate gauge metric on emit. Empty list means @@ -340,9 +315,6 @@ impl Default for PrecomputeConfig { window: WindowSpec::default(), matchers: Vec::new(), aggregate_by: Vec::new(), - delta_transmission: false, - delta_threshold: 0, - encoding: Encoding::default(), quantiles: Vec::new(), sketch_params: SketchParams::default(), max_series: 0, @@ -409,12 +381,10 @@ impl PrecomputeConfig { } } -/// Versioned bundle of configs delivered by -/// [`crate::control_channel::ControlChannel`]. +/// Versioned bundle of configs delivered by the control plane. #[derive(Clone, Debug, Default, PartialEq, serde::Serialize, serde::Deserialize)] pub struct PrecomputeConfigSet { - /// Monotonically increasing across plans; ack with - /// `ControlChannel::ack`. + /// Monotonically increasing across plans. pub version: u64, /// `PrecomputeConfig`s active for this host. pub configs: Vec, @@ -438,28 +408,12 @@ mod tests { assert_eq!(cfg.sketch_type, SketchType::Unspecified); assert_eq!(cfg.mode, AggregationMode::Tumbling); assert_eq!(cfg.on_overflow, OnOverflow::Drop); - assert_eq!(cfg.encoding, Encoding::Unspecified); assert_eq!(cfg.window.size, Duration::ZERO); - assert!(!cfg.delta_transmission); assert!(!cfg.global_aggregation); assert!(!cfg.omit_resource_attrs); assert!(!cfg.emit_window_stats); } - #[test] - fn aggregation_mode_names_match_go() { - assert_eq!(AggregationMode::Tumbling.name(), "Tumbling"); - assert_eq!(AggregationMode::Sliding.name(), "Sliding"); - assert_eq!(AggregationMode::Batch.name(), "Batch"); - } - - #[test] - fn on_overflow_names_match_go() { - assert_eq!(OnOverflow::Drop.name(), "Drop"); - assert_eq!(OnOverflow::Block.name(), "Block"); - assert_eq!(OnOverflow::EvictOldest.name(), "EvictOldest"); - } - #[test] fn precompute_config_set_finds_by_agg_id() { let set = PrecomputeConfigSet { @@ -503,11 +457,9 @@ mod tests { Some("200".to_string()) ); - let mut hll = SketchParams::new(); - hll.insert("precision".into(), 12.0); assert_eq!( - sketch_size_string(SketchType::HLLSketch, &hll), - Some("12".to_string()) + sketch_size_string(SketchType::HLLSketch, &SketchParams::new()), + Some("14".to_string()) ); let mut cs = SketchParams::new(); diff --git a/asap-precompute-rs/src/envelope.rs b/asap-precompute-rs/src/envelope.rs index cf6e69f..b09549a 100644 --- a/asap-precompute-rs/src/envelope.rs +++ b/asap-precompute-rs/src/envelope.rs @@ -1,39 +1,14 @@ -//! Runtime view of the [`asap_sketchlib`] `SketchEnvelope` proto. -//! -//! The runtime's `SketchEnvelope` struct wraps the proto-encoded -//! payload bytes plus the surrounding runtime metadata that today's -//! OTel modified-OTLP variants (Strategy A) and tomorrow's Strategy-B -//! adapters (Telegraf / Vector / OTAP) both need (window bounds, -//! agg_id, encoding tag, sketch_type tag, host-neutral labels, metric -//! name, count, temporality). -//! -//! The actual prost-generated proto type is re-exported as -//! [`ProtoSketchEnvelope`] for adapters that need proto-level access -//! (e.g. for round-tripping the `payload` against `asap_sketchlib`'s -//! sketch-state oneof). +//! Runtime sketch envelope: an ASAPv1 sketch payload from +//! [`asap_sketchlib`] plus the runtime metadata around it (window +//! bounds, `agg_id`, sketch type, labels, metric name, count, +//! temporality). use crate::config::AggId; use crate::observation::KeyValue; -/// Re-export of the prost-generated proto `SketchEnvelope` type from -/// `asap_sketchlib`. -/// -/// This is the actual on-the-wire envelope (oneof'd over each sketch -/// state). Adapters that need to materialize the full proto -/// representation (e.g. to round-trip through `asap_sketchlib`'s -/// per-sketch state messages) reach for this type. The Layer-3 runtime -/// itself works only with the [`SketchEnvelope`] struct in this -/// module, which carries proto-encoded bytes in its `payload` field. -pub type ProtoSketchEnvelope = asap_sketchlib::proto::sketchlib::SketchEnvelope; - /// Identifies which sketch algorithm produced the envelope's payload -/// bytes. -/// -/// The `asap_sketchlib` proto today -/// carries the sketch type implicitly via its `sketch_state` oneof, -/// but the Layer-3 runtime keeps it as an explicit tag because -/// envelopes flow through code paths that don't always unmarshal the -/// inner state. +/// bytes. The ASAPv1 payload carries its own kind id; the runtime keeps +/// this explicit tag so routing does not need to parse the payload. #[derive( Copy, Clone, Debug, Default, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize, )] @@ -56,8 +31,7 @@ pub enum SketchType { } impl SketchType { - /// Returns the canonical name for the sketch type, matching the - /// well-known string spellings used by Strategy-B adapters. + /// Returns the canonical name for the sketch type. pub fn name(&self) -> &'static str { match self { Self::DDSketch => "DDSketch", @@ -69,73 +43,28 @@ impl SketchType { Self::Unspecified => "Unspecified", } } -} - -/// Describes how the bytes in [`SketchEnvelope::payload`] are encoded. -#[derive( - Copy, Clone, Debug, Default, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize, -)] -pub enum Encoding { - /// Encoding was not set; treated as `ProtoFull` for backwards- - /// compatibility but adapters should reject at strict-mode - /// boundaries. - #[default] - Unspecified, - /// `payload` is a proto-encoded full `SketchEnvelope` state. - ProtoFull, - /// `payload` is a proto-encoded sparse delta against the - /// receiver's cached snapshot of this series. - ProtoDelta, - /// `payload` is a msgpack-encoded full state (some sketches expose - /// msgpack as a faster wire format). - Msgpack, - /// `payload` is a msgpack-encoded sparse delta against the - /// receiver's cached snapshot of this series — the msgpack twin of - /// [`Self::ProtoDelta`]. Carried by the four delta-capable sketches - /// (DDSketch / HLL / CountSketch / CountMinSketch) as the - /// struct-of-parallel-arrays layout produced by `asap_sketchlib`'s - /// `compute_delta_msgpack`. KLL never emits this (full-only). - MsgpackDelta, -} - -impl Encoding { - /// Returns the canonical encoding name. - pub fn name(&self) -> &'static str { - match self { - Self::ProtoFull => "PROTO_FULL", - Self::ProtoDelta => "PROTO_DELTA", - Self::Msgpack => "MSGPACK", - Self::MsgpackDelta => "MSGPACK_DELTA", - Self::Unspecified => "UNSPECIFIED", - } - } - /// Whether this encoding serializes sketch state as MessagePack - /// (full or delta) rather than proto. Used by the sketch wrappers to - /// pick the wire codec. - pub fn is_msgpack(&self) -> bool { - matches!(self, Self::Msgpack | Self::MsgpackDelta) + /// Parses a canonical [`Self::name`]; an empty string is `Unspecified`. + pub fn from_name(name: &str) -> Option { + [ + Self::DDSketch, + Self::KLLSketch, + Self::HLLSketch, + Self::CountSketch, + Self::CountMinSketch, + Self::CountMinSketchWithHeap, + Self::Unspecified, + ] + .into_iter() + .find(|t| t.name() == name) + .or_else(|| name.is_empty().then_some(Self::Unspecified)) } } -/// Runtime view of the `asap_sketchlib` `SketchEnvelope` proto plus -/// the surrounding metadata that today's OTel modified-OTLP variants -/// and tomorrow's Strategy-B adapters both need (window bounds, -/// `agg_id`, encoding tag, sketch-type tag, host-neutral labels). -/// -/// The `payload` bytes are byte-identical to today's `asap_sketchlib` -/// proto encoding. +/// One emitted sketch (or scalar estimate) plus its runtime metadata. /// -/// The on-the-wire representation depends on the platform's encoding -/// strategy: -/// - Strategy A (OTel today): `payload` rides a typed -/// `pmetric::Metric.data` oneof variant (DDSketch / KLLSketch / …). -/// - Strategy B (Telegraf / Vector / OTAP): the whole struct is -/// decomposed into the well-known `_asap_envelope` / -/// `_asap_sketch_type` / `_asap_agg_id` / etc. carrier keys. -/// -/// Either way `payload` is byte-identical and is what -/// [`crate::precompute::Precompute::observe_envelope`] deserializes. +/// `payload` is the sketch's ASAPv1 envelope bytes and is what +/// [`crate::precompute::Precompute::observe_envelope`] merges. #[derive(Clone, Debug, Default, PartialEq, serde::Serialize, serde::Deserialize)] pub struct SketchEnvelope { /// Wire schema version. Adapters @@ -151,13 +80,6 @@ pub struct SketchEnvelope { /// OTel: `pmetric::ResourceMetrics::Resource()` attrs). Carried /// in-process so the adapter's encode path can faithfully /// reconstruct the host's resource hierarchy on emission. - /// - /// NOT on the proto wire today: proto's `repeated KeyValue labels` - /// carries only data-point labels; cross-host hops that need to - /// preserve resource scope can prefix-encode (e.g. - /// `resource.region=…`) into `labels` at encode time. Strategy-B - /// hosts (Telegraf / Vector / flat-attribute platforms) flatten - /// resource into `labels`. pub resource_labels: Vec, /// Host-neutral string-string label pairs identifying the /// data-point-level (metric × attrs) series. @@ -168,31 +90,16 @@ pub struct SketchEnvelope { /// Exclusive upper bound of the window the payload summarizes /// (Unix milliseconds). pub window_end_ms: u64, - /// How `payload` is laid out. - pub encoding: Encoding, - /// Proto-encoded sketch state or delta. The bandwidth-invariant - /// blob. + /// ASAPv1 sketch envelope bytes; empty on the estimate path. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub payload: Vec, - /// Determinism contract for cross-language reconstruction. Comes - /// from `asap_sketchlib`'s proto `HashSpec`. - /// - /// Wrapped in `Option` because not every adapter populates it, - /// and the proto type has no `Default` impl chain we can rely on. - /// `serde(skip)` because the proto type does not derive serde — - /// shutdown/restore use cases serialize the rest of the envelope - /// and re-fetch the hash spec from configuration. - #[serde(skip)] - pub hash_spec: Option, /// Metric name the envelope was emitted for. /// /// Today's per-processor `flushWindow` builds the output /// `pmetric::Metric` and sets its `Name()` from a config field; /// once the shims delegate to this runtime, the runtime needs to /// know the metric name per-emission so `Adapter::encode` can - /// stamp it onto the synthesized output Metric. This is an - /// in-process Rust field — NOT a proto wire field. The canonical - /// wire bytes remain `payload`. + /// stamp it onto the synthesized output Metric. pub metric_name: String, /// Total observation count this envelope represents (sum of /// `dp.Count()` for the producing window's input samples). @@ -221,31 +128,6 @@ pub struct SketchEnvelope { mod tests { use super::*; - #[test] - fn sketch_type_names_match_go() { - assert_eq!(SketchType::DDSketch.name(), "DDSketch"); - assert_eq!(SketchType::KLLSketch.name(), "KLLSketch"); - assert_eq!(SketchType::HLLSketch.name(), "HLLSketch"); - assert_eq!(SketchType::CountSketch.name(), "CountSketch"); - assert_eq!(SketchType::CountMinSketch.name(), "CountMinSketch"); - assert_eq!( - SketchType::CountMinSketchWithHeap.name(), - "CountMinSketchWithHeap" - ); - assert_eq!(SketchType::Unspecified.name(), "Unspecified"); - } - - #[test] - fn encoding_names_match_go() { - assert_eq!(Encoding::ProtoFull.name(), "PROTO_FULL"); - assert_eq!(Encoding::ProtoDelta.name(), "PROTO_DELTA"); - assert_eq!(Encoding::Msgpack.name(), "MSGPACK"); - assert_eq!(Encoding::MsgpackDelta.name(), "MSGPACK_DELTA"); - assert_eq!(Encoding::Unspecified.name(), "UNSPECIFIED"); - assert!(Encoding::Msgpack.is_msgpack() && Encoding::MsgpackDelta.is_msgpack()); - assert!(!Encoding::ProtoFull.is_msgpack() && !Encoding::ProtoDelta.is_msgpack()); - } - #[test] fn sketch_envelope_serde_round_trip() { let env = SketchEnvelope { @@ -256,9 +138,7 @@ mod tests { labels: vec![KeyValue::new("path", "/api")], window_start_ms: 1_000, window_end_ms: 2_000, - encoding: Encoding::ProtoFull, payload: vec![1, 2, 3, 4], - hash_spec: None, metric_name: "http_request_duration".into(), count: 100, aggregation_temporality: 1, diff --git a/asap-precompute-rs/src/kll_windows.rs b/asap-precompute-rs/src/kll_windows.rs index 6047f8f..f7517ea 100644 --- a/asap-precompute-rs/src/kll_windows.rs +++ b/asap-precompute-rs/src/kll_windows.rs @@ -8,7 +8,7 @@ use std::sync::Mutex; use std::time::Duration; use crate::config::{AggregationMode, PrecomputeConfig, PrecomputeConfigSet}; -use crate::envelope::{Encoding, SketchEnvelope, SketchType}; +use crate::envelope::{SketchEnvelope, SketchType}; use crate::matchers::series_attrs; use crate::observation::{KeyValue, Observation, ObservationValueKind}; use crate::precompute::{Precompute, PrecomputeError, Sketch, StatsSnapshot}; @@ -43,8 +43,8 @@ pub struct KllWindowStore { impl KllWindowStore { /// Create a store. Retention must be a positive multiple of the pane size. - /// Only KLL/ASAPv1 full sketches are accepted; no delta replay or partial - /// pane splitting is supported. + /// Only KLL sketches are accepted; partial pane splitting is not + /// supported. pub fn new( config: PrecomputeConfig, pane: Duration, @@ -64,11 +64,8 @@ impl KllWindowStore { "retention must be a positive multiple of a nonzero millisecond pane", )); } - if config.sketch_type != SketchType::KLLSketch - || config.encoding != Encoding::Msgpack - || config.delta_transmission - { - return Err(invalid("requires KLL with full ASAPv1/Msgpack encoding")); + if config.sketch_type != SketchType::KLLSketch { + return Err(invalid("requires KLL")); } let k = config.sketch_params.get("k").copied().unwrap_or(200.0); if !k.is_finite() || k < 8.0 || k > i32::MAX as f64 || k.fract() != 0.0 { @@ -154,7 +151,7 @@ impl KllWindowStore { .or_default() .entry(key) .or_insert_with(|| PaneSeries { - sketch: KLLWrapper::new(k, seed).with_wire_encoding(Encoding::Msgpack), + sketch: KLLWrapper::new(k, seed), resource_labels: resource, labels, count: 0, @@ -168,10 +165,7 @@ impl KllWindowStore { /// to the same pane. Overlapping wide sketches and unaligned panes are /// rejected because their observations cannot be assigned to one pane. pub fn merge_pane(&mut self, envelope: &SketchEnvelope) -> Result<(), PrecomputeError> { - if envelope.sketch_type != SketchType::KLLSketch - || envelope.encoding != Encoding::Msgpack - || envelope.payload.is_empty() - { + if envelope.sketch_type != SketchType::KLLSketch || envelope.payload.is_empty() { return Err(invalid( "inbound pane must contain a full ASAPv1 KLL sketch", )); @@ -189,9 +183,8 @@ impl KllWindowStore { "inbound sketch must cover exactly one aligned pane", )); } - let mut sketch = KLLWrapper::new(self.k, self.seed).with_wire_encoding(Encoding::Msgpack); // Decode before changing state: invalid payloads cannot advance retention. - sketch.apply_delta_encoded(&envelope.payload, Encoding::Msgpack)?; + let sketch = KLLWrapper::from_snapshot(&envelope.payload)?; let cutoff = self.closed_until.saturating_sub(self.retention_ms); if envelope.window_start_ms < cutoff { return Err(PrecomputeError::LateData); @@ -206,7 +199,7 @@ impl KllWindowStore { .or_default() .entry(key) .or_insert_with(|| PaneSeries { - sketch: KLLWrapper::new(self.k, self.seed).with_wire_encoding(Encoding::Msgpack), + sketch: KLLWrapper::new(self.k, self.seed), resource_labels: resource, labels, count: 0, @@ -267,8 +260,7 @@ impl KllWindowStore { for (_, pane) in self.panes.range(start_ms..end_ms) { for (key, source) in pane { let entry = merged.entry(key.clone()).or_insert_with(|| PaneSeries { - sketch: KLLWrapper::new(self.k, self.seed) - .with_wire_encoding(Encoding::Msgpack), + sketch: KLLWrapper::new(self.k, self.seed), resource_labels: source.resource_labels.clone(), labels: source.labels.clone(), count: 0, @@ -288,9 +280,7 @@ impl KllWindowStore { labels: entry.labels, window_start_ms: start_ms, window_end_ms: end_ms, - encoding: Encoding::Msgpack, payload: entry.sketch.snapshot()?, - hash_spec: None, metric_name: self.config.metric_name.clone(), count: entry.count, aggregation_temporality: 1, @@ -327,13 +317,10 @@ impl KllWindowStore { } let mut results = Vec::new(); for envelope in self.query(start_ms, end_ms)? { - let mut sketch = - KLLWrapper::new(self.k, self.seed).with_wire_encoding(Encoding::Msgpack); - sketch.apply_delta_encoded(&envelope.payload, Encoding::Msgpack)?; + let sketch = KLLWrapper::from_snapshot(&envelope.payload)?; for point in sketch.estimate(quantiles, 0) { let mut result = envelope.clone(); result.payload.clear(); - result.encoding = Encoding::Unspecified; result.value = point.value; result.labels.extend(point.labels); results.push(result); diff --git a/asap-precompute-rs/src/lib.rs b/asap-precompute-rs/src/lib.rs index edff0a0..2a931a9 100644 --- a/asap-precompute-rs/src/lib.rs +++ b/asap-precompute-rs/src/lib.rs @@ -1,14 +1,10 @@ //! `asap-precompute-rs` is the host-neutral edge precompute runtime. //! -//! This crate owns the windowing, snapshot caching, and delta encoding -//! runtime for the Rust **edge** runtime. Future Rust-based -//! edge agents (Vector adapter, OTAP-Rust, Arrow-backed shims) -//! consume this crate. -//! -//! Per-platform Adapter implementations (the Layer-4 shims) translate -//! their host's native event into [`Observation`], hand it to a -//! [`Precompute`], and translate the runtime's emitted -//! [`SketchEnvelope`] back to the host's native event. +//! This crate owns the windowing runtime for the Rust **edge** runtime. +//! A host translates its native event into [`Observation`], hands it to +//! a [`Precompute`], and translates each emitted [`SketchEnvelope`] +//! (an ASAPv1 sketch payload from `asap_sketchlib` plus metadata) back +//! to its own format; the OTAP processor in `otap` is the bundled host. #![warn(missing_docs)] // The sketch code intentionally guards against NaN with the @@ -17,9 +13,7 @@ // rewrite behavior-sensitive comparisons. #![allow(clippy::neg_cmp_op_on_partial_ord)] -pub mod adapter; pub mod config; -pub mod control_channel; pub mod envelope; pub mod kll_windows; pub mod matchers; @@ -28,14 +22,13 @@ pub mod observation; pub mod otap; pub mod precompute; pub mod sketches; -pub mod snapshot_cache; pub mod window; pub use config::{ AggId, AggregationMode, OnOverflow, PrecomputeConfig, PrecomputeConfigSet, SketchParams, WindowSpec, }; -pub use envelope::{Encoding, SketchEnvelope, SketchType}; +pub use envelope::{SketchEnvelope, SketchType}; pub use matchers::{LabelMatcher, MatchOp}; pub use observation::{KeyValue, Observation, ObservationValue, ObservationValueKind}; pub use precompute::{ diff --git a/asap-precompute-rs/src/observation.rs b/asap-precompute-rs/src/observation.rs index 79ea7cb..8dab77a 100644 --- a/asap-precompute-rs/src/observation.rs +++ b/asap-precompute-rs/src/observation.rs @@ -12,7 +12,7 @@ use crate::envelope::SketchEnvelope; /// /// Labels are `Vec` rather than a host-specific attribute map /// because attribute maps are host-specific (e.g. `pcommon::Map` is -/// OTel-only). Telegraf, Vector, and OTAP adapters carry the same +/// OTel-only). Host adapters carry the same /// struct without pulling host crates in. #[derive(Clone, Debug, Default, PartialEq, serde::Serialize, serde::Deserialize)] pub struct Observation { @@ -24,7 +24,7 @@ pub struct Observation { pub metric: String, /// `(key, value)` pairs from the host's resource scope (e.g. /// `pmetric::ResourceMetrics::Resource()` attributes on OTel, - /// analogous fields on Telegraf / Vector / OTAP). Today's OTel + /// analogous fields on other hosts). Today's OTel /// processors include resource attrs in the series key, so the /// runtime must too — preserved here as a separate slice so /// adapters don't have to flatten. @@ -185,7 +185,7 @@ impl ObservationValue { #[cfg(test)] mod tests { use super::*; - use crate::envelope::{Encoding, SketchType}; + use crate::envelope::SketchType; #[test] fn observation_constructs_with_all_four_value_kinds() { @@ -224,7 +224,6 @@ mod tests { let env = SketchEnvelope { schema_version: 1, sketch_type: SketchType::DDSketch, - encoding: Encoding::ProtoFull, ..Default::default() }; let o_env = Observation::new( @@ -237,12 +236,4 @@ mod tests { assert_eq!(o_env.value.kind, ObservationValueKind::Envelope); assert!(o_env.value.envelope.is_some()); } - - #[test] - fn kind_names_match_go_reference() { - assert_eq!(ObservationValueKind::Float.name(), "Float"); - assert_eq!(ObservationValueKind::Hash.name(), "Hash"); - assert_eq!(ObservationValueKind::Bytes.name(), "Bytes"); - assert_eq!(ObservationValueKind::Envelope.name(), "Envelope"); - } } diff --git a/asap-precompute-rs/src/otap/codec.rs b/asap-precompute-rs/src/otap/codec.rs index 7d8e2d0..6deafcb 100644 --- a/asap-precompute-rs/src/otap/codec.rs +++ b/asap-precompute-rs/src/otap/codec.rs @@ -3,26 +3,8 @@ //! Direct `SketchEnvelope <-> OtapPdata` binding — the real `OtapPdata` //! is built straight from a `&[SketchEnvelope]` slice, and a real -//! `OtapPdata` is decoded straight into `Vec`. No -//! intermediate `RecordBatch` or `OtapMetricRecords` two-batch family -//! in between. -//! -//! # Why direct, not via `encode_batch`/`decode_batch`/`records::{flatten,lift}` -//! -//! Those three exist as a *general*, real-OTAP-free carrier format — -//! useful for other Strategy-B adapters (Telegraf, Vector) that might -//! one day consume the same flat shape, and for testing this codec -//! without any OTAP dependency at all. But this repo only ever ships -//! to OTAP, and no second adapter has actually shown up. Going through -//! them here would mean: `SketchEnvelope` -> flat `RecordBatch` -//! (`encode_batch`) -> two-batch `OtapMetricRecords` (`lift`) -> real -//! `OtapPdata` (this module) — three representations for what's really -//! one job. This module skips straight to the last step, implementing -//! OTAP's own `MetricsView` trait directly over `&[SketchEnvelope]`. -//! -//! `encode_batch`/`decode_batch`/`records::{flatten,lift}` still exist as a -//! standalone Arrow adapter surface, but are not an additional node-to-node -//! transport. +//! `OtapPdata` is decoded straight into `Vec`, by +//! implementing OTAP's own `MetricsView` over `&[SketchEnvelope]`. //! //! # Native protocol mapping //! @@ -84,41 +66,17 @@ use otel_arrow_dfe_pdata_views::views::metrics::{ use otel_arrow_dfe_pdata_views::views::resource::ResourceView; use crate::config::{sketch_size_string, SketchParams}; -use crate::envelope::{Encoding, SketchEnvelope}; +use crate::envelope::{SketchEnvelope, SketchType}; use crate::observation::{KeyValue, Observation, ObservationValue}; -use super::decode::{parse_encoding, parse_sketch_type}; -use super::schema::{ - OTAP_ATTR_AGG_ID as ATTR_AGG_ID, OTAP_ATTR_ENCODING as ATTR_ENCODING, - OTAP_ATTR_ENVELOPE as ATTR_ENVELOPE, OTAP_ATTR_HASH_FUNCTION as ATTR_HASH_FUNCTION, - OTAP_ATTR_HASH_SEED as ATTR_HASH_SEED, OTAP_ATTR_SCHEMA_VERSION as ATTR_SCHEMA_VERSION, - OTAP_ATTR_SERIES_ID as ATTR_SERIES_ID, OTAP_ATTR_SKETCH_SIZE as ATTR_SKETCH_SIZE, - OTAP_ATTR_SKETCH_TYPE as ATTR_SKETCH_TYPE, OTAP_ATTR_WINDOW_END_MS as ATTR_WINDOW_END_MS, - OTAP_ATTR_WINDOW_START_MS as ATTR_WINDOW_START_MS, +use crate::sketches::{ + CMSHeapWrapper, CMSWrapper, CountSketchWrapper, DDSketchWrapper, HLLWrapper, KLLWrapper, }; -fn resolve_hash_seed( - sketch_type: crate::envelope::SketchType, - spec: Option<&asap_sketchlib::proto::sketchlib::HashSpec>, -) -> (Option, Option) { - let Some(spec) = spec else { - return (None, None); - }; - let seed = if matches!( - sketch_type, - crate::envelope::SketchType::CountSketch | crate::envelope::SketchType::CountMinSketch - ) { - spec.seed_list.first().copied() - } else { - spec.seed_list - .get(spec.canonical_seed_index as usize) - .copied() - }; - let function = asap_sketchlib::proto::sketchlib::HashAlgorithm::try_from(spec.algorithm) - .ok() - .map(|algorithm| algorithm.as_str_name().to_owned()); - (seed, function) -} +use super::schema::{ + ATTR_AGG_ID, ATTR_ENVELOPE, ATTR_SCHEMA_VERSION, ATTR_SERIES_ID, ATTR_SKETCH_SIZE, + ATTR_SKETCH_TYPE, ATTR_WINDOW_END_MS, ATTR_WINDOW_START_MS, +}; /// Failure modes for [`encode_envelopes_to_pdata`] / [`decode_pdata_to_observations`]. #[derive(Debug, Error)] @@ -185,7 +143,7 @@ impl OtapSketchEncoder { /// Encodes one flush while retaining series identity across later flushes. pub fn encode(&mut self, envelopes: &[SketchEnvelope]) -> Result { for env in envelopes { - validate_self_describing_payload(env.sketch_type, env.encoding, &env.payload) + validate_self_describing_payload(env.sketch_type, &env.payload) .map_err(CodecError::Encode)?; } let series_ids = envelopes @@ -219,32 +177,25 @@ impl OtapSketchEncoder { } } -/// Enforces the native OTAP carrier contract for sketch records. -/// -/// `sketch.envelope` contains the canonical sketchlib ASAPv1 envelope: -/// magic, version, kind ID, length-prefixed metadata, and payload. -fn validate_self_describing_payload( - sketch_type: crate::envelope::SketchType, - encoding: Encoding, - payload: &[u8], -) -> Result<(), String> { +/// Enforces the native OTAP carrier contract for sketch records: a +/// non-empty `sketch.envelope` must decode as the sketchlib ASAPv1 envelope +/// of the declared `sketch.type`. +fn validate_self_describing_payload(sketch_type: SketchType, payload: &[u8]) -> Result<(), String> { if payload.is_empty() { return Ok(()); } - if encoding != Encoding::Msgpack { - return Err(format!( - "sketch.envelope requires the self-describing ASAPv1 MessagePack format, got {}", - encoding.name() - )); - } - if sketch_type == crate::envelope::SketchType::CountMinSketchWithHeap { - return crate::sketches::CMSHeapWrapper::from_snapshot(payload) - .map(|_| ()) - .map_err(|error| format!("invalid self-describing CMSHeap sketch.envelope: {error}")); - } - asap_sketchlib::sketches::KLL::::deserialize_from_bytes(payload) - .map(|_| ()) - .map_err(|error| format!("invalid self-describing KLL sketch.envelope: {error}")) + let decoded = match sketch_type { + SketchType::DDSketch => DDSketchWrapper::from_snapshot(payload).map(drop), + SketchType::KLLSketch => KLLWrapper::from_snapshot(payload).map(drop), + SketchType::HLLSketch => HLLWrapper::from_snapshot(payload).map(drop), + SketchType::CountSketch => CountSketchWrapper::from_snapshot(payload).map(drop), + SketchType::CountMinSketch => CMSWrapper::from_snapshot(payload).map(drop), + SketchType::CountMinSketchWithHeap => CMSHeapWrapper::from_snapshot(payload).map(drop), + SketchType::Unspecified => { + return Err("sketch.envelope requires a sketch.type".into()); + } + }; + decoded.map_err(|error| format!("invalid {} sketch.envelope: {error}", sketch_type.name())) } fn append_labels_to_key(key: &mut String, labels: &[KeyValue]) { @@ -264,9 +215,7 @@ fn append_labels_to_key(key: &mut String, labels: &[KeyValue]) { fn same_resource(left: &SketchEnvelope, right: &SketchEnvelope) -> bool { if left.agg_id != right.agg_id || left.sketch_type != right.sketch_type - || left.encoding != right.encoding || left.schema_version != right.schema_version - || left.hash_spec != right.hash_spec { return false; } @@ -289,8 +238,6 @@ struct AsapMetricsView<'a> { struct AsapResourceGroup { representative: usize, sketch_size: Option, - hash_seed: Option, - hash_function: Option, scopes: Vec, } @@ -311,14 +258,10 @@ impl<'a> AsapMetricsView<'a> { .iter() .position(|group| same_resource(&envelopes[group.representative], env)) .unwrap_or_else(|| { - let (hash_seed, hash_function) = - resolve_hash_seed(env.sketch_type, env.hash_spec.as_ref()); resources.push(AsapResourceGroup { representative: row, sketch_size: sketch_params .and_then(|params| sketch_size_string(env.sketch_type, params)), - hash_seed, - hash_function, scopes: Vec::new(), }); resources.len() - 1 @@ -343,10 +286,8 @@ impl<'a> AsapMetricsView<'a> { } /// One row's worth of attributes, built directly from the - /// envelope's own fields — no intermediate Arrow columns. The - /// `_asap_*` Strategy-B carriers only appear when this row is - /// actually an envelope-payload row (see this module's doc, "A - /// real bug this rewrite fixes"). + /// envelope's own fields. `sketch.envelope` only appears when this + /// row actually carries a sketch payload. fn attributes_for_row(&self, row: usize) -> Vec> { let env = &self.envelopes[row]; let mut attrs = Vec::with_capacity(3); @@ -800,7 +741,7 @@ impl<'a> AnyValueView<'a> for AsapAnyValueView<'a> { fn as_int64(&self) -> Option { match self.0 { // Lossy above i64::MAX, which ASAP never actually - // produces here (the only `_asap_*` int-typed attributes + // produces here (the only `sketch.*` int-typed attributes // are small counters/timestamps well under that bound). AsapAnyValue::Int(v) => Some(*v as i64), _ => None, @@ -880,10 +821,6 @@ impl<'v, 'a> ResourceView for AsapResourceView<'v, 'a> { key: ATTR_SKETCH_TYPE, value: AsapAnyValue::Str(env.sketch_type.name()), }, - AsapAttribute { - key: ATTR_ENCODING, - value: AsapAnyValue::Str(env.encoding.name()), - }, AsapAttribute { key: ATTR_SCHEMA_VERSION, value: AsapAnyValue::Int(u64::from(env.schema_version)), @@ -895,18 +832,6 @@ impl<'v, 'a> ResourceView for AsapResourceView<'v, 'a> { value: AsapAnyValue::Str(value), }); } - if let Some(value) = group.hash_seed { - attrs.push(AsapAttribute { - key: ATTR_HASH_SEED, - value: AsapAnyValue::Int(value), - }); - } - if let Some(value) = group.hash_function.as_deref() { - attrs.push(AsapAttribute { - key: ATTR_HASH_FUNCTION, - value: AsapAnyValue::Str(value), - }); - } attrs.into_iter() } @@ -1329,9 +1254,9 @@ pub fn describe_pdata_protocol(pdata: &OtapPdata) -> Result if let Some(envelope) = observation.value.envelope { writeln!( output, - " +-- Metric {}\n |-- labels {{{}}}\n |-- value self-describing sketch envelope ({} bytes)\n |-- schema agg_id={} {} {} v{}\n +-- window [{}, {})", + " +-- Metric {}\n |-- labels {{{}}}\n |-- value self-describing sketch envelope ({} bytes)\n |-- schema agg_id={} {} v{}\n +-- window [{}, {})", observation.metric, labels, envelope.payload.len(), envelope.agg_id, - envelope.sketch_type.name(), envelope.encoding.name(), envelope.schema_version, + envelope.sketch_type.name(), envelope.schema_version, envelope.window_start_ms, envelope.window_end_ms ) .expect("write String"); @@ -1426,12 +1351,7 @@ pub fn decode_pdata_to_observations(pdata: OtapPdata) -> Result { schema.schema_version = val.as_int64().filter(|v| *v >= 0).map(|v| v as u32) } - ATTR_ENCODING => { - schema.encoding = val - .as_string() - .map(|s| String::from_utf8_lossy(s).into_owned()) - } - ATTR_SKETCH_SIZE | ATTR_HASH_SEED | ATTR_HASH_FUNCTION => {} + ATTR_SKETCH_SIZE => {} _ => { if let Some(s) = val.as_string() { resource_labels @@ -1525,12 +1445,20 @@ struct DecodeAccumulator { observations: Vec, } +/// Parses an optional `sketch.type` value; absent means `Unspecified`. +fn parse_sketch_type(raw: Option<&str>) -> Result { + match raw { + None => Ok(SketchType::Unspecified), + Some(raw) => SketchType::from_name(raw) + .ok_or_else(|| CodecError::Decode(format!("unknown sketch type {raw:?}"))), + } +} + #[derive(Default)] struct DecodeSchema { sketch_type: Option, agg_id: Option, schema_version: Option, - encoding: Option, } impl DecodeAccumulator { @@ -1574,22 +1502,8 @@ impl DecodeAccumulator { .ok_or_else(|| { CodecError::Decode("summary sketch is missing sketch.envelope".into()) })?; - let sketch_type = inherited_schema - .sketch_type - .as_deref() - .map(|value| parse_sketch_type(0, value)) - .transpose() - .map_err(|error| CodecError::Decode(error.to_string()))? - .unwrap_or(crate::envelope::SketchType::Unspecified); - let encoding = inherited_schema - .encoding - .as_deref() - .map(|value| parse_encoding(0, value)) - .transpose() - .map_err(|error| CodecError::Decode(error.to_string()))? - .unwrap_or(crate::envelope::Encoding::Unspecified); - validate_self_describing_payload(sketch_type, encoding, &payload) - .map_err(CodecError::Decode)?; + let sketch_type = parse_sketch_type(inherited_schema.sketch_type.as_deref())?; + validate_self_describing_payload(sketch_type, &payload).map_err(CodecError::Decode)?; let timestamp_ms = dp.time_unix_nano() / 1_000_000; let envelope = SketchEnvelope { schema_version: inherited_schema.schema_version.unwrap_or(0), @@ -1599,9 +1513,7 @@ impl DecodeAccumulator { labels: labels.clone(), window_start_ms: window_start_ms.unwrap_or(dp.start_time_unix_nano() / 1_000_000), window_end_ms: window_end_ms.unwrap_or(timestamp_ms), - encoding, payload, - hash_spec: None, metric_name: metric_name.to_string(), count: dp.count(), aggregation_temporality: 0, @@ -1618,14 +1530,9 @@ impl DecodeAccumulator { } /// Builds and appends one `Observation` from a data point — a - /// no-op if the data point carries no value. - /// - /// Mirrors `decode.rs`'s `decode_batch` row-decode logic exactly - /// (same `_asap_*` key recognition, same envelope-vs-scalar - /// routing, same hard error on an unrecognized - /// `_asap_sketch_type`/`_asap_encoding`), just reading attributes - /// directly off the real `NumberDataPointView` instead of off - /// Arrow columns. + /// no-op if the data point carries no value. A non-empty + /// `sketch.envelope` attribute routes it as a sketch envelope, validated + /// like a Summary carrier; an unrecognized `sketch.type` is an error. fn push_data_point( &mut self, dp: D, @@ -1649,7 +1556,6 @@ impl DecodeAccumulator { let mut schema_version = inherited_schema.schema_version; let mut window_start_ms: Option = None; let mut window_end_ms: Option = None; - let mut encoding_raw = inherited_schema.encoding.clone(); let mut labels: Vec = series_labels.to_vec(); for attr in dp.attributes() { @@ -1672,17 +1578,10 @@ impl DecodeAccumulator { ATTR_WINDOW_END_MS => { window_end_ms = val.as_int64().filter(|v| *v >= 0).map(|v| v as u64); } - ATTR_ENCODING => { - encoding_raw = val - .as_string() - .map(|s| String::from_utf8_lossy(s).into_owned()); - } _ => { // Non-reserved: treated as a per-row label. Only - // string-typed values are representable as a - // label (matches decode.rs's Utf8-only label - // column behavior) — a non-string, non-reserved - // attribute is dropped, same as the legacy path. + // string-typed values are representable as a label; + // a non-string, non-reserved attribute is dropped. if let Some(s) = val.as_string() { labels.push(KeyValue::new(key, String::from_utf8_lossy(s).into_owned())); } @@ -1690,26 +1589,13 @@ impl DecodeAccumulator { } } - // `.filter(|b| !b.is_empty())`: defensive guard against the - // same empty-but-present-Binary-cell class of bug this - // module's encoder fixes on the way out (see module doc, "A - // real bug this rewrite fixes") — an empty `_asap_envelope` - // from some other producer still routes to the scalar path - // here rather than an empty-payload envelope. + // An empty `sketch.envelope` from some other producer routes to + // the scalar path rather than an empty-payload envelope. let observation_value = match envelope_bytes.filter(|b| !b.is_empty()) { Some(payload) => { - let sketch_type = match sketch_type_raw { - Some(s) => { - parse_sketch_type(0, &s).map_err(|e| CodecError::Decode(e.to_string()))? - } - None => crate::envelope::SketchType::Unspecified, - }; - let encoding = match encoding_raw { - Some(s) => { - parse_encoding(0, &s).map_err(|e| CodecError::Decode(e.to_string()))? - } - None => crate::envelope::Encoding::Unspecified, - }; + let sketch_type = parse_sketch_type(sketch_type_raw.as_deref())?; + validate_self_describing_payload(sketch_type, &payload) + .map_err(CodecError::Decode)?; ObservationValue::envelope(SketchEnvelope { schema_version: schema_version.unwrap_or(0), sketch_type, @@ -1718,9 +1604,7 @@ impl DecodeAccumulator { labels: labels.clone(), window_start_ms: window_start_ms.unwrap_or(0), window_end_ms: window_end_ms.unwrap_or(0), - encoding, payload, - hash_spec: None, metric_name: metric_name.to_string(), count: 0, aggregation_temporality: 0, @@ -1744,7 +1628,6 @@ impl DecodeAccumulator { #[cfg(test)] mod tests { use super::*; - use crate::envelope::SketchType; use crate::observation::{KeyValue as Kv, ObservationValueKind}; use crate::precompute::Sketch; use crate::sketches::KLLWrapper; @@ -1763,9 +1646,7 @@ mod tests { labels: vec![Kv::new(attr_key, attr_value)], window_start_ms: 1_000, window_end_ms: 2_000, - encoding: Encoding::Unspecified, payload: Vec::new(), - hash_spec: None, metric_name: metric_name.to_string(), count: 0, aggregation_temporality: 0, @@ -1866,9 +1747,8 @@ mod tests { fn summary_carrier_round_trips_a_self_describing_sketch_envelope() { let mut env = one_envelope("sketch_stream", 0.0, "region", "us-east"); env.sketch_type = SketchType::KLLSketch; - env.encoding = Encoding::Msgpack; env.agg_id = 7; - let mut sketch = KLLWrapper::new(200, Some(42)).with_wire_encoding(Encoding::Msgpack); + let mut sketch = KLLWrapper::new(200, Some(42)); sketch.update(42.0); env.payload = sketch .snapshot() @@ -1885,7 +1765,6 @@ mod tests { let decoded_env = obs.value.envelope.as_ref().expect("envelope"); assert_eq!(decoded_env.payload, expected_payload); assert_eq!(decoded_env.sketch_type, SketchType::KLLSketch); - assert_eq!(decoded_env.encoding, Encoding::Msgpack); assert_eq!(decoded_env.agg_id, 7); assert_eq!(decoded_env.window_start_ms, 1_000); assert_eq!(decoded_env.window_end_ms, 2_000); @@ -1896,7 +1775,6 @@ mod tests { use crate::sketches::CMSHeapWrapper; let mut env = one_envelope("topk_stream", 0.0, "region", "us-east"); env.sketch_type = SketchType::CountMinSketchWithHeap; - env.encoding = Encoding::Msgpack; let mut sketch = CMSHeapWrapper::new(4, 64, 3); sketch.update(b"checkout", 5); env.payload = sketch.snapshot().expect("CMSHeap ASAPv1 envelope"); @@ -1911,30 +1789,54 @@ mod tests { assert!(encode_envelopes_to_pdata(std::slice::from_ref(&env)).is_err()); } + #[test] + fn summary_carrier_round_trips_every_sketch_type() { + use crate::sketches::{CMSWrapper, CountSketchWrapper, DDSketchWrapper, HLLWrapper}; + let mut dd = DDSketchWrapper::new(0.01); + dd.update(3.0); + let mut hll = HLLWrapper::new(); + hll.update(b"user-1"); + let mut cs = CountSketchWrapper::new(4, 64); + cs.update(b"key", 2); + let mut cms = CMSWrapper::new(4, 64); + cms.update(b"key", 2); + let sketches: [(SketchType, &dyn Sketch); 4] = [ + (SketchType::DDSketch, &dd), + (SketchType::HLLSketch, &hll), + (SketchType::CountSketch, &cs), + (SketchType::CountMinSketch, &cms), + ]; + for (sketch_type, sketch) in sketches { + let mut env = one_envelope("sketch_stream", 0.0, "region", "us-east"); + env.sketch_type = sketch_type; + env.payload = sketch.snapshot().unwrap(); + let pdata = encode_envelopes_to_pdata(std::slice::from_ref(&env)).expect("encode"); + let outcome = decode_pdata_to_observations(pdata).expect("decode"); + let decoded = outcome.observations[0].value.envelope.as_ref().unwrap(); + assert_eq!(decoded.sketch_type, sketch_type); + assert_eq!(decoded.payload, env.payload); + } + } + #[test] fn asapv1_rejects_non_self_describing_envelope_bytes() { let mut env = one_envelope("sketch_stream", 0.0, "region", "us-east"); env.sketch_type = SketchType::KLLSketch; - env.encoding = Encoding::Msgpack; env.payload = vec![0xde, 0xad, 0xbe, 0xef]; let error = encode_envelopes_to_pdata(&[env]).expect_err("invalid ASAPv1 must fail"); assert!(error .to_string() - .contains("self-describing KLL sketch.envelope")); + .contains("invalid KLLSketch sketch.envelope")); } #[test] - fn summary_carrier_rejects_non_self_describing_encoding() { + fn sketch_payload_without_sketch_type_is_rejected() { + let mut sketch = KLLWrapper::new(200, Some(42)); + sketch.update(1.0); let mut env = one_envelope("sketch_stream", 0.0, "region", "us-east"); - env.sketch_type = SketchType::KLLSketch; - env.encoding = Encoding::ProtoFull; - env.payload = vec![0x0a, 0x00]; - - let error = encode_envelopes_to_pdata(&[env]).expect_err("protobuf must fail"); - assert!(error - .to_string() - .contains("requires the self-describing ASAPv1 MessagePack format")); + env.payload = sketch.snapshot().unwrap(); + assert!(encode_envelopes_to_pdata(&[env]).is_err()); } /// The bug this rewrite fixes: an estimate-mode envelope (empty @@ -2013,7 +1915,7 @@ mod tests { let resource_attrs = arrow_records .get(ArrowPayloadType::ResourceAttrs) .expect("SCHEMA uses ResourceAttrs"); - assert_eq!(resource_attrs.num_rows(), 4); + assert_eq!(resource_attrs.num_rows(), 3); let scope_attrs = arrow_records .get(ArrowPayloadType::ScopeAttrs) diff --git a/asap-precompute-rs/src/otap/config.rs b/asap-precompute-rs/src/otap/config.rs index 86deae0..a48497a 100644 --- a/asap-precompute-rs/src/otap/config.rs +++ b/asap-precompute-rs/src/otap/config.rs @@ -1,4 +1,4 @@ -//! ASAP processor configuration and five-sketch `sketch_type` dispatch. +//! ASAP processor configuration and `sketch_type` dispatch. //! //! A single plugin parameterized by `sketch_type` with a //! dispatch table that maps the user-facing string to: @@ -17,11 +17,12 @@ use std::time::Duration; use crate::config::{PrecomputeConfig, SketchParams, WindowSpec}; -use crate::envelope::{Encoding, SketchType}; +use crate::envelope::SketchType; use crate::precompute::{BoxedObserver, SketchFactory, SketchObserver}; use crate::sketches::{ - CMSObserver, CMSWrapper, CountSketchObserver, CountSketchWrapper, DDSketchObserver, - DDSketchWrapper, HLLObserver, HLLWrapper, KLLObserver, KLLWrapper, + CMSHeapObserver, CMSHeapWrapper, CMSObserver, CMSWrapper, CountSketchObserver, + CountSketchWrapper, DDSketchObserver, DDSketchWrapper, HLLObserver, HLLWrapper, KLLObserver, + KLLWrapper, }; // -- Sketch-type spellings accepted in the plugin's TOML @@ -38,6 +39,8 @@ pub const SKETCH_TYPE_HLL: &str = "hll"; pub const SKETCH_TYPE_COUNTSKETCH: &str = "countsketch"; /// `sketch_type = "countminsketch"`. pub const SKETCH_TYPE_COUNTMINSKETCH: &str = "countminsketch"; +/// `sketch_type = "countminsketch_heap"`. +pub const SKETCH_TYPE_COUNTMINSKETCH_HEAP: &str = "countminsketch_heap"; /// Dispatch error when [`PluginConfig`] cannot be turned into a /// runtime config + sketch factory + observer triple. @@ -49,8 +52,7 @@ pub enum ConfigError { /// Specific validation failure. reason: String, }, - /// User-facing `sketch_type` string didn't match any of the five - /// supported sketches. + /// User-facing `sketch_type` string didn't match a supported sketch. #[error("asap_sketches: unsupported sketch_type {value:?} (want one of: {valid})")] UnknownSketchType { /// Raw value from config. @@ -66,18 +68,6 @@ pub enum ConfigError { /// Raw value from config (in milliseconds, post-parse). value: u128, }, - - /// Sketch payload transport is restricted to codecs that emit the - /// canonical self-describing ASAPv1 envelope. - #[error( - "asap_sketches: sketch transport requires kll with msgpack/ASAPv1; got {sketch_type:?} with {encoding} (set transmit_sketch=false for scalar estimates)" - )] - UnsupportedSketchTransport { - /// Configured sketch implementation. - sketch_type: String, - /// Configured wire encoding. - encoding: &'static str, - }, } /// Plugin-level config block. Exposes the `sketch_type` knob plus the @@ -90,7 +80,8 @@ pub enum ConfigError { #[derive(Clone, Debug)] pub struct PluginConfig { /// One of `"ddsketch"` / `"kll"` / `"hll"` / `"countsketch"` / - /// `"countminsketch"` (case-insensitive). Mandatory. + /// `"countminsketch"` / `"countminsketch_heap"` (case-insensitive). + /// Mandatory. pub sketch_type: String, /// Window rotation period. Mandatory. pub window_size: Duration, @@ -114,19 +105,11 @@ pub struct PluginConfig { /// Whether to append `sample_count` / `window_duration_seconds` /// to every emitted envelope's labels. pub emit_window_stats: bool, - /// Outbound wire format for emitted envelopes. Sketch transport requires - /// KLL with `Msgpack`/`MsgpackDelta`, which emits the canonical ASAPv1 - /// self-describing format. Other combinations remain available only when - /// `transmit_sketch=false` emits scalar estimates. - pub encoding: Encoding, - /// Whether to delta-encode emitted state against the cached outbound - /// snapshot (per-window against-empty). Combined with a msgpack - /// `encoding`, this emits `MsgpackDelta` frames. - pub delta_transmission: bool, /// When `false`, emit estimated scalar results instead of sketch bytes /// (see [`crate::config::PrecomputeConfig::transmit_sketch`]): one Gauge /// per [`Self::quantiles`] entry for DDSketch / KLL, a cardinality Gauge - /// for HLL. Defaults to `true` (transmit the sketch). + /// for HLL, one Gauge per heavy hitter for CMS with heap. Defaults to + /// `true` (transmit the ASAPv1 sketch). pub transmit_sketch: bool, /// Quantiles emitted per series when `transmit_sketch = false` and the /// sketch is a quantile sketch (DDSketch / KLL). Ignored otherwise. @@ -146,8 +129,6 @@ impl Default for PluginConfig { omit_resource_attrs: false, global_aggregation: false, emit_window_stats: false, - encoding: Encoding::Msgpack, - delta_transmission: false, transmit_sketch: true, quantiles: Vec::new(), } @@ -188,14 +169,6 @@ pub fn resolve(config: &PluginConfig) -> Result<(PrecomputeConfig, SketchDispatc }); } let dispatch = build_dispatch(config)?; - if config.transmit_sketch - && (dispatch.sketch_type != SketchType::KLLSketch || !config.encoding.is_msgpack()) - { - return Err(ConfigError::UnsupportedSketchTransport { - sketch_type: config.sketch_type.clone(), - encoding: config.encoding.name(), - }); - } let sliding = !config.window_slide.is_zero(); let pcfg = PrecomputeConfig { agg_id: config.agg_id, @@ -213,9 +186,6 @@ pub fn resolve(config: &PluginConfig) -> Result<(PrecomputeConfig, SketchDispatc matchers: Vec::new(), aggregate_by: Vec::new(), transmit_sketch: config.transmit_sketch, - delta_transmission: config.delta_transmission, - delta_threshold: 0, - encoding: config.encoding, quantiles: config.quantiles.clone(), sketch_params: config.sketch_params.clone(), max_series: 0, @@ -241,10 +211,6 @@ pub fn resolve(config: &PluginConfig) -> Result<(PrecomputeConfig, SketchDispatc fn build_dispatch(config: &PluginConfig) -> Result { let normalized = config.sketch_type.to_ascii_lowercase(); let params = &config.sketch_params; - // Baked into each msgpack-capable wrapper so its `snapshot` / - // `compute_delta_against` / `delta_against_empty_base` pick the right - // wire codec (KLL ignores it — proto-only). - let encoding = config.encoding; match normalized.as_str() { SKETCH_TYPE_DDSKETCH => { let alpha = positive_in_unit_interval( @@ -253,9 +219,7 @@ fn build_dispatch(config: &PluginConfig) -> Result ); Ok(SketchDispatch { sketch_type: SketchType::DDSketch, - factory: Box::new(move || { - Box::new(DDSketchWrapper::new(alpha).with_wire_encoding(encoding)) - }), + factory: Box::new(move || Box::new(DDSketchWrapper::new(alpha))), observer: boxed_observer(DDSketchObserver), }) } @@ -269,28 +233,15 @@ fn build_dispatch(config: &PluginConfig) -> Result }; Ok(SketchDispatch { sketch_type: SketchType::KLLSketch, - factory: Box::new(move || { - Box::new(KLLWrapper::new(k as i32, seed).with_wire_encoding(encoding)) - }), + factory: Box::new(move || Box::new(KLLWrapper::new(k as i32, seed))), observer: boxed_observer(KLLObserver), }) } - SKETCH_TYPE_HLL => { - let precision = clamp_positive_int( - crate::config::sketch_param_get(params, "precision", 12.0), - 12, - ); - Ok(SketchDispatch { - sketch_type: SketchType::HLLSketch, - factory: Box::new(move || { - Box::new( - HLLWrapper::new(asap_sketchlib::HllVariant::Regular, precision as u32) - .with_wire_encoding(encoding), - ) - }), - observer: boxed_observer(HLLObserver), - }) - } + SKETCH_TYPE_HLL => Ok(SketchDispatch { + sketch_type: SketchType::HLLSketch, + factory: Box::new(|| Box::new(HLLWrapper::new())), + observer: boxed_observer(HLLObserver), + }), SKETCH_TYPE_COUNTSKETCH => { let depth = clamp_positive_int(crate::config::sketch_param_get(params, "depth", 4.0), 4); @@ -305,10 +256,7 @@ fn build_dispatch(config: &PluginConfig) -> Result Ok(SketchDispatch { sketch_type: SketchType::CountSketch, factory: Box::new(move || { - Box::new( - CountSketchWrapper::new(depth as usize, width as usize) - .with_wire_encoding(encoding), - ) + Box::new(CountSketchWrapper::new(depth as usize, width as usize)) }), observer: boxed_observer(CountSketchObserver { default_key }), }) @@ -323,17 +271,33 @@ fn build_dispatch(config: &PluginConfig) -> Result Ok(SketchDispatch { sketch_type: SketchType::CountMinSketch, factory: Box::new(move || { - Box::new( - CMSWrapper::new(depth as usize, width as usize) - .with_wire_encoding(encoding), - ) + Box::new(CMSWrapper::new(depth as usize, width as usize)) }), observer: boxed_observer(CMSObserver), }) } + SKETCH_TYPE_COUNTMINSKETCH_HEAP => { + let rows = clamp_positive_int(crate::config::sketch_param_get(params, "rows", 4.0), 4); + let columns = clamp_positive_int( + crate::config::sketch_param_get(params, "columns", 2048.0), + 2048, + ); + let k = clamp_positive_int(crate::config::sketch_param_get(params, "k", 10.0), 10); + Ok(SketchDispatch { + sketch_type: SketchType::CountMinSketchWithHeap, + factory: Box::new(move || { + Box::new(CMSHeapWrapper::new( + rows as usize, + columns as usize, + k as usize, + )) + }), + observer: boxed_observer(CMSHeapObserver), + }) + } _ => Err(ConfigError::UnknownSketchType { value: config.sketch_type.clone(), - valid: "ddsketch, kll, hll, countsketch, countminsketch", + valid: "ddsketch, kll, hll, countsketch, countminsketch, countminsketch_heap", }), } } @@ -373,45 +337,42 @@ mod tests { } } - /// The out-of-box sketch transport must already satisfy the ASAPv1 - /// self-describing carrier contract. + /// Every sketch type transmits its ASAPv1 sketch; the factory's sketch + /// snapshots to a payload the processor accepts. #[test] - fn default_transport_is_self_describing_kll() { - let cfg = PluginConfig::default(); - assert_eq!(cfg.sketch_type, SKETCH_TYPE_KLL); - assert_eq!(cfg.encoding, Encoding::Msgpack); - assert!(resolve(&cfg).is_ok()); - } - - /// Until a sketchlib ASAPv1 codec exists for a sketch type, transport of - /// its private proto/legacy-msgpack state must fail during configuration. - #[test] - fn sketch_transport_rejects_non_asapv1_combinations() { - for (sketch_type, encoding) in [ - (SKETCH_TYPE_DDSKETCH, Encoding::Msgpack), - (SKETCH_TYPE_HLL, Encoding::Msgpack), - (SKETCH_TYPE_COUNTMINSKETCH, Encoding::Msgpack), - (SKETCH_TYPE_COUNTSKETCH, Encoding::Msgpack), - (SKETCH_TYPE_KLL, Encoding::ProtoFull), + fn every_sketch_type_transmits_sketches() { + use crate::observation::{Observation, ObservationValue}; + for sketch_type in [ + SKETCH_TYPE_DDSKETCH, + SKETCH_TYPE_KLL, + SKETCH_TYPE_HLL, + SKETCH_TYPE_COUNTSKETCH, + SKETCH_TYPE_COUNTMINSKETCH, + SKETCH_TYPE_COUNTMINSKETCH_HEAP, ] { let mut cfg = make_config(sketch_type); - cfg.encoding = encoding; cfg.transmit_sketch = true; - assert!( - resolve(&cfg).is_err(), - "{sketch_type}/{encoding:?} must not emit a private sketch payload" - ); + let (pcfg, dispatch) = resolve(&cfg).expect("resolve"); + assert!(pcfg.transmit_sketch); + let mut sketch = (dispatch.factory)(); + let obs = Observation::new(1, "m", vec![], vec![], ObservationValue::float(1.5)); + dispatch.observer.observe(sketch.as_mut(), &obs).unwrap(); + let bytes = sketch.snapshot().unwrap(); + assert_eq!(&bytes[..6], b"ASAPv1", "{sketch_type}"); + let mut fresh = (dispatch.factory)(); + fresh.merge_snapshot(&bytes).unwrap(); } } #[test] - fn dispatch_handles_all_five_canonical_sketch_types() { + fn dispatch_handles_all_canonical_sketch_types() { for (in_str, expected) in [ ("ddsketch", SketchType::DDSketch), ("kll", SketchType::KLLSketch), ("hll", SketchType::HLLSketch), ("countsketch", SketchType::CountSketch), ("countminsketch", SketchType::CountMinSketch), + ("countminsketch_heap", SketchType::CountMinSketchWithHeap), ] { let (pcfg, dispatch) = resolve(&make_config(in_str)).expect("resolve"); assert_eq!(dispatch.sketch_type, expected); @@ -463,30 +424,4 @@ mod tests { let (_, dispatch) = resolve(&cfg).expect("resolve"); let _ = (dispatch.factory)(); } - - #[test] - fn msgpack_encoding_flows_into_precompute_config() { - let mut cfg = make_config("ddsketch"); - cfg.encoding = Encoding::MsgpackDelta; - cfg.delta_transmission = true; - let (pcfg, dispatch) = resolve(&cfg).expect("resolve"); - assert_eq!(pcfg.encoding, Encoding::MsgpackDelta); - assert!(pcfg.delta_transmission); - // Factory builds a wrapper with the encoding baked in. - let _ = (dispatch.factory)(); - } - - #[test] - fn estimate_mode_allows_all_kll_encodings() { - // Scalar estimate mode does not put private sketch bytes on the wire. - for enc in [ - Encoding::ProtoFull, - Encoding::Msgpack, - Encoding::MsgpackDelta, - ] { - let mut cfg = make_config("kll"); - cfg.encoding = enc; - assert!(resolve(&cfg).is_ok(), "kll should accept {enc:?}"); - } - } } diff --git a/asap-precompute-rs/src/otap/mod.rs b/asap-precompute-rs/src/otap/mod.rs index f959a55..9cf90bd 100644 --- a/asap-precompute-rs/src/otap/mod.rs +++ b/asap-precompute-rs/src/otap/mod.rs @@ -1,21 +1,14 @@ //! OpenTelemetry Arrow Protocol integration. //! //! With `otap-engine`, [`codec`] binds sketch envelopes directly to native -//! `OtapPdata`: sketch records are OTLP Summary data points carrying canonical -//! ASAPv1 bytes in `SummaryDpAttrs["sketch.envelope"]`; scalar estimates are -//! Gauge number data points. [`processor`] registers the corresponding local -//! OTAP dataflow processor. -//! -//! The flat `RecordBatch` codec and `records` projection remain available to -//! standalone adapters. Node-to-node transport must use native `OtapPdata` or -//! standard OTLP metrics protobuf, not a private sketch batch type. +//! `OtapPdata`: sketch records are OTLP Summary data points carrying ASAPv1 +//! bytes from `asap_sketchlib` in `SummaryDpAttrs["sketch.envelope"]`; scalar +//! estimates are Gauge number data points. [`processor`] registers the +//! corresponding local OTAP dataflow processor. -mod decode; -mod encode; mod schema; pub mod config; -pub mod records; #[cfg(feature = "otap-engine")] pub mod codec; @@ -23,10 +16,7 @@ pub mod codec; pub mod processor; pub use config::{ConfigError, PluginConfig, SketchDispatch}; -pub use decode::{decode_batch, OtapDecodeError}; -pub use encode::{encode_batch, OtapEncodeError}; -pub use records::{flatten, lift, OtapMetricRecords, OtapRecordsError}; pub use schema::{ - ATTR_AGG_ID, ATTR_ENCODING, ATTR_ENVELOPE, ATTR_SCHEMA_VERSION, ATTR_SKETCH_TYPE, - ATTR_WINDOW_END_MS, ATTR_WINDOW_START_MS, COLUMN_METRIC, COLUMN_TIME_UNIX_NANO, COLUMN_VALUE, + ATTR_AGG_ID, ATTR_ENVELOPE, ATTR_SCHEMA_VERSION, ATTR_SERIES_ID, ATTR_SKETCH_SIZE, + ATTR_SKETCH_TYPE, ATTR_WINDOW_END_MS, ATTR_WINDOW_START_MS, }; diff --git a/asap-precompute-rs/src/otap/processor.rs b/asap-precompute-rs/src/otap/processor.rs index 65ace8e..5b899a9 100644 --- a/asap-precompute-rs/src/otap/processor.rs +++ b/asap-precompute-rs/src/otap/processor.rs @@ -92,7 +92,7 @@ use otel_arrow_dfe_otap::pdata::OtapPdata; use otel_arrow_dfe_otap::OTAP_PROCESSOR_FACTORIES; use crate::config::PrecomputeConfigSet; -use crate::envelope::{Encoding, SketchEnvelope}; +use crate::envelope::SketchEnvelope; use crate::otap::config::resolve as resolve_plugin_config; use crate::otap::PluginConfig; use crate::precompute::{Precompute, PrecomputeError, PrecomputeImpl}; @@ -114,7 +114,8 @@ pub const ASAP_SKETCHES_PROCESSOR_URN: &str = "urn:asap:processor:asap_sketches" #[serde(deny_unknown_fields)] pub struct AsapSketchesUserConfig { /// One of `"ddsketch"` / `"kll"` / `"hll"` / `"countsketch"` / - /// `"countminsketch"` (case-insensitive). See `sample.toml`. + /// `"countminsketch"` / `"countminsketch_heap"` (case-insensitive). + /// See `sample.toml`. pub sketch_type: String, /// Window rotation period — humantime format (e.g. `"10s"`). #[serde(with = "humantime_serde")] @@ -144,24 +145,12 @@ pub struct AsapSketchesUserConfig { /// Add sample-count and window-duration labels to emitted rows. #[serde(default)] pub emit_window_stats: bool, - /// Sketch payload encoding. Defaults to self-describing ASAPv1 MessagePack. - #[serde(default)] - pub encoding: Option, - /// Emit deltas after the first full snapshot. - #[serde(default)] - pub delta_transmission: bool, /// Emit sketch envelopes when true; emit scalar estimates when false. #[serde(default = "default_transmit_sketch")] pub transmit_sketch: bool, /// Quantiles emitted by DDSketch/KLL in estimate mode. #[serde(default)] pub quantiles: Vec, - /// Bootstrap controller URL. Optional. - #[serde(default)] - pub controller_url: Option, - /// Bootstrap agent identifier reported to the controller. - #[serde(default)] - pub agent_id: Option, } impl AsapSketchesUserConfig { @@ -197,8 +186,6 @@ impl AsapSketchesUserConfig { omit_resource_attrs: self.omit_resource_attrs, global_aggregation: self.global_aggregation, emit_window_stats: self.emit_window_stats, - encoding: self.encoding.unwrap_or(Encoding::Msgpack), - delta_transmission: self.delta_transmission, transmit_sketch: self.transmit_sketch, quantiles: self.quantiles, }; @@ -247,7 +234,6 @@ fn requires_precompute_rebuild(current: &PluginConfig, next: &PluginConfig) -> b || ((!current.window_slide.is_zero() || !next.window_slide.is_zero()) && current.window_size != next.window_size) || current.sketch_params != next.sketch_params - || current.encoding != next.encoding || current.default_key != next.default_key || current.agg_id != next.agg_id || current.omit_resource_attrs != next.omit_resource_attrs @@ -556,7 +542,7 @@ mod tests { #[test] fn sliding_config_selects_pane_timer_and_validates_alignment() { let parse = |size: &str, slide: &str, transmit: bool| { - serde_json::from_value::(serde_json::json!({"sketch_type":"kll","window_size":size,"window_slide":slide,"output_metric_name":"rolling","encoding":"Msgpack","transmit_sketch":transmit,"quantiles":[0.5,0.99]})).unwrap().into_plugin_config() + serde_json::from_value::(serde_json::json!({"sketch_type":"kll","window_size":size,"window_slide":slide,"output_metric_name":"rolling","transmit_sketch":transmit,"quantiles":[0.5,0.99]})).unwrap().into_plugin_config() }; let config = parse("60s", "5s", false).unwrap(); assert_eq!( @@ -591,8 +577,6 @@ mod tests { assert_eq!(cfg.agg_id, 0); assert!(cfg.transmit_sketch); assert!(cfg.quantiles.is_empty()); - assert!(cfg.encoding.is_none()); - assert!(cfg.controller_url.is_none()); } #[test] @@ -670,8 +654,6 @@ mod tests { "sketch_type": "ddsketch", "window_size": "5s", "output_metric_name": "request.duration.p99", - "encoding": "Msgpack", - "delta_transmission": true, "transmit_sketch": false, "quantiles": [0.5, 0.99], "omit_resource_attrs": true, @@ -680,8 +662,6 @@ mod tests { })) .expect("extended control-plane shape parses"); let plugin = user.into_plugin_config().expect("extended config resolves"); - assert_eq!(plugin.encoding, Encoding::Msgpack); - assert!(plugin.delta_transmission); assert!(!plugin.transmit_sketch); assert_eq!(plugin.quantiles, vec![0.5, 0.99]); assert!(plugin.omit_resource_attrs); @@ -722,10 +702,6 @@ mod tests { runtime_only_change.window_size = Duration::from_secs(60); assert!(!requires_precompute_rebuild(¤t, &runtime_only_change)); - let mut encoding_change = current.clone(); - encoding_change.encoding = Encoding::ProtoFull; - assert!(requires_precompute_rebuild(¤t, &encoding_change)); - let mut series_identity_change = current.clone(); series_identity_change.global_aggregation = true; assert!(requires_precompute_rebuild( diff --git a/asap-precompute-rs/src/otap/schema.rs b/asap-precompute-rs/src/otap/schema.rs index f033e77..a6f437e 100644 --- a/asap-precompute-rs/src/otap/schema.rs +++ b/asap-precompute-rs/src/otap/schema.rs @@ -1,146 +1,21 @@ -//! Well-known column / attribute names for the OTAP-Rust codec. +//! Attribute keys of the native OTAP sketch carrier. //! -//! Strategy-B carrier keys are case-sensitive, prefix-included. -//! Every Strategy-B adapter -//! (Telegraf today, OTAP / Vector tomorrow) uses these exact -//! spellings — the codec MUST NOT change them. - -// ---------- Well-known scalar / metric columns ---------- - -/// Timestamp column (nanoseconds since the Unix epoch). Either `UInt64` -/// or Arrow's typed `Timestamp(Nanosecond, _)` is accepted on decode; -/// encode emits `UInt64` for portability across consumers that don't -/// have a timezone-aware Arrow runtime. -pub const COLUMN_TIME_UNIX_NANO: &str = "time_unix_nano"; - -/// Metric-name column. `Utf8` on decode and encode. -pub const COLUMN_METRIC: &str = "metric"; - -/// Scalar value column. `Float64` on decode and encode. -pub const COLUMN_VALUE: &str = "value"; - -// ---------- Strategy-B per-row attribute keys ---------- -// -// In the OTAP carrier these ride as -// `AttributeValueType::Bytes` / typed values on the per-row attribute -// child batch (NOT as sibling top-level columns on the metrics batch -// — OTAP's strict schema validator rejects extension columns). The -// Phase-B flat-RecordBatch shape carries them as plain columns; the -// Phase-C plugin shell projects/joins them back to the per-row -// attribute child batch when emitting an OtapArrowRecords downstream. - -/// Strategy-B envelope payload key. `Binary` column. Presence routes -/// the row through [`crate::observation::ObservationValueKind::Envelope`]. -pub const ATTR_ENVELOPE: &str = "_asap_envelope"; - -/// Strategy-B sketch-type tag. `Utf8` column carrying canonical names -/// (`"DDSketch"` / `"KLLSketch"` / `"HLLSketch"` / `"CountSketch"` / -/// `"CountMinSketch"`). -pub const ATTR_SKETCH_TYPE: &str = "_asap_sketch_type"; - -/// Strategy-B controller-plan join key. `UInt64` column. -pub const ATTR_AGG_ID: &str = "_asap_agg_id"; - -/// Strategy-B envelope schema version. `UInt32` column. -pub const ATTR_SCHEMA_VERSION: &str = "_asap_schema_version"; - -/// Strategy-B window inclusive lower bound (Unix milliseconds). -/// `UInt64` column. -pub const ATTR_WINDOW_START_MS: &str = "_asap_window_start_ms"; - -/// Strategy-B window exclusive upper bound (Unix milliseconds). -/// `UInt64` column. -pub const ATTR_WINDOW_END_MS: &str = "_asap_window_end_ms"; - -/// Strategy-B encoding tag. `Utf8` column carrying canonical names -/// (`"PROTO_FULL"` / `"PROTO_DELTA"` / `"MSGPACK"`). -pub const ATTR_ENCODING: &str = "_asap_encoding"; - -/// Stable series identity carried on the OTAP instrumentation-scope -/// attribute child batch. Number data points reference their scope through -/// OTAP's native `scope_id` join, so this value is not repeated per record. -pub const ATTR_SERIES_ID: &str = "_asap_series_id"; - -/// Optional sketch size/accuracy parameters on the Resource SCHEMA row. -pub const ATTR_SKETCH_SIZE: &str = "_asap_sketch_size"; -/// Optional hash seed on the Resource SCHEMA row. -pub const ATTR_HASH_SEED: &str = "_asap_hash_seed"; -/// Optional hash algorithm on the Resource SCHEMA row. -pub const ATTR_HASH_FUNCTION: &str = "_asap_hash_function"; - -// Native OTAP Summary carrier keys. These intentionally use the public -// `sketch.*` namespace; the `_asap_*` constants above remain for the legacy -// flat Gauge carrier and its compatibility codecs. -pub const OTAP_ATTR_ENVELOPE: &str = "sketch.envelope"; -pub const OTAP_ATTR_SKETCH_TYPE: &str = "sketch.type"; -pub const OTAP_ATTR_AGG_ID: &str = "sketch.agg_id"; -pub const OTAP_ATTR_SCHEMA_VERSION: &str = "sketch.schema_version"; -pub const OTAP_ATTR_WINDOW_START_MS: &str = "sketch.window_start_ms"; -pub const OTAP_ATTR_WINDOW_END_MS: &str = "sketch.window_end_ms"; -pub const OTAP_ATTR_ENCODING: &str = "sketch.encoding"; -pub const OTAP_ATTR_SERIES_ID: &str = "sketch.series_id"; -pub const OTAP_ATTR_SKETCH_SIZE: &str = "sketch.size"; -pub const OTAP_ATTR_HASH_SEED: &str = "sketch.hash_seed"; -pub const OTAP_ATTR_HASH_FUNCTION: &str = "sketch.hash_function"; - -/// Returns true if `name` is one of the well-known scalar columns or -/// `_asap_*` carrier keys; everything else is treated as a per-row -/// label by [`crate::otap::decode_batch`]. -pub fn is_reserved_column(name: &str) -> bool { - matches!( - name, - COLUMN_TIME_UNIX_NANO - | COLUMN_METRIC - | COLUMN_VALUE - | ATTR_ENVELOPE - | ATTR_SKETCH_TYPE - | ATTR_AGG_ID - | ATTR_SCHEMA_VERSION - | ATTR_WINDOW_START_MS - | ATTR_WINDOW_END_MS - | ATTR_ENCODING - | ATTR_SERIES_ID - | ATTR_SKETCH_SIZE - | ATTR_HASH_SEED - | ATTR_HASH_FUNCTION - ) -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn well_known_keys_match_edge_framework_doc() { - // Cross-check the Strategy-B key spellings — these MUST be - // byte-identical for cross-platform interop. - assert_eq!(ATTR_ENVELOPE, "_asap_envelope"); - assert_eq!(ATTR_SKETCH_TYPE, "_asap_sketch_type"); - assert_eq!(ATTR_AGG_ID, "_asap_agg_id"); - assert_eq!(ATTR_SCHEMA_VERSION, "_asap_schema_version"); - assert_eq!(ATTR_WINDOW_START_MS, "_asap_window_start_ms"); - assert_eq!(ATTR_WINDOW_END_MS, "_asap_window_end_ms"); - assert_eq!(ATTR_ENCODING, "_asap_encoding"); - } - - #[test] - fn reserved_names_recognized() { - for k in [ - COLUMN_TIME_UNIX_NANO, - COLUMN_METRIC, - COLUMN_VALUE, - ATTR_ENVELOPE, - ATTR_SKETCH_TYPE, - ATTR_AGG_ID, - ATTR_SCHEMA_VERSION, - ATTR_WINDOW_START_MS, - ATTR_WINDOW_END_MS, - ATTR_ENCODING, - ] { - assert!(is_reserved_column(k), "{} should be reserved", k); - } - assert!(!is_reserved_column("region")); - assert!(!is_reserved_column("path")); - assert!(!is_reserved_column("")); - } -} +//! SCHEMA keys ride on the Resource, `sketch.series_id` on the Scope, and +//! RECORD keys on each data point (see `docs/data_model.md`). + +/// ASAPv1 sketch envelope bytes from `asap_sketchlib` (RECORD). +pub const ATTR_ENVELOPE: &str = "sketch.envelope"; +/// Canonical [`crate::envelope::SketchType::name`] (SCHEMA). +pub const ATTR_SKETCH_TYPE: &str = "sketch.type"; +/// Controller-plan join key (SCHEMA). +pub const ATTR_AGG_ID: &str = "sketch.agg_id"; +/// Envelope schema version (SCHEMA). +pub const ATTR_SCHEMA_VERSION: &str = "sketch.schema_version"; +/// Inclusive window lower bound, Unix milliseconds (RECORD). +pub const ATTR_WINDOW_START_MS: &str = "sketch.window_start_ms"; +/// Exclusive window upper bound, Unix milliseconds (RECORD). +pub const ATTR_WINDOW_END_MS: &str = "sketch.window_end_ms"; +/// Stable series identity on the Scope (DICTIONARY / LABELS). +pub const ATTR_SERIES_ID: &str = "sketch.series_id"; +/// Optional sketch size/accuracy parameters (SCHEMA). +pub const ATTR_SKETCH_SIZE: &str = "sketch.size"; diff --git a/asap-precompute-rs/src/precompute.rs b/asap-precompute-rs/src/precompute.rs index 2fbd181..b306a7c 100644 --- a/asap-precompute-rs/src/precompute.rs +++ b/asap-precompute-rs/src/precompute.rs @@ -8,10 +8,9 @@ use std::time::Duration; use thiserror::Error; use crate::config::{PrecomputeConfig, PrecomputeConfigSet}; -use crate::envelope::{Encoding, SketchEnvelope, SketchType}; +use crate::envelope::{SketchEnvelope, SketchType}; use crate::matchers::series_attrs; use crate::observation::{KeyValue, Observation, ObservationValueKind}; -use crate::snapshot_cache::SnapshotCache; use crate::window::{SeriesEntry, WindowState}; /// Narrow interface the Layer-3 runtime needs from a Layer-1 sketch @@ -26,82 +25,23 @@ use crate::window::{SeriesEntry, WindowState}; /// bytes), so the routing layer above lives in the per-shim glue /// (see [`SketchObserver`]), not here. pub trait Sketch: Send + Sync { - /// Serializes the current sketch state to a portable - /// proto-encoded byte slice. - /// - /// Used by [`crate::snapshot_cache::SnapshotCache::compute_delta`] - /// and as the `ProtoFull` payload. + /// Serializes the current sketch state as sketchlib's self-describing + /// ASAPv1 envelope. An empty sketch returns an empty `Vec`. fn snapshot(&self) -> Result, PrecomputeError>; - /// Computes a sparse delta between this sketch and the previous - /// snapshot bytes. - /// - /// If the resulting delta is at least as large as a full snapshot - /// scaled by `threshold`, returns the full state with - /// `is_full = true` so the caller can avoid wasted work - /// re-marshaling. `threshold` is the per-sketch absolute count - /// cap (see today's `computeDDSketchDelta` / - /// `countsketch.ComputeDelta`). - fn compute_delta_against( - &self, - prev: &[u8], - threshold: u64, - ) -> Result; + /// Decodes an ASAPv1 envelope produced by [`Self::snapshot`] of the same + /// sketch type and merges it in. Empty bytes are a no-op. + fn merge_snapshot(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError>; - /// Merges a previously-computed delta into this sketch in place. - /// - /// Used by [`Precompute::observe_envelope`] when an inbound - /// envelope is encoded as `ProtoDelta`. - fn apply_delta(&mut self, delta: &[u8]) -> Result<(), PrecomputeError>; - - /// Encoding-aware ingest of an inbound frame (full or delta). - /// - /// `encoding` is the inbound envelope's [`Encoding`] tag, which may - /// differ from this node's configured outbound format (e.g. a central - /// merger receiving `MsgpackDelta` while emitting `Msgpack`). The - /// default delegates to [`Self::apply_delta`], preserving the proto - /// behavior (which auto-detects a full envelope vs a proto delta); - /// msgpack-capable wrappers override this to dispatch the proto vs - /// msgpack decoder off the tag. - fn apply_delta_encoded( - &mut self, - payload: &[u8], - _encoding: Encoding, - ) -> Result<(), PrecomputeError> { - self.apply_delta(payload) + /// Folds another sketch of the same type into this one. + fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError> { + self.merge_snapshot(&other.snapshot()?) } - /// Folds another sketch (typically a freshly-decoded envelope - /// payload) into this one. - /// - /// Used by [`Precompute::observe_envelope`] on `ProtoFull` - /// inbound envelopes. - fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError>; - /// Zeros the sketch in place. Used by window rotation and by /// sketch object pools. fn reset(&mut self); - /// Per-window delta base for a window-reset producer. - /// - /// When a family opts in to "true per-window deltas", - /// this returns the snapshot bytes - /// the [`crate::snapshot_cache::SnapshotCache`] should cache as the - /// outbound base AFTER each window-close emit — i.e. the snapshot of - /// an EMPTY sketch of the same shape. The next window's - /// [`Self::compute_delta_against`] then diffs against empty, so its - /// delta is that window's own full per-window state encoded as a - /// delta (no cross-window subtraction). - /// - /// The default returns `None`: such families keep the legacy - /// always-refresh behavior (the cache is refreshed to the - /// just-emitted full state). DDSketch, CMS, CountSketch, and HLL - /// override this to opt in to per-window deltas; KLL keeps the default - /// (full-only). - fn delta_against_empty_base(&self) -> Result>, PrecomputeError> { - Ok(None) - } - /// Produce estimated output points for the `transmit_sketch = false` /// mode. `quantiles` are the configured quantiles (quantile sketches /// emit one point per quantile with a `quantile` label); `top_k` @@ -124,26 +64,13 @@ pub trait Sketch: Send + Sync { /// Implementations that want observer downcasting (e.g. real /// sketch wrappers in [`crate::sketches`]) override this with /// `fn as_any_mut(&mut self) -> &mut dyn Any { self }`. Test - /// fakes that route observations via byte-level apply_delta + /// fakes that route observations via byte-level merge_snapshot /// (see `tests/runtime.rs::FakeSketch`) can keep the default /// impl which never matches a real downcast — the FakeObserver /// doesn't call `as_any_mut`. fn as_any_mut(&mut self) -> &mut dyn Any; } -/// Output of [`Sketch::compute_delta_against`]. -#[derive(Clone, Debug, Default, PartialEq)] -pub struct DeltaResult { - /// Wire bytes — either a sparse delta (when `is_full = false`) - /// or the full snapshot (when `is_full = true`). - pub payload: Vec, - /// Whether `payload` is the full snapshot rather than a sparse - /// delta. Caller uses this to set - /// [`crate::envelope::Encoding::ProtoFull`] vs - /// [`crate::envelope::Encoding::ProtoDelta`]. - pub is_full: bool, -} - /// Implemented by sketches that can answer quantile queries. /// /// DDSketch and KLL are the two `QuantileSketch` implementations. @@ -274,7 +201,7 @@ pub enum PrecomputeError { /// Config-side type. config: SketchType, }, - /// Generic catch-all for nested errors (proto decode, sketch + /// Generic catch-all for nested errors (ASAPv1 decode, sketch /// internal failure, etc.). #[error("precompute: {0}")] Other(String), @@ -393,7 +320,6 @@ pub struct PrecomputeImpl { sketch_factory: Option, observer: Option, window: Mutex, - snapshot_cache: SnapshotCache, stats: Mutex, sketch_type: SketchType, closed: AtomicBool, @@ -423,7 +349,6 @@ impl PrecomputeImpl { sketch_factory, observer, window: Mutex::new(WindowState::new()), - snapshot_cache: SnapshotCache::new(), stats: Mutex::new(StatsSnapshot::default()), sketch_type, closed: AtomicBool::new(false), @@ -448,7 +373,7 @@ impl PrecomputeImpl { } /// Walks the closed series, serializes each into a - /// [`SketchEnvelope`] (honoring `delta_transmission`), and + /// [`SketchEnvelope`], and /// updates the rolling stats counters. fn finish_rotate( &self, @@ -482,17 +407,13 @@ impl PrecomputeImpl { envelopes } - /// Turns a closed series entry into a [`SketchEnvelope`]. - /// Honors `delta_transmission` via the snapshot cache. + /// Turns a closed series entry into [`SketchEnvelope`]s. fn serialize_series( &self, entry: &SeriesEntry, cfg: &PrecomputeConfig, rng: [u64; 2], ) -> Result, PrecomputeError> { - // Rebuild the same key the window used at admit time. - let series_key = cfg.series_key_for_entry(&entry.resource_labels, &entry.labels); - // Base labels shared by both output modes (series attrs + optional // operator-visibility window stats). let mut base_labels = series_attrs(&entry.labels, &cfg.aggregate_by); @@ -534,9 +455,7 @@ impl PrecomputeImpl { window_start_ms: rng[0], window_end_ms: rng[1], // No sketch bytes on the estimate path. - encoding: Encoding::Unspecified, payload: Vec::new(), - hash_spec: None, metric_name: cfg.metric_name.clone(), count: entry.count, aggregation_temporality: cfg.temporality, @@ -547,28 +466,8 @@ impl PrecomputeImpl { return Ok(out); } - // Sketch-on-the-wire mode. - let (payload, encoding) = if cfg.delta_transmission { - let result = self.snapshot_cache.compute_delta( - &series_key, - entry.sketch.as_ref(), - cfg.delta_threshold, - )?; - let enc = if result.is_full { - full_encoding(cfg.encoding) - } else { - delta_encoding(cfg.encoding) - }; - (result.payload, enc) - } else { - let snap = entry.sketch.snapshot()?; - // Even without delta transmission, refreshing the cached - // outbound snapshot keeps the cache consistent for any - // later config change that flips delta_transmission to - // true. - self.snapshot_cache.cache_outbound(&series_key, &snap); - (snap, full_encoding(cfg.encoding)) - }; + // Sketch-on-the-wire mode: the full ASAPv1 snapshot. + let payload = entry.sketch.snapshot()?; if payload.is_empty() { return Ok(Vec::new()); } @@ -580,9 +479,7 @@ impl PrecomputeImpl { labels: base_labels, window_start_ms: rng[0], window_end_ms: rng[1], - encoding, payload, - hash_spec: None, metric_name: cfg.metric_name.clone(), count: entry.count, aggregation_temporality: cfg.temporality, @@ -591,29 +488,6 @@ impl PrecomputeImpl { } } -/// Maps a configured wire format to the [`Encoding`] tag stamped on a -/// **full** frame: msgpack configs emit `Msgpack`, everything else -/// `ProtoFull`. The wrapper's `snapshot()` produces bytes in the matching -/// format because its `wire_encoding` is baked from the same `cfg.encoding`. -fn full_encoding(configured: Encoding) -> Encoding { - if configured.is_msgpack() { - Encoding::Msgpack - } else { - Encoding::ProtoFull - } -} - -/// Maps a configured wire format to the [`Encoding`] tag stamped on a -/// **delta** frame: msgpack configs emit `MsgpackDelta`, everything else -/// `ProtoDelta`. -fn delta_encoding(configured: Encoding) -> Encoding { - if configured.is_msgpack() { - Encoding::MsgpackDelta - } else { - Encoding::ProtoDelta - } -} - impl Precompute for PrecomputeImpl { fn observe(&self, obs: &Observation) -> Result<(), PrecomputeError> { if self.closed.load(Ordering::Acquire) { @@ -703,8 +577,7 @@ impl Precompute for PrecomputeImpl { let mut window = self.window.lock().expect("window lock poisoned"); let mut stats = self.stats.lock().expect("stats lock poisoned"); stats.input_envelopes = stats.input_envelopes.saturating_add(1); - let result = - window.observe_envelope(env, &cfg, sketch_factory, &self.snapshot_cache, &mut stats); + let result = window.observe_envelope(env, &cfg, sketch_factory, &mut stats); if let Err(err) = &result { if matches!(err, PrecomputeError::SeriesCapExceeded) { stats.dropped_overflow = stats.dropped_overflow.saturating_add(1); diff --git a/asap-precompute-rs/src/sketches/cms.rs b/asap-precompute-rs/src/sketches/cms.rs index 932a1a3..13f3c63 100644 --- a/asap-precompute-rs/src/sketches/cms.rs +++ b/asap-precompute-rs/src/sketches/cms.rs @@ -1,368 +1,73 @@ -//! CountMinSketch wrapper over [`asap_sketchlib::CountMinSketch`]. +//! Count-Min sketch wrapper over [`asap_sketchlib::CountMin`]. //! -//! Implements [`Sketch`] + [`FrequencySketch`]. +//! Implements [`Sketch`] + [`FrequencySketch`]. Snapshots are sketchlib's +//! ASAPv1 envelope (kind `0x02 0x00`). -use asap_sketchlib::proto::sketchlib::{ - sketch_envelope, CountMinState, CounterType, SketchEnvelope as ProtoEnvelope, -}; -use asap_sketchlib::{CountMinSketch, MessagePackCodec}; -use prost::Message; +use asap_sketchlib::{CountMin, DataInput, RegularPath, Vector2D, MATRIX_MAX_ROWS}; -use crate::envelope::Encoding; use crate::observation::Observation; -use crate::precompute::{ - DeltaResult, FrequencyEntry, FrequencySketch, PrecomputeError, Sketch, SketchObserver, -}; +use crate::precompute::{FrequencyEntry, FrequencySketch, PrecomputeError, Sketch, SketchObserver}; -/// Width, in bits, of the single 64-bit per-item hash that sketchlib's -/// CMS / CountSketch bit-slice across rows. Each row consumes -/// `ceil(log2(cols))` bits; once `rows * bitsPerRow > 64` the high rows -/// read shifted-out (zero) bits and collapse onto column 0. -const MAX_ROW_HASH_BITS: usize = 64; +type Cms = CountMin, RegularPath>; -/// Clamp `rows` so `rows * ceil(log2(cols)) <= 64`. `cols` is assumed -/// already rounded to a power of two, so `cols.trailing_zeros()` == -/// `log2(cols)` == the per-row bit width. When `bits_per_row == 0` -/// (cols == 1) every row maps to column 0 regardless, so the slicing -/// never overflows and `rows` is left as-is. Returns at least 1. -pub(crate) fn clamp_rows_for_hash_bits(rows: usize, cols: usize) -> usize { - let bits_per_row = cols.trailing_zeros() as usize; - if bits_per_row == 0 { - return rows.max(1); - } - let max_rows = (MAX_ROW_HASH_BITS / bits_per_row).max(1); - rows.min(max_rows).max(1) -} - -/// CountMinSketch wrapper. +/// Count-Min sketch wrapper. Keys are raw bytes. pub struct CMSWrapper { - sk: CountMinSketch, - rows: usize, - cols: usize, - /// Outbound wire format for this series' snapshots/deltas. Baked from - /// `cfg.encoding` by the OTAP sketch factory. - wire_encoding: Encoding, + sk: Cms, } impl CMSWrapper { - /// Construct a CMS with the given dimensions. - /// - /// The dimensions are normalized as follows for byte-parity: - /// - /// - `cols` is rounded UP to the next power of two (after a floor of - /// 1). sketchlib's CMS folds inserts with `% cols` but masks / - /// bit-slices the query hash assuming a power-of-two width, so a - /// non-pow2 cols mis-indexes. The canonical `cols = 2000` config - /// rounds to `2048`. - /// - `rows` is clamped so `rows * log2(cols) <= 64` (the single - /// 64-bit per-item hash budget); beyond that the high rows read - /// shifted-out (zero) bits and collapse onto column 0. - /// - /// The normalized `rows`/`cols` are stored on the struct AND used to - /// build the underlying `CountMinSketch`, so `build_state` serializes - /// the normalized dimensions onto the wire. + /// Construct a CMS with `rows` hash rows (clamped to `1..=MATRIX_MAX_ROWS`, + /// the number of independent sketchlib row seeds) and `cols` columns. pub fn new(rows: usize, cols: usize) -> Self { - let cols = cols.max(1).next_power_of_two(); - let rows = clamp_rows_for_hash_bits(rows, cols); Self { - sk: CountMinSketch::new(rows, cols), - rows, - cols, - wire_encoding: Encoding::ProtoFull, + sk: Cms::with_dimensions(rows.clamp(1, MATRIX_MAX_ROWS), cols.max(1)), } } - /// Set the outbound wire format (proto vs msgpack). Builder form used - /// by the OTAP sketch factory to bake in `cfg.encoding`. - pub fn with_wire_encoding(mut self, encoding: Encoding) -> Self { - self.wire_encoding = encoding; - self + /// Decode a full ASAPv1 snapshot. + pub fn from_snapshot(bytes: &[u8]) -> Result { + Cms::deserialize_from_bytes(bytes) + .map(|sk| Self { sk }) + .map_err(|e| PrecomputeError::Other(format!("CMSWrapper decode: {e}"))) } - /// Insert a string-keyed weighted observation. - pub fn update(&mut self, key: &str, value: f64) { - self.sk.update(key, value); + /// Add `count` occurrences of `key`. + pub fn update(&mut self, key: &[u8], count: i64) { + self.sk.insert_many(&DataInput::Bytes(key), count); } - /// Borrow the underlying `CountMinSketch`. - pub fn inner(&self) -> &CountMinSketch { + /// Borrow the underlying sketch. + pub fn inner(&self) -> &Cms { &self.sk } - - fn build_state(&self) -> CountMinState { - // Emit packed sint64 `counts_int` (Opt-2: 4–8× smaller than - // f64 for typical small-integer counter values) and per-row - // L1/L2 norms (the sketch maintains `L1[r] += weight` and - // `L2[r] += curr*curr - prev*prev`, which collapse to - // `sum_c count[r][c]` and `sum_c count[r][c]^2` for the - // unweighted unit-step stream the parity harness drives — the - // only producer pattern this wire path serves today). Omit - // `sum_counts` / `sum2_counts` (Frequency-Only mode) to match - // the portable serialization payload bit-for-bit. - let matrix = self.sk.sketch(); - let mut counts_int = Vec::with_capacity(self.rows * self.cols); - let mut l1 = Vec::with_capacity(self.rows); - let mut l2 = Vec::with_capacity(self.rows); - for row in matrix.iter().take(self.rows) { - let mut row_l1 = 0.0f64; - let mut row_l2 = 0.0f64; - for &cell in row.iter().take(self.cols) { - counts_int.push(cell as i64); - row_l1 += cell; - row_l2 += cell * cell; - } - l1.push(row_l1); - l2.push(row_l2); - } - CountMinState { - rows: self.rows as u32, - cols: self.cols as u32, - counter_type: CounterType::Int64 as i32, - counts_int, - counts_float: Vec::new(), - sum_counts: Vec::new(), - sum2_counts: Vec::new(), - l1, - l2, - } - } - - fn encode_envelope(&self) -> Vec { - let env = ProtoEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(sketch_envelope::SketchState::CountMin(self.build_state())), - }; - let mut buf = Vec::with_capacity(env.encoded_len()); - env.encode(&mut buf).expect("prost encode"); - buf - } - - fn decode_envelope(bytes: &[u8]) -> Result { - let env = ProtoEnvelope::decode(bytes) - .map_err(|e| PrecomputeError::Other(format!("CMSWrapper decode: {e}")))?; - let state = match env.sketch_state { - Some(sketch_envelope::SketchState::CountMin(s)) => s, - _ => { - return Err(PrecomputeError::Other( - "CMSWrapper: envelope did not carry CountMinState".into(), - )); - } - }; - let rows = state.rows as usize; - let cols = state.cols as usize; - let mut matrix = vec![vec![0.0f64; cols]; rows]; - if !state.counts_float.is_empty() { - for (r, row) in matrix.iter_mut().enumerate().take(rows) { - for (c, cell) in row.iter_mut().enumerate().take(cols) { - let idx = r * cols + c; - if idx < state.counts_float.len() { - *cell = state.counts_float[idx]; - } - } - } - } else if !state.counts_int.is_empty() { - for (r, row) in matrix.iter_mut().enumerate().take(rows) { - for (c, cell) in row.iter_mut().enumerate().take(cols) { - let idx = r * cols + c; - if idx < state.counts_int.len() { - *cell = state.counts_int[idx] as f64; - } - } - } - } - Ok(CountMinSketch::from_legacy_matrix(matrix, rows, cols)) - } - - fn is_empty(&self) -> bool { - let m = self.sk.sketch(); - m.iter().all(|row| row.iter().all(|&v| v == 0.0)) - } } impl Sketch for CMSWrapper { fn snapshot(&self) -> Result, PrecomputeError> { - if self.is_empty() { + if self.sk.as_storage().as_slice().iter().all(|&c| c == 0) { return Ok(Vec::new()); } - if self.wire_encoding.is_msgpack() { - return self - .sk - .to_msgpack() - .map_err(|e| PrecomputeError::Other(format!("CMSWrapper msgpack snapshot: {e}"))); - } - Ok(self.encode_envelope()) - } - - fn compute_delta_against( - &self, - prev: &[u8], - threshold: u64, - ) -> Result { - // Decode the prior snapshot envelope, then diff via - // `asap_sketchlib`'s `CountMinSketch::compute_delta`. On an empty / - // undecodable prior, or an empty current sketch, fall back to a - // full snapshot so the emit path always produces a valid payload. - // - // Under per-window delta-against-empty (the snapshot cache resets - // the cached base to the empty-sketch snapshot at each window - // close), `prev` decodes to an empty `CountMinSketch`, so the - // computed delta IS this window's full per-cell matrix encoded as - // cell deltas — no cross-window subtraction. - if self.is_empty() { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - if prev.is_empty() { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - if self.wire_encoding.is_msgpack() { - let prev_sk = match CountMinSketch::from_msgpack(prev) { - Ok(sk) => sk, - Err(_) => { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - }; - return match self.sk.compute_delta_msgpack(&prev_sk, threshold as f64) { - Ok(delta) => Ok(DeltaResult { - payload: delta, - is_full: false, - }), - Err(_) => { - let full = self.snapshot()?; - Ok(DeltaResult { - payload: full, - is_full: true, - }) - } - }; - } - let prev_sk = match Self::decode_envelope(prev) { - Ok(sk) => sk, - Err(_) => { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - }; - match self.sk.compute_delta(&prev_sk, threshold as f64) { - Ok(delta) => Ok(DeltaResult { - payload: delta, - is_full: false, - }), - Err(_) => { - let full = self.snapshot()?; - Ok(DeltaResult { - payload: full, - is_full: true, - }) - } - } - } - - fn apply_delta(&mut self, payload: &[u8]) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - // Dispatch on payload shape. A full-state envelope (the - // `SketchEnvelope{CountMinState}` wire format) takes the decode + - // merge path; otherwise the payload is a `CountMinDelta` proto and - // is applied additively via `apply_delta_bytes`. - if let Ok(other) = Self::decode_envelope(payload) { - return self - .sk - .merge(&other) - .map_err(|e| PrecomputeError::Other(format!("CMSWrapper merge: {e}"))); - } self.sk - .apply_delta_bytes(payload) - .map_err(|e| PrecomputeError::Other(format!("CMSWrapper apply_delta: {e}"))) + .serialize_to_bytes() + .map_err(|e| PrecomputeError::Other(format!("CMSWrapper snapshot: {e}"))) } - fn apply_delta_encoded( - &mut self, - payload: &[u8], - encoding: Encoding, - ) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - match encoding { - Encoding::MsgpackDelta => self.sk.apply_delta_msgpack_bytes(payload).map_err(|e| { - PrecomputeError::Other(format!("CMSWrapper msgpack apply_delta: {e}")) - }), - Encoding::Msgpack => { - let other = CountMinSketch::from_msgpack(payload).map_err(|e| { - PrecomputeError::Other(format!("CMSWrapper msgpack decode: {e}")) - })?; - self.sk - .merge(&other) - .map_err(|e| PrecomputeError::Other(format!("CMSWrapper merge: {e}"))) - } - _ => self.apply_delta(payload), - } - } - - fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError> { - let bytes = other.snapshot()?; + fn merge_snapshot(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError> { if bytes.is_empty() { return Ok(()); } - // `other.snapshot()` is in that wrapper's `wire_encoding`; same - // Precompute ⇒ same encoding as ours, so decode by our own tag. - let decoded = if self.wire_encoding.is_msgpack() { - CountMinSketch::from_msgpack(&bytes).map_err(|e| { - PrecomputeError::Other(format!("CMSWrapper msgpack merge decode: {e}")) - })? - } else { - Self::decode_envelope(&bytes)? - }; - self.sk - .merge(&decoded) - .map_err(|e| PrecomputeError::Other(format!("CMSWrapper merge: {e}"))) + let other = Self::from_snapshot(bytes)?; + if (self.sk.rows(), self.sk.cols()) != (other.sk.rows(), other.sk.cols()) { + return Err(PrecomputeError::Other( + "CMSWrapper merge: dimension mismatch".into(), + )); + } + self.sk.merge(&other.sk); + Ok(()) } fn reset(&mut self) { - self.sk = CountMinSketch::new(self.rows, self.cols); - } - - fn delta_against_empty_base(&self) -> Result>, PrecomputeError> { - // CMS opts in to per-window deltas. After a window-close emit the - // snapshot cache caches THIS - // — the encoded envelope of an EMPTY CountMinSketch of the same - // dimensions — so the next window's `compute_delta_against` diffs - // against empty and emits that window's own per-cell matrix as a - // delta (no cross-window subtraction). - // - // We encode the empty envelope rather than returning - // `Sketch::snapshot()` of an empty sketch, because the latter - // short-circuits to empty bytes (the runtime drops empty - // payloads), and empty bytes would make `compute_delta_against` - // fall back to a full snapshot instead of a delta. - if self.wire_encoding.is_msgpack() { - let empty = CountMinSketch::new(self.rows, self.cols); - let bytes = empty.to_msgpack().map_err(|e| { - PrecomputeError::Other(format!("CMSWrapper msgpack empty base: {e}")) - })?; - return Ok(Some(bytes)); - } - let empty = CMSWrapper::new(self.rows, self.cols); - Ok(Some(empty.encode_envelope())) + self.sk = Cms::with_dimensions(self.sk.rows(), self.sk.cols()); } fn as_any_mut(&mut self) -> &mut dyn std::any::Any { @@ -372,34 +77,21 @@ impl Sketch for CMSWrapper { impl FrequencySketch for CMSWrapper { fn estimate_count(&self, key: &[u8]) -> f64 { - if key.is_empty() { - return 0.0; - } - let s = std::str::from_utf8(key).unwrap_or_default(); - if s.is_empty() { - return 0.0; - } - self.sk.estimate(s) + self.sk.estimate(&DataInput::Bytes(key)) as f64 } + /// A plain Count-Min does not track keys; use + /// [`super::CMSHeapWrapper`] for top-k. fn top_k(&self, _k: usize) -> Vec { - // CountMinSketch is a frequency estimator over a known key - // set; it does not natively track top-k. Returning an empty - // slice is the expected behavior. Vec::new() } } -/// Observer routing observations into the wrapper, counting the -/// per-attribute-set frequency. +/// Observer counting each observation's key once. /// -/// The key is the observation's `bytes` value when present (preserves -/// pre-shaped callers / unit tests) or, for OTAP scalar observations -/// (Float-kind, empty bytes), the full label set via -/// [`crate::matchers::attributes_key`]. Each observation contributes -/// weight `1.0`. We accept any value kind (including Float decoded from the -/// OTAP `value` column) so CMS records the attribute-set on the OTAP -/// edge instead of silently dropping every observation. +/// The key is the observation's `bytes` value when present, otherwise its +/// full label set via [`crate::matchers::attributes_key`]. Any value kind +/// is accepted (OTAP scalar rows arrive as `Float` with empty bytes). pub struct CMSObserver; impl SketchObserver for CMSObserver { @@ -410,12 +102,14 @@ impl SketchObserver for CMSObserver { .ok_or_else(|| { PrecomputeError::Other("CMSObserver: sketch is not a CMSWrapper".into()) })?; - let key = if !obs.value.bytes.is_empty() { - String::from_utf8_lossy(&obs.value.bytes).into_owned() + if obs.value.bytes.is_empty() { + w.update( + crate::matchers::attributes_key(&obs.labels, &[]).as_bytes(), + 1, + ); } else { - crate::matchers::attributes_key(&obs.labels, &[]) - }; - w.update(&key, 1.0); + w.update(&obs.value.bytes, 1); + } Ok(()) } } @@ -426,50 +120,50 @@ mod tests { #[test] fn new_wrapper_is_empty() { - let w = CMSWrapper::new(4, 32); - assert_eq!(w.snapshot().unwrap().len(), 0); + assert!(CMSWrapper::new(4, 32).snapshot().unwrap().is_empty()); } #[test] fn update_then_estimate() { let mut w = CMSWrapper::new(8, 64); - for _ in 0..50 { - w.update("k1", 1.0); - } - for _ in 0..3 { - w.update("k2", 1.0); - } - let k1 = w.estimate_count(b"k1"); - let k2 = w.estimate_count(b"k2"); - assert!(k1 >= 50.0, "k1 underestimate: {k1}"); - assert!(k2 >= 3.0, "k2 underestimate: {k2}"); + w.update(b"k1", 50); + w.update(b"k2", 3); + assert!(w.estimate_count(b"k1") >= 50.0); + assert!(w.estimate_count(b"k2") >= 3.0); } #[test] - fn snapshot_roundtrip_preserves_matrix() { + fn snapshot_round_trips_through_asapv1() { let mut w = CMSWrapper::new(4, 8); - w.update("k", 1.0); + w.update(b"k", 1); let bytes = w.snapshot().unwrap(); - let decoded = CMSWrapper::decode_envelope(&bytes).unwrap(); - assert_eq!(decoded.sketch(), w.sk.sketch()); + let decoded = CMSWrapper::from_snapshot(&bytes).unwrap(); + assert_eq!( + decoded.inner().as_storage().as_slice(), + w.inner().as_storage().as_slice() + ); } - // B6 regression: on the OTAP edge, observations arrive Float-kind - // with empty bytes. CMS must (a) accept them instead of erroring - // and (b) count the per-attribute-set key, NOT the metric name. + #[test] + fn merge_adds_counts() { + let mut a = CMSWrapper::new(4, 64); + let mut b = CMSWrapper::new(4, 64); + a.update(b"k", 2); + b.update(b"k", 5); + a.merge(&b).unwrap(); + assert_eq!(a.estimate_count(b"k"), 7.0); + } + + /// OTAP scalar rows arrive Float-kind with empty bytes; CMS must count + /// the attribute set, not the metric name. #[test] fn observe_counts_attribute_set_not_metric_name() { use crate::matchers::attributes_key; - use crate::observation::{KeyValue, Observation, ObservationValue}; - use crate::precompute::SketchObserver; + use crate::observation::{KeyValue, ObservationValue}; let mut sketch = CMSWrapper::new(8, 64); - let observer = CMSObserver; let labels = vec![KeyValue::new("host", "h1")]; - // Float-kind, empty bytes — exactly what the OTAP decoder - // produces for a scalar row. - let n = 25; - for _ in 0..n { + for _ in 0..25 { let obs = Observation::new( 1_000, "flow_count", @@ -477,21 +171,11 @@ mod tests { labels.clone(), ObservationValue::float(1.0), ); - observer.observe(&mut sketch, &obs).expect("observe"); + CMSObserver.observe(&mut sketch, &obs).expect("observe"); } - let attr_key = attributes_key(&labels, &[]); assert_eq!(attr_key, "host=h1;"); - let counted = sketch.estimate_count(attr_key.as_bytes()); - assert!( - counted >= n as f64, - "attribute-set key undercounted: {counted} < {n}" - ); - // The metric NAME must not be the counted subject. - let metric_count = sketch.estimate_count(b"flow_count"); - assert!( - metric_count < n as f64, - "metric name was counted ({metric_count}); should be the attribute set" - ); + assert!(sketch.estimate_count(attr_key.as_bytes()) >= 25.0); + assert!(sketch.estimate_count(b"flow_count") < 25.0); } } diff --git a/asap-precompute-rs/src/sketches/cms_heap.rs b/asap-precompute-rs/src/sketches/cms_heap.rs index f9c4388..db71498 100644 --- a/asap-precompute-rs/src/sketches/cms_heap.rs +++ b/asap-precompute-rs/src/sketches/cms_heap.rs @@ -2,14 +2,16 @@ //! //! Implements [`Sketch`] + [`FrequencySketch`]. Snapshots are sketchlib's //! ASAPv1 envelope (kind `0x03 0x00`): the counter matrix plus the heap's -//! keys and counts. There is no delta encoding; every payload is full state. +//! keys and counts. use std::any::Any; -use asap_sketchlib::{CMSHeap, DataInput, HeapItem}; +use asap_sketchlib::{CMSHeap, DataInput, HeapItem, MATRIX_MAX_ROWS}; -use super::cms::clamp_rows_for_hash_bits; -use crate::precompute::{DeltaResult, FrequencyEntry, FrequencySketch, PrecomputeError, Sketch}; +use crate::observation::{KeyValue, Observation}; +use crate::precompute::{ + EstimatePoint, FrequencyEntry, FrequencySketch, PrecomputeError, Sketch, SketchObserver, +}; /// Count-Min sketch plus heavy-hitter heap. Keys are raw bytes. pub struct CMSHeapWrapper { @@ -17,14 +19,11 @@ pub struct CMSHeapWrapper { } impl CMSHeapWrapper { - /// Construct with `top_k` heap slots. `cols` is rounded up to a power of - /// two and `rows` clamped to the 64-bit hash budget, as for - /// [`super::CMSWrapper`]. + /// Construct with `top_k` heap slots; `rows` is clamped to + /// `1..=MATRIX_MAX_ROWS` as for [`super::CMSWrapper`]. pub fn new(rows: usize, cols: usize, top_k: usize) -> Self { - let cols = cols.max(1).next_power_of_two(); - let rows = clamp_rows_for_hash_bits(rows, cols); Self { - sk: CMSHeap::new(rows, cols, top_k.max(1)), + sk: CMSHeap::new(rows.clamp(1, MATRIX_MAX_ROWS), cols.max(1), top_k.max(1)), } } @@ -67,28 +66,33 @@ impl Sketch for CMSHeapWrapper { .map_err(|e| PrecomputeError::Other(format!("CMSHeapWrapper snapshot: {e}"))) } - fn compute_delta_against(&self, _: &[u8], _: u64) -> Result { - Ok(DeltaResult { - payload: self.snapshot()?, - is_full: true, - }) - } - - fn apply_delta(&mut self, payload: &[u8]) -> Result<(), PrecomputeError> { - if payload.is_empty() { + fn merge_snapshot(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError> { + if bytes.is_empty() { return Ok(()); } - self.merge_from(&Self::from_snapshot(payload)?) - } - - fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError> { - self.apply_delta(&other.snapshot()?) + self.merge_from(&Self::from_snapshot(bytes)?) } fn reset(&mut self) { self.sk = CMSHeap::new(self.sk.rows(), self.sk.cols(), self.sk.heap().capacity()); } + /// One point per heap entry (the configured `k`), labelled with its + /// `key` and 1-based `rank`. + fn estimate(&self, _quantiles: &[f64], _top_k: usize) -> Vec { + self.top_k(self.sk.heap().capacity()) + .into_iter() + .enumerate() + .map(|(rank, entry)| EstimatePoint { + labels: vec![ + KeyValue::new("key", String::from_utf8_lossy(&entry.key).into_owned()), + KeyValue::new("rank", (rank + 1).to_string()), + ], + value: entry.count, + }) + .collect() + } + fn as_any_mut(&mut self) -> &mut dyn Any { self } @@ -120,6 +124,33 @@ impl FrequencySketch for CMSHeapWrapper { } } +/// Observer counting each observation's key once into a [`CMSHeapWrapper`]. +/// +/// The key is the observation's `bytes` value when present, otherwise its +/// full label set via [`crate::matchers::attributes_key`], as for +/// [`super::CMSObserver`]. +pub struct CMSHeapObserver; + +impl SketchObserver for CMSHeapObserver { + fn observe(&self, sketch: &mut dyn Sketch, obs: &Observation) -> Result<(), PrecomputeError> { + let w = sketch + .as_any_mut() + .downcast_mut::() + .ok_or_else(|| { + PrecomputeError::Other("CMSHeapObserver: sketch is not a CMSHeapWrapper".into()) + })?; + if obs.value.bytes.is_empty() { + w.update( + crate::matchers::attributes_key(&obs.labels, &[]).as_bytes(), + 1, + ); + } else { + w.update(&obs.value.bytes, 1); + } + Ok(()) + } +} + #[cfg(test)] mod tests { use super::*; @@ -164,6 +195,14 @@ mod tests { assert_eq!(left.estimate_count(&4u64.to_be_bytes()), 14.0); } + #[test] + fn estimate_emits_ranked_heavy_hitters() { + let points = keyed(&[(1, 5), (2, 50), (3, 20)]).estimate(&[], 0); + assert_eq!(points.len(), 3); + assert_eq!(points[0].value, 50.0); + assert_eq!(points[0].labels[1], KeyValue::new("rank", "1")); + } + #[test] fn empty_sketch_has_empty_snapshot() { assert!(CMSHeapWrapper::new(4, 64, 2).snapshot().unwrap().is_empty()); diff --git a/asap-precompute-rs/src/sketches/countsketch.rs b/asap-precompute-rs/src/sketches/countsketch.rs index e6aab22..9fec5ea 100644 --- a/asap-precompute-rs/src/sketches/countsketch.rs +++ b/asap-precompute-rs/src/sketches/countsketch.rs @@ -1,354 +1,73 @@ -//! CountSketch wrapper over [`asap_sketchlib::CountSketch`]. +//! Count Sketch wrapper over [`asap_sketchlib::Count`]. //! -//! Implements [`Sketch`] + [`FrequencySketch`]. +//! Implements [`Sketch`] + [`FrequencySketch`]. Snapshots are sketchlib's +//! ASAPv1 envelope (kind `0x04 0x00`). -use asap_sketchlib::proto::sketchlib::{ - sketch_envelope, CountSketchState, CounterType, SketchEnvelope as ProtoEnvelope, -}; -use asap_sketchlib::{CountSketch, MessagePackCodec}; -use prost::Message; +use asap_sketchlib::{Count, DataInput, RegularPath, Vector2D, MATRIX_MAX_ROWS}; -use crate::envelope::Encoding; use crate::observation::Observation; -use crate::precompute::{ - DeltaResult, FrequencyEntry, FrequencySketch, PrecomputeError, Sketch, SketchObserver, -}; +use crate::precompute::{FrequencyEntry, FrequencySketch, PrecomputeError, Sketch, SketchObserver}; -/// Width, in bits, of the single 64-bit per-item hash that sketchlib's -/// CountSketch bit-slices across rows. See `cms.rs::MAX_ROW_HASH_BITS`. -const MAX_ROW_HASH_BITS: usize = 64; +type Cs = Count, RegularPath>; -/// Clamp `rows` so `rows * ceil(log2(cols)) <= 64`. Identical to the CMS -/// helper (`cms.rs::clamp_rows_for_hash_bits`). `cols` is assumed already -/// rounded to a power of two. Returns at least 1. -fn clamp_rows_for_hash_bits(rows: usize, cols: usize) -> usize { - let bits_per_row = cols.trailing_zeros() as usize; - if bits_per_row == 0 { - return rows.max(1); - } - let max_rows = (MAX_ROW_HASH_BITS / bits_per_row).max(1); - rows.min(max_rows).max(1) -} - -/// CountSketch wrapper. +/// Count Sketch wrapper. Keys are raw bytes. pub struct CountSketchWrapper { - sk: CountSketch, - rows: usize, - cols: usize, - /// Outbound wire format for this series' snapshots/deltas. Baked from - /// `cfg.encoding` by the OTAP sketch factory. - wire_encoding: Encoding, + sk: Cs, } impl CountSketchWrapper { - /// Construct a CountSketch with the given dimensions. - /// - /// A non-power-of-two `cols` and the narrow-hash condition - /// (`rows * log2(cols) > 64`) are invalid. `new()` returns `Self` - /// (not `Result`), so instead of rejecting we apply the SAME - /// normalization as `CMSWrapper::new`: round `cols` up to the next - /// power of two and clamp `rows` by the 64-bit per-item hash budget. - /// - /// Any VALID config (power-of-two `cols` within the budget — e.g. the - /// canonical `2048` / depth `4`) is left as-is; an invalid config is - /// repaired here identically rather than panicking, preserving - /// byte-parity. + /// Construct a Count Sketch with `rows` hash rows (clamped to + /// `1..=MATRIX_MAX_ROWS`) and `cols` columns. pub fn new(rows: usize, cols: usize) -> Self { - let cols = cols.max(1).next_power_of_two(); - let rows = clamp_rows_for_hash_bits(rows, cols); Self { - sk: CountSketch::new(rows, cols), - rows, - cols, - wire_encoding: Encoding::ProtoFull, + sk: Cs::with_dimensions(rows.clamp(1, MATRIX_MAX_ROWS), cols.max(1)), } } - /// Set the outbound wire format (proto vs msgpack). Builder form used - /// by the OTAP sketch factory to bake in `cfg.encoding`. - pub fn with_wire_encoding(mut self, encoding: Encoding) -> Self { - self.wire_encoding = encoding; - self + /// Decode a full ASAPv1 snapshot. + pub fn from_snapshot(bytes: &[u8]) -> Result { + Cs::deserialize_from_bytes(bytes) + .map(|sk| Self { sk }) + .map_err(|e| PrecomputeError::Other(format!("CountSketchWrapper decode: {e}"))) } - /// Insert a string-keyed observation. - pub fn update(&mut self, key: &str, value: f64) { - self.sk.update(key, value); + /// Add `count` occurrences of `key`. + pub fn update(&mut self, key: &[u8], count: i64) { + self.sk.insert_many(&DataInput::Bytes(key), count); } - /// Borrow the underlying `CountSketch`. - pub fn inner(&self) -> &CountSketch { + /// Borrow the underlying sketch. + pub fn inner(&self) -> &Cs { &self.sk } - - fn build_state(&self) -> CountSketchState { - // Emit packed sint64 `counts_int` (Opt-2: 4–8× smaller than f64 - // for typical small-integer counter values) and per-row L2 - // norms derived as `l2[r] = sum_c counts[r][c]^2`. Both fields - // are required for cross-language byte parity against the golden - // fixture; without them the envelope diverges in counter_type, - // counts_*, and l2 simultaneously. - let mut counts_int = Vec::with_capacity(self.rows * self.cols); - let mut l2 = Vec::with_capacity(self.rows); - for row in self.sk.matrix.iter().take(self.rows) { - let mut row_l2 = 0.0f64; - for &cell in row.iter().take(self.cols) { - counts_int.push(cell as i64); - row_l2 += cell * cell; - } - l2.push(row_l2); - } - CountSketchState { - rows: self.rows as u32, - cols: self.cols as u32, - counter_type: CounterType::Int64 as i32, - counts_int, - counts_float: Vec::new(), - l2, - topk: None, - } - } - - fn encode_envelope(&self) -> Vec { - let env = ProtoEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(sketch_envelope::SketchState::CountSketch( - self.build_state(), - )), - }; - let mut buf = Vec::with_capacity(env.encoded_len()); - env.encode(&mut buf).expect("prost encode"); - buf - } - - fn decode_envelope(bytes: &[u8]) -> Result { - let env = ProtoEnvelope::decode(bytes) - .map_err(|e| PrecomputeError::Other(format!("CountSketchWrapper decode: {e}")))?; - let state = match env.sketch_state { - Some(sketch_envelope::SketchState::CountSketch(s)) => s, - _ => { - return Err(PrecomputeError::Other( - "CountSketchWrapper: envelope did not carry CountSketchState".into(), - )); - } - }; - let rows = state.rows as usize; - let cols = state.cols as usize; - let mut matrix = vec![vec![0.0f64; cols]; rows]; - if !state.counts_float.is_empty() { - for (r, row) in matrix.iter_mut().enumerate().take(rows) { - for (c, cell) in row.iter_mut().enumerate().take(cols) { - let idx = r * cols + c; - if idx < state.counts_float.len() { - *cell = state.counts_float[idx]; - } - } - } - } else if !state.counts_int.is_empty() { - for (r, row) in matrix.iter_mut().enumerate().take(rows) { - for (c, cell) in row.iter_mut().enumerate().take(cols) { - let idx = r * cols + c; - if idx < state.counts_int.len() { - *cell = state.counts_int[idx] as f64; - } - } - } - } - Ok(CountSketch::from_legacy_matrix(matrix, rows, cols)) - } - - /// Whether the sketch matrix is all zero. - fn is_empty(&self) -> bool { - self.sk - .matrix - .iter() - .all(|row| row.iter().all(|&v| v == 0.0)) - } } impl Sketch for CountSketchWrapper { fn snapshot(&self) -> Result, PrecomputeError> { - if self.is_empty() { + if self.sk.as_storage().as_slice().iter().all(|&c| c == 0) { return Ok(Vec::new()); } - if self.wire_encoding.is_msgpack() { - return self.sk.to_msgpack().map_err(|e| { - PrecomputeError::Other(format!("CountSketchWrapper msgpack snapshot: {e}")) - }); - } - Ok(self.encode_envelope()) - } - - fn compute_delta_against( - &self, - prev: &[u8], - threshold: u64, - ) -> Result { - // Decode the prior snapshot envelope, then diff via - // `asap_sketchlib`'s `CountSketch::compute_delta`. On an empty / - // undecodable prior, or an empty current sketch, fall back to a - // full snapshot so the emit path always produces a valid payload. - // - // Under per-window delta-against-empty (the snapshot cache resets - // the cached base to the empty-sketch snapshot at each window - // close), `prev` decodes to an empty `CountSketch`, so the computed - // delta IS this window's full (signed) per-cell matrix encoded as - // cell deltas — no cross-window subtraction. - if self.is_empty() { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - if prev.is_empty() { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - if self.wire_encoding.is_msgpack() { - let prev_sk = match CountSketch::from_msgpack(prev) { - Ok(sk) => sk, - Err(_) => { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - }; - return match self.sk.compute_delta_msgpack(&prev_sk, threshold as f64) { - Ok(delta) => Ok(DeltaResult { - payload: delta, - is_full: false, - }), - Err(_) => { - let full = self.snapshot()?; - Ok(DeltaResult { - payload: full, - is_full: true, - }) - } - }; - } - let prev_sk = match Self::decode_envelope(prev) { - Ok(sk) => sk, - Err(_) => { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - }; - match self.sk.compute_delta(&prev_sk, threshold as f64) { - Ok(delta) => Ok(DeltaResult { - payload: delta, - is_full: false, - }), - Err(_) => { - let full = self.snapshot()?; - Ok(DeltaResult { - payload: full, - is_full: true, - }) - } - } - } - - fn apply_delta(&mut self, payload: &[u8]) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - // Dispatch on payload shape. A full-state envelope (the - // `SketchEnvelope{CountSketchState}` wire format) takes the decode + - // merge path; otherwise the payload is a `CountSketchDelta` proto - // and is applied additively via `apply_delta_bytes`. - if let Ok(other) = Self::decode_envelope(payload) { - return self - .sk - .merge(&other) - .map_err(|e| PrecomputeError::Other(format!("CountSketchWrapper merge: {e}"))); - } self.sk - .apply_delta_bytes(payload) - .map_err(|e| PrecomputeError::Other(format!("CountSketchWrapper apply_delta: {e}"))) - } - - fn apply_delta_encoded( - &mut self, - payload: &[u8], - encoding: Encoding, - ) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - match encoding { - Encoding::MsgpackDelta => self.sk.apply_delta_msgpack_bytes(payload).map_err(|e| { - PrecomputeError::Other(format!("CountSketchWrapper msgpack apply_delta: {e}")) - }), - Encoding::Msgpack => { - let other = CountSketch::from_msgpack(payload).map_err(|e| { - PrecomputeError::Other(format!("CountSketchWrapper msgpack decode: {e}")) - })?; - self.sk - .merge(&other) - .map_err(|e| PrecomputeError::Other(format!("CountSketchWrapper merge: {e}"))) - } - _ => self.apply_delta(payload), - } + .serialize_to_bytes() + .map_err(|e| PrecomputeError::Other(format!("CountSketchWrapper snapshot: {e}"))) } - fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError> { - let bytes = other.snapshot()?; + fn merge_snapshot(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError> { if bytes.is_empty() { return Ok(()); } - // `other.snapshot()` is in that wrapper's `wire_encoding`; same - // Precompute ⇒ same encoding as ours, so decode by our own tag. - let decoded = if self.wire_encoding.is_msgpack() { - CountSketch::from_msgpack(&bytes).map_err(|e| { - PrecomputeError::Other(format!("CountSketchWrapper msgpack merge decode: {e}")) - })? - } else { - Self::decode_envelope(&bytes)? - }; - self.sk - .merge(&decoded) - .map_err(|e| PrecomputeError::Other(format!("CountSketchWrapper merge: {e}"))) + let other = Self::from_snapshot(bytes)?; + if (self.sk.rows(), self.sk.cols()) != (other.sk.rows(), other.sk.cols()) { + return Err(PrecomputeError::Other( + "CountSketchWrapper merge: dimension mismatch".into(), + )); + } + self.sk.merge(&other.sk); + Ok(()) } fn reset(&mut self) { - self.sk = CountSketch::new(self.rows, self.cols); - } - - fn delta_against_empty_base(&self) -> Result>, PrecomputeError> { - // CountSketch opts in to per-window deltas. After a window-close - // emit the snapshot cache - // caches THIS — the encoded envelope of an EMPTY CountSketch of the - // same dimensions — so the next window's `compute_delta_against` - // diffs against empty and emits that window's own (signed) per-cell - // matrix as a delta (no cross-window subtraction). - // - // We encode the empty envelope rather than returning - // `Sketch::snapshot()` of an empty sketch, because the latter - // short-circuits to empty bytes (the runtime drops empty - // payloads), and empty bytes would make `compute_delta_against` - // fall back to a full snapshot instead of a delta. - if self.wire_encoding.is_msgpack() { - let empty = CountSketch::new(self.rows, self.cols); - let bytes = empty.to_msgpack().map_err(|e| { - PrecomputeError::Other(format!("CountSketchWrapper msgpack empty base: {e}")) - })?; - return Ok(Some(bytes)); - } - let empty = CountSketchWrapper::new(self.rows, self.cols); - Ok(Some(empty.encode_envelope())) + self.sk = Cs::with_dimensions(self.sk.rows(), self.sk.cols()); } fn as_any_mut(&mut self) -> &mut dyn std::any::Any { @@ -358,38 +77,23 @@ impl Sketch for CountSketchWrapper { impl FrequencySketch for CountSketchWrapper { fn estimate_count(&self, key: &[u8]) -> f64 { - if key.is_empty() { - return 0.0; - } - let s = std::str::from_utf8(key).unwrap_or_default(); - if s.is_empty() { - return 0.0; - } - self.sk.estimate(s) + self.sk.estimate(&DataInput::Bytes(key)) } + /// A plain Count Sketch does not track keys. fn top_k(&self, _k: usize) -> Vec { - // The wire-format `CountSketch` doesn't carry a TopK heap. - // Returning empty matches the behavior when `TopK == nil` - // (the legacy CMS processor never queries TopK; the CountSketch - // wrapper exposes TopK only when the underlying sketch tracks it). Vec::new() } } -/// Observer that routes observations into the wrapper, counting the -/// per-attribute-set frequency. +/// Observer counting each observation's key. /// -/// The key is the observation's `bytes` field when present (preserves -/// pre-shaped callers / unit tests), else the full label set via -/// [`crate::matchers::attributes_key`], falling back to -/// [`Self::default_key`] only when there are no labels either. The weight -/// is the value's `float` when a pre-shaped `bytes` key is present -/// (existing-test compat) or `1.0` for the OTAP labels path. Float-kind -/// input is accepted. +/// The key is the observation's `bytes` field when present (weight: the +/// value's `float`, rounded), else the full label set via +/// [`crate::matchers::attributes_key`] (weight 1), falling back to +/// [`Self::default_key`] when there are no labels either. pub struct CountSketchObserver { - /// Default key used when the observation has neither a `bytes` - /// field nor any labels. + /// Key used when the observation has neither `bytes` nor labels. pub default_key: String, } @@ -403,21 +107,17 @@ impl SketchObserver for CountSketchObserver { "CountSketchObserver: sketch is not a CountSketchWrapper".into(), ) })?; - let (key_str, weight): (String, f64) = if !obs.value.bytes.is_empty() { - ( - String::from_utf8_lossy(&obs.value.bytes).into_owned(), - obs.value.float, - ) + if !obs.value.bytes.is_empty() { + w.update(&obs.value.bytes, obs.value.float.round() as i64); + return Ok(()); + } + let attr_key = crate::matchers::attributes_key(&obs.labels, &[]); + let key = if attr_key.is_empty() { + &self.default_key } else { - let attr_key = crate::matchers::attributes_key(&obs.labels, &[]); - let key = if !attr_key.is_empty() { - attr_key - } else { - self.default_key.clone() - }; - (key, 1.0) + &attr_key }; - w.update(&key_str, weight); + w.update(key.as_bytes(), 1); Ok(()) } } @@ -428,48 +128,42 @@ mod tests { #[test] fn new_wrapper_is_empty() { - let w = CountSketchWrapper::new(4, 32); - assert_eq!(w.snapshot().unwrap().len(), 0); + assert!(CountSketchWrapper::new(4, 32) + .snapshot() + .unwrap() + .is_empty()); } #[test] fn update_then_estimate() { let mut w = CountSketchWrapper::new(8, 64); - for _ in 0..100 { - w.update("hot-key", 1.0); - } - for _ in 0..5 { - w.update("cold-key", 1.0); - } + w.update(b"hot-key", 100); + w.update(b"cold-key", 5); let hot = w.estimate_count(b"hot-key"); let cold = w.estimate_count(b"cold-key"); - // Median-of-rows estimator can over/undercount but should - // place "hot" well above "cold". assert!(hot.abs() > cold.abs(), "hot={hot} cold={cold}"); } #[test] - fn snapshot_roundtrip_preserves_matrix() { + fn snapshot_round_trips_through_asapv1() { let mut w = CountSketchWrapper::new(4, 8); - w.update("k", 1.0); + w.update(b"k", 1); let bytes = w.snapshot().unwrap(); - let decoded = CountSketchWrapper::decode_envelope(&bytes).unwrap(); - assert_eq!(decoded.matrix, w.sk.matrix); + let decoded = CountSketchWrapper::from_snapshot(&bytes).unwrap(); + assert_eq!( + decoded.inner().as_storage().as_slice(), + w.inner().as_storage().as_slice() + ); } - // B6 regression: on the OTAP edge, observations arrive Float-kind - // with empty bytes. CountSketch must count the per-attribute-set - // key with weight 1.0, NOT degenerately count the metric NAME via - // default_key. + /// OTAP scalar rows arrive Float-kind with empty bytes; Count Sketch + /// must count the attribute set with weight 1, not `default_key`. #[test] fn observe_counts_attribute_set_not_metric_name() { use crate::matchers::attributes_key; - use crate::observation::{KeyValue, Observation, ObservationValue}; - use crate::precompute::SketchObserver; + use crate::observation::{KeyValue, ObservationValue}; let mut sketch = CountSketchWrapper::new(8, 64); - // default_key set to the metric name to prove we do NOT fall - // back to it when labels are present. let observer = CountSketchObserver { default_key: "events_per_path".into(), }; @@ -481,26 +175,14 @@ mod tests { "events_per_path", vec![], labels.clone(), - // Float-kind, empty bytes — OTAP scalar row shape. ObservationValue::float(1.0), ); observer.observe(&mut sketch, &obs).expect("observe"); } - let attr_key = attributes_key(&labels, &[]); assert_eq!(attr_key, "path=/api;"); let counted = sketch.estimate_count(attr_key.as_bytes()); - // CountSketch's median-of-rows estimator can over/undercount, - // but the attribute-set key should be near N. - assert!( - counted >= (n as f64) * 0.5, - "attribute-set key undercounted: {counted} (n={n})" - ); - // The metric NAME (== default_key) must NOT be the counted key. - let metric_count = sketch.estimate_count(b"events_per_path"); - assert!( - metric_count.abs() < counted, - "metric name was counted ({metric_count}) vs attr set ({counted})" - ); + assert!(counted >= (n as f64) * 0.5, "counted={counted}"); + assert!(sketch.estimate_count(b"events_per_path").abs() < counted); } } diff --git a/asap-precompute-rs/src/sketches/ddsketch.rs b/asap-precompute-rs/src/sketches/ddsketch.rs index 6e96350..52d6e4b 100644 --- a/asap-precompute-rs/src/sketches/ddsketch.rs +++ b/asap-precompute-rs/src/sketches/ddsketch.rs @@ -1,37 +1,16 @@ -//! DDSketch wrapper over [`asap_sketchlib::DdSketch`]. +//! DDSketch wrapper over [`asap_sketchlib::DDSketch`]. //! -//! Adapts the wire-format-aligned `DdSketch` struct to the -//! host-neutral [`Sketch`] + [`QuantileSketch`] interfaces. +//! Implements [`Sketch`] + [`QuantileSketch`]. Snapshots are sketchlib's +//! ASAPv1 envelope (kind `0x05 0x00`). -use asap_sketchlib::proto::sketchlib::{ - sketch_envelope, DdSketchState, SketchEnvelope as ProtoEnvelope, -}; -use asap_sketchlib::{DdSketch, MessagePackCodec}; -use prost::Message; +use asap_sketchlib::DDSketch; -use crate::envelope::Encoding; use crate::observation::{KeyValue, Observation}; -use crate::precompute::{ - DeltaResult, EstimatePoint, PrecomputeError, QuantileSketch, Sketch, SketchObserver, -}; +use crate::precompute::{EstimatePoint, PrecomputeError, QuantileSketch, Sketch, SketchObserver}; /// DDSketch wrapper. -/// -/// Owns one `asap_sketchlib::DdSketch` (the wire-format-aligned variant -/// with public-field `store_counts` / `store_offset` / aggregates). -/// -/// # Snapshot format -/// -/// `Snapshot` produces a `prost`-encoded `SketchEnvelope` carrying the -/// inner `DDSketchState` proto. pub struct DDSketchWrapper { - sk: DdSketch, - alpha: f64, - /// Outbound wire format for this series' snapshots/deltas. Baked from - /// `cfg.encoding` by the OTAP sketch factory; `snapshot` / - /// `compute_delta_against` / `delta_against_empty_base` honor it to - /// pick proto vs msgpack. - wire_encoding: Encoding, + sk: DDSketch, } impl DDSketchWrapper { @@ -39,249 +18,53 @@ impl DDSketchWrapper { /// `alpha` must satisfy `0 < alpha < 1`. pub fn new(alpha: f64) -> Self { Self { - sk: DdSketch::new(alpha), - alpha, - wire_encoding: Encoding::ProtoFull, + sk: DDSketch::new(alpha), } } - /// Set the outbound wire format (proto vs msgpack). Builder form used - /// by the OTAP sketch factory to bake in `cfg.encoding`. - pub fn with_wire_encoding(mut self, encoding: Encoding) -> Self { - self.wire_encoding = encoding; - self + /// Decode a full ASAPv1 snapshot. + pub fn from_snapshot(bytes: &[u8]) -> Result { + DDSketch::deserialize_from_bytes(bytes) + .map(|sk| Self { sk }) + .map_err(|e| PrecomputeError::Other(format!("DDSketchWrapper decode: {e}"))) } - /// Insert a single positive observation. + /// Insert a single observation. Non-finite values are ignored. pub fn update(&mut self, value: f64) { - self.sk.update(value); + self.sk.add(&value); } - /// Borrow the underlying `DdSketch`. - pub fn inner(&self) -> &DdSketch { + /// Borrow the underlying sketch. + pub fn inner(&self) -> &DDSketch { &self.sk } - - fn build_state(&self) -> DdSketchState { - // asap_sketchlib owns the proto mapping, including the negative-value - // and zero stores and the gamma-roundtripped alpha. - self.sk.to_proto() - } - - fn encode_envelope(&self) -> Vec { - let env = ProtoEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(sketch_envelope::SketchState::Ddsketch(self.build_state())), - }; - let mut buf = Vec::with_capacity(env.encoded_len()); - env.encode(&mut buf).expect("prost encode"); - buf - } - - fn decode_envelope(bytes: &[u8]) -> Result { - let env = ProtoEnvelope::decode(bytes) - .map_err(|e| PrecomputeError::Other(format!("DDSketchWrapper decode: {e}")))?; - let state = match env.sketch_state { - Some(sketch_envelope::SketchState::Ddsketch(s)) => s, - _ => { - return Err(PrecomputeError::Other( - "DDSketchWrapper: envelope did not carry DDSketchState".into(), - )); - } - }; - if !(state.alpha > 0.0 && state.alpha < 1.0) { - return Err(PrecomputeError::Other(format!( - "DDSketchWrapper: alpha {} out of range", - state.alpha - ))); - } - // `DdSketchState` carries no count/sum/min/max scalars: the count is - // the sum of all stores and min/max are bucket-estimated, within - // DDSketch's alpha relative-accuracy bound. - Ok(DdSketch::from_proto(state)) - } } impl Sketch for DDSketchWrapper { fn snapshot(&self) -> Result, PrecomputeError> { - if self.sk.total_count() == 0 { - // Empty sketch produces empty snapshot — the runtime drops - // empty payloads rather than emitting zero-byte envelopes. + if self.sk.get_count() == 0 { return Ok(Vec::new()); } - if self.wire_encoding.is_msgpack() { - return self.sk.to_msgpack().map_err(|e| { - PrecomputeError::Other(format!("DDSketchWrapper msgpack snapshot: {e}")) - }); - } - Ok(self.encode_envelope()) - } - - fn compute_delta_against( - &self, - prev: &[u8], - threshold: u64, - ) -> Result { - // Decode the prior snapshot envelope, then diff via - // `asap_sketchlib`'s `DdSketch::compute_delta`. On an empty / - // undecodable prior, or an empty current sketch, fall back to a - // full snapshot so the emit path always produces a valid payload. - // - // Under per-window delta-against-empty (the snapshot cache resets - // the cached base to the empty-sketch snapshot at each window - // close), `prev` decodes to an empty `DdSketch`, so the computed - // delta IS this window's full bucket store encoded as bucket - // deltas — no cross-window subtraction. - if self.sk.total_count() == 0 { - // Empty current sketch produces an empty snapshot; the - // runtime drops empty payloads. - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - if prev.is_empty() { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - if self.wire_encoding.is_msgpack() { - // Msgpack path: the cached base is a msgpack-encoded full - // `DdSketch`; diff against it and emit the sparse msgpack delta. - let prev_sk = match DdSketch::from_msgpack(prev) { - Ok(sk) => sk, - Err(_) => { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - }; - let delta = self.sk.compute_delta_msgpack(&prev_sk, threshold); - return Ok(DeltaResult { - payload: delta, - is_full: false, - }); - } - let prev_sk = match Self::decode_envelope(prev) { - Ok(sk) => sk, - Err(_) => { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - }; - let delta = self.sk.compute_delta(&prev_sk, threshold); - Ok(DeltaResult { - payload: delta, - is_full: false, - }) - } - - fn apply_delta(&mut self, delta: &[u8]) -> Result<(), PrecomputeError> { - if delta.is_empty() { - return Ok(()); - } - // Dispatch on payload shape. A full-state envelope (the - // `SketchEnvelope{DDSketchState}` wire format) takes the decode + - // merge path; otherwise the payload is a `DDSketchDelta` proto and - // is applied additively. - if let Ok(other) = Self::decode_envelope(delta) { - return self - .sk - .merge(&other) - .map_err(|e| PrecomputeError::Other(format!("DDSketchWrapper merge: {e}"))); - } self.sk - .apply_delta_bytes(delta) - .map_err(|e| PrecomputeError::Other(format!("DDSketchWrapper apply_delta: {e}"))) - } - - fn apply_delta_encoded( - &mut self, - payload: &[u8], - encoding: Encoding, - ) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - match encoding { - Encoding::MsgpackDelta => self.sk.apply_delta_msgpack_bytes(payload).map_err(|e| { - PrecomputeError::Other(format!("DDSketchWrapper msgpack apply_delta: {e}")) - }), - Encoding::Msgpack => { - let other = DdSketch::from_msgpack(payload).map_err(|e| { - PrecomputeError::Other(format!("DDSketchWrapper msgpack decode: {e}")) - })?; - self.sk - .merge(&other) - .map_err(|e| PrecomputeError::Other(format!("DDSketchWrapper merge: {e}"))) - } - _ => self.apply_delta(payload), - } + .serialize_to_bytes() + .map_err(|e| PrecomputeError::Other(format!("DDSketchWrapper snapshot: {e}"))) } - fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError> { - // The runtime always merges sketches owned by the same - // Precompute (same alpha). Our trait is generic, so we - // round-trip through the snapshot bytes. - let bytes = other.snapshot()?; + fn merge_snapshot(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError> { if bytes.is_empty() { return Ok(()); } - // `other.snapshot()` is in that wrapper's `wire_encoding`; same - // Precompute ⇒ same encoding as ours, so decode by our own tag. - let decoded = if self.wire_encoding.is_msgpack() { - DdSketch::from_msgpack(&bytes).map_err(|e| { - PrecomputeError::Other(format!("DDSketchWrapper msgpack merge decode: {e}")) - })? - } else { - Self::decode_envelope(&bytes)? - }; self.sk - .merge(&decoded) + .merge(&Self::from_snapshot(bytes)?.sk) .map_err(|e| PrecomputeError::Other(format!("DDSketchWrapper merge: {e}"))) } fn reset(&mut self) { - self.sk = DdSketch::new(self.alpha); - } - - fn delta_against_empty_base(&self) -> Result>, PrecomputeError> { - // DDSketch opts in to per-window deltas. After a window-close - // emit the snapshot cache - // caches THIS — the encoded envelope of an EMPTY DDSketch of the - // same alpha — so the next window's `compute_delta_against` diffs - // against empty and emits that window's own bucket store as a - // delta (no cross-window subtraction). - // - // Note: we deliberately encode the empty envelope rather than - // returning `Sketch::snapshot()` of an empty sketch, because the - // latter short-circuits to empty bytes (the runtime drops empty - // payloads), and empty bytes would make `compute_delta_against` - // fall back to a full snapshot instead of a delta. - if self.wire_encoding.is_msgpack() { - let empty = DdSketch::new(self.alpha); - let bytes = empty.to_msgpack().map_err(|e| { - PrecomputeError::Other(format!("DDSketchWrapper msgpack empty base: {e}")) - })?; - return Ok(Some(bytes)); - } - let empty = DDSketchWrapper::new(self.alpha); - Ok(Some(empty.encode_envelope())) + self.sk = DDSketch::new(self.sk.alpha()); } fn estimate(&self, quantiles: &[f64], _top_k: usize) -> Vec { - if self.sk.total_count() == 0 { + if self.sk.get_count() == 0 { return Vec::new(); } quantiles @@ -300,10 +83,9 @@ impl Sketch for DDSketchWrapper { impl QuantileSketch for DDSketchWrapper { fn quantile(&self, q: f64) -> f64 { - if self.sk.total_count() == 0 { - return f64::NAN; - } - self.sk.quantile(q.clamp(0.0, 1.0)).unwrap_or(f64::NAN) + self.sk + .get_value_at_quantile(q.clamp(0.0, 1.0)) + .unwrap_or(f64::NAN) } } @@ -312,9 +94,12 @@ pub struct DDSketchObserver; impl SketchObserver for DDSketchObserver { fn observe(&self, sketch: &mut dyn Sketch, obs: &Observation) -> Result<(), PrecomputeError> { - // Use a `&mut dyn Sketch -> &mut DDSketchWrapper` downcast via - // the panic-safe method below. - let w = downcast_mut(sketch)?; + let w = sketch + .as_any_mut() + .downcast_mut::() + .ok_or_else(|| { + PrecomputeError::Other("DDSketchObserver: sketch is not a DDSketchWrapper".into()) + })?; match obs.value.kind { crate::observation::ObservationValueKind::Float => { w.update(obs.value.float); @@ -328,74 +113,50 @@ impl SketchObserver for DDSketchObserver { } } -fn downcast_mut(sketch: &mut dyn Sketch) -> Result<&mut DDSketchWrapper, PrecomputeError> { - sketch - .as_any_mut() - .downcast_mut::() - .ok_or_else(|| { - PrecomputeError::Other("DDSketchObserver: sketch is not a DDSketchWrapper".into()) - }) -} - #[cfg(test)] mod tests { use super::*; + fn filled(range: std::ops::RangeInclusive) -> DDSketchWrapper { + let mut w = DDSketchWrapper::new(0.01); + for i in range { + w.update(i as f64); + } + w + } + #[test] fn new_wrapper_is_empty() { - let w = DDSketchWrapper::new(0.01); - assert_eq!(w.sk.total_count(), 0); - assert_eq!(w.snapshot().unwrap().len(), 0); + assert!(DDSketchWrapper::new(0.01).snapshot().unwrap().is_empty()); } #[test] fn update_then_quantile_within_bound() { - let mut w = DDSketchWrapper::new(0.01); - for i in 1..=100 { - w.update(i as f64); - } - let p50 = w.quantile(0.5); - // Median of [1..=100] is 50 or 51; with α=0.01 the relative - // error is ≤ 1%. + let p50 = filled(1..=100).quantile(0.5); assert!((p50 - 50.0).abs() / 50.0 < 0.05, "p50={p50}"); } #[test] - fn snapshot_decodes_back_to_equivalent_state() { - let mut w = DDSketchWrapper::new(0.01); - for i in 1..=10 { - w.update(i as f64); - } + fn snapshot_round_trips_through_asapv1() { + let w = filled(1..=10); let bytes = w.snapshot().unwrap(); - let decoded = DDSketchWrapper::decode_envelope(&bytes).unwrap(); - // count is recovered exactly by summing the bucket store; the - // count/sum scalars are no longer on the wire. - assert_eq!(decoded.total_count(), w.sk.total_count()); - assert_eq!(decoded.store_counts, w.sk.store_counts); - assert_eq!(decoded.store_offset, w.sk.store_offset); + assert_eq!(&bytes[..6], b"ASAPv1"); + let decoded = DDSketchWrapper::from_snapshot(&bytes).unwrap(); + assert_eq!(decoded.inner().get_count(), 10); + assert_eq!(decoded.snapshot().unwrap(), bytes); } #[test] fn merge_combines_counts() { - let mut a = DDSketchWrapper::new(0.01); - let mut b = DDSketchWrapper::new(0.01); - for i in 1..=5 { - a.update(i as f64); - } - for i in 6..=10 { - b.update(i as f64); - } - let other_bytes = b.snapshot().unwrap(); - a.apply_delta(&other_bytes).unwrap(); - assert_eq!(a.sk.total_count(), 10); + let mut a = filled(1..=5); + a.merge(&filled(6..=10)).unwrap(); + assert_eq!(a.inner().get_count(), 10); } #[test] fn reset_zeros_state() { - let mut w = DDSketchWrapper::new(0.01); - w.update(1.0); - w.update(2.0); + let mut w = filled(1..=2); w.reset(); - assert_eq!(w.sk.total_count(), 0); + assert_eq!(w.inner().get_count(), 0); } } diff --git a/asap-precompute-rs/src/sketches/hll.rs b/asap-precompute-rs/src/sketches/hll.rs index ee013a4..9f121e9 100644 --- a/asap-precompute-rs/src/sketches/hll.rs +++ b/asap-precompute-rs/src/sketches/hll.rs @@ -1,294 +1,74 @@ -//! HLL wrapper over [`asap_sketchlib::HllSketch`]. +//! HyperLogLog wrapper over sketchlib's precision-14 Ertl-MLE +//! [`asap_sketchlib::HyperLogLog`]. //! -//! HLL is the canonical [`CardinalitySketch`] implementation in this -//! crate. +//! Implements [`Sketch`] + [`CardinalitySketch`]. Snapshots are sketchlib's +//! ASAPv1 envelope (kind `0x01 0x02`). -use asap_sketchlib::proto::sketchlib::{ - sketch_envelope, HllVariant, HyperLogLogState, SketchEnvelope as ProtoEnvelope, -}; -use asap_sketchlib::{HllSketch, HllVariant as RsHllVariant, MessagePackCodec}; -use prost::Message; +use asap_sketchlib::{DataInput, ErtlMLE, HyperLogLog}; -use crate::envelope::Encoding; use crate::observation::Observation; use crate::precompute::{ - CardinalitySketch, DeltaResult, EstimatePoint, PrecomputeError, Sketch, SketchObserver, + CardinalitySketch, EstimatePoint, PrecomputeError, Sketch, SketchObserver, }; -/// HLL wrapper. Owns one `asap_sketchlib::HllSketch`. +/// HyperLogLog wrapper (2^14 registers, Ertl MLE estimator). +#[derive(Default)] pub struct HLLWrapper { - sk: HllSketch, - variant: RsHllVariant, - precision: u32, - /// Outbound wire format for this series' snapshots/deltas. Baked from - /// `cfg.encoding` by the OTAP sketch factory. - wire_encoding: Encoding, + sk: HyperLogLog, } impl HLLWrapper { - /// Construct an empty HLL with the given variant and precision. - pub fn new(variant: RsHllVariant, precision: u32) -> Self { - Self { - sk: HllSketch::new(variant, precision), - variant, - precision, - wire_encoding: Encoding::ProtoFull, - } + /// Construct an empty HLL. + pub fn new() -> Self { + Self::default() } - /// Set the outbound wire format (proto vs msgpack). Builder form used - /// by the OTAP sketch factory to bake in `cfg.encoding`. - pub fn with_wire_encoding(mut self, encoding: Encoding) -> Self { - self.wire_encoding = encoding; - self + /// Decode a full ASAPv1 snapshot. + pub fn from_snapshot(bytes: &[u8]) -> Result { + HyperLogLog::::deserialize_from_bytes(bytes) + .map(|sk| Self { sk }) + .map_err(|e| PrecomputeError::Other(format!("HLLWrapper decode: {e}"))) } - /// Insert a byte slice, routed to a hashed-bytes path. + /// Insert a byte-slice key. pub fn update(&mut self, value: &[u8]) { - self.sk.update(value); + self.sk.insert(&DataInput::Bytes(value)); } - /// Borrow the underlying `HllSketch`. - pub fn inner(&self) -> &HllSketch { + /// Borrow the underlying sketch. + pub fn inner(&self) -> &HyperLogLog { &self.sk } - fn build_state(&self) -> HyperLogLogState { - let proto_variant = match self.variant { - RsHllVariant::Unspecified => HllVariant::Unspecified as i32, - RsHllVariant::Regular => HllVariant::Regular as i32, - RsHllVariant::Datafusion => HllVariant::ErtlMle as i32, - RsHllVariant::Hip => HllVariant::Hip as i32, - }; - HyperLogLogState { - variant: proto_variant, - precision: self.precision, - registers: self.sk.registers.clone(), - hip_kxq0: self.sk.hip_kxq0, - hip_kxq1: self.sk.hip_kxq1, - hip_est: self.sk.hip_est, - // Emit the dense register encoding (tag 3); the sparse - // encoding (tag 7) is left unset, matching the existing - // wire form. - registers_sparse: None, - } - } - - fn encode_envelope(&self) -> Vec { - let env = ProtoEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(sketch_envelope::SketchState::Hll(self.build_state())), - }; - let mut buf = Vec::with_capacity(env.encoded_len()); - env.encode(&mut buf).expect("prost encode"); - buf - } - - fn decode_envelope(bytes: &[u8]) -> Result { - let env = ProtoEnvelope::decode(bytes) - .map_err(|e| PrecomputeError::Other(format!("HLLWrapper decode: {e}")))?; - let state = match env.sketch_state { - Some(sketch_envelope::SketchState::Hll(s)) => s, - _ => { - return Err(PrecomputeError::Other( - "HLLWrapper: envelope did not carry HyperLogLogState".into(), - )); - } - }; - let variant = match HllVariant::try_from(state.variant) { - Ok(HllVariant::Regular) => RsHllVariant::Regular, - Ok(HllVariant::ErtlMle) => RsHllVariant::Datafusion, - Ok(HllVariant::Hip) => RsHllVariant::Hip, - _ => RsHllVariant::Unspecified, - }; - Ok(HllSketch::from_raw( - variant, - state.precision, - state.registers, - state.hip_kxq0, - state.hip_kxq1, - state.hip_est, - )) + fn is_empty(&self) -> bool { + self.sk.registers_as_slice().iter().all(|&r| r == 0) } } impl Sketch for HLLWrapper { fn snapshot(&self) -> Result, PrecomputeError> { - if self.sk.registers.iter().all(|&r| r == 0) { + if self.is_empty() { return Ok(Vec::new()); } - if self.wire_encoding.is_msgpack() { - return self - .sk - .to_msgpack() - .map_err(|e| PrecomputeError::Other(format!("HLLWrapper msgpack snapshot: {e}"))); - } - Ok(self.encode_envelope()) - } - - fn compute_delta_against( - &self, - prev: &[u8], - threshold: u64, - ) -> Result { - // Decode the prior snapshot envelope, then diff via - // `asap_sketchlib`'s `HllSketch::compute_delta` (a register delta - // carrying every register where this window's value exceeds the - // prior's). On an empty / undecodable prior, or an empty current - // sketch, fall back to a full snapshot so the emit path always - // produces a valid payload. - // - // Under per-window delta-against-empty (the snapshot cache resets - // the cached base to the empty-sketch snapshot at each window - // close), `prev` decodes to an empty `HllSketch`, so the register - // delta carries every non-zero register of THIS window — i.e. this - // window's own register state, with no cross-window register-MAX - // leakage. - if self.sk.registers.iter().all(|&r| r == 0) { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - if prev.is_empty() { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - if self.wire_encoding.is_msgpack() { - let prev_sk = match HllSketch::from_msgpack(prev) { - Ok(sk) => sk, - Err(_) => { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - }; - let delta = self.sk.compute_delta_msgpack(&prev_sk, threshold); - return Ok(DeltaResult { - payload: delta, - is_full: false, - }); - } - let prev_sk = match Self::decode_envelope(prev) { - Ok(sk) => sk, - Err(_) => { - let full = self.snapshot()?; - return Ok(DeltaResult { - payload: full, - is_full: true, - }); - } - }; - let delta = self.sk.compute_delta(&prev_sk, threshold); - Ok(DeltaResult { - payload: delta, - is_full: false, - }) - } - - fn apply_delta(&mut self, payload: &[u8]) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - // Dispatch on payload shape. A full-state envelope (the - // `SketchEnvelope{HyperLogLogState}` wire format) takes the decode + - // merge path; otherwise the payload is an `HllDelta` proto and is - // applied via register max-merge through `apply_delta_bytes`. - if let Ok(other) = Self::decode_envelope(payload) { - return self - .sk - .merge(&other) - .map_err(|e| PrecomputeError::Other(format!("HLLWrapper merge: {e}"))); - } self.sk - .apply_delta_bytes(payload) - .map_err(|e| PrecomputeError::Other(format!("HLLWrapper apply_delta: {e}"))) - } - - fn apply_delta_encoded( - &mut self, - payload: &[u8], - encoding: Encoding, - ) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - match encoding { - Encoding::MsgpackDelta => self.sk.apply_delta_msgpack_bytes(payload).map_err(|e| { - PrecomputeError::Other(format!("HLLWrapper msgpack apply_delta: {e}")) - }), - Encoding::Msgpack => { - let other = HllSketch::from_msgpack(payload).map_err(|e| { - PrecomputeError::Other(format!("HLLWrapper msgpack decode: {e}")) - })?; - self.sk - .merge(&other) - .map_err(|e| PrecomputeError::Other(format!("HLLWrapper merge: {e}"))) - } - _ => self.apply_delta(payload), - } + .serialize_to_bytes() + .map_err(|e| PrecomputeError::Other(format!("HLLWrapper snapshot: {e}"))) } - fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError> { - let bytes = other.snapshot()?; + fn merge_snapshot(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError> { if bytes.is_empty() { return Ok(()); } - // `other.snapshot()` is in that wrapper's `wire_encoding`; same - // Precompute ⇒ same encoding as ours, so decode by our own tag. - let decoded = if self.wire_encoding.is_msgpack() { - HllSketch::from_msgpack(&bytes).map_err(|e| { - PrecomputeError::Other(format!("HLLWrapper msgpack merge decode: {e}")) - })? - } else { - Self::decode_envelope(&bytes)? - }; - self.sk - .merge(&decoded) - .map_err(|e| PrecomputeError::Other(format!("HLLWrapper merge: {e}"))) + self.sk.merge(&Self::from_snapshot(bytes)?.sk); + Ok(()) } fn reset(&mut self) { - self.sk = HllSketch::new(self.variant, self.precision); - } - - fn delta_against_empty_base(&self) -> Result>, PrecomputeError> { - // HLL opts in to per-window deltas. After a window-close emit the - // snapshot cache caches THIS — the encoded envelope of an EMPTY HLL - // of the same variant / precision — so the next window's - // `compute_delta_against` diffs against empty and emits that - // window's own register state as a delta. The empty-base reset is - // what makes window-scoped cardinality correct: HLL merges by - // register-wise MAX over a never-reset base, which would over-count - // without the reset. - // - // We encode the empty envelope rather than returning - // `Sketch::snapshot()` of an empty sketch, because the latter - // short-circuits to empty bytes (the runtime drops empty - // payloads), and empty bytes would make `compute_delta_against` - // fall back to a full snapshot instead of a delta. - if self.wire_encoding.is_msgpack() { - let empty = HllSketch::new(self.variant, self.precision); - let bytes = empty.to_msgpack().map_err(|e| { - PrecomputeError::Other(format!("HLLWrapper msgpack empty base: {e}")) - })?; - return Ok(Some(bytes)); - } - let empty = HLLWrapper::new(self.variant, self.precision); - Ok(Some(empty.encode_envelope())) + self.sk = HyperLogLog::::new(); } fn estimate(&self, _quantiles: &[f64], _top_k: usize) -> Vec { - if self.sk.registers.iter().all(|&r| r == 0) { + if self.is_empty() { return Vec::new(); } vec![EstimatePoint { @@ -304,14 +84,14 @@ impl Sketch for HLLWrapper { impl CardinalitySketch for HLLWrapper { fn estimate_cardinality(&self) -> f64 { - self.sk.estimate() + self.sk.estimate() as f64 } } /// Observer routing observations into an [`HLLWrapper`]. /// -/// Accepts both `Bytes` (preferred — opaque key) and `Float` (the -/// float bytes are hashed). +/// Accepts both `Bytes` (preferred — opaque key) and `Float` / `Hash` +/// (their little-endian bytes are hashed). pub struct HLLObserver; impl SketchObserver for HLLObserver { @@ -324,8 +104,7 @@ impl SketchObserver for HLLObserver { })?; match obs.value.kind { crate::observation::ObservationValueKind::Float => { - let bytes = obs.value.float.to_le_bytes(); - w.update(&bytes); + w.update(&obs.value.float.to_le_bytes()); Ok(()) } crate::observation::ObservationValueKind::Bytes => { @@ -333,8 +112,7 @@ impl SketchObserver for HLLObserver { Ok(()) } crate::observation::ObservationValueKind::Hash => { - let bytes = obs.value.hash.to_le_bytes(); - w.update(&bytes); + w.update(&obs.value.hash.to_le_bytes()); Ok(()) } other => Err(PrecomputeError::Other(format!( @@ -349,52 +127,43 @@ impl SketchObserver for HLLObserver { mod tests { use super::*; + fn filled(range: std::ops::Range) -> HLLWrapper { + let mut w = HLLWrapper::new(); + for i in range { + w.update(&i.to_le_bytes()); + } + w + } + #[test] fn new_wrapper_is_empty() { - let w = HLLWrapper::new(RsHllVariant::Regular, 12); - assert_eq!(w.snapshot().unwrap().len(), 0); + let w = HLLWrapper::new(); + assert!(w.snapshot().unwrap().is_empty()); assert_eq!(w.estimate_cardinality(), 0.0); } #[test] fn update_then_estimate() { - let mut w = HLLWrapper::new(RsHllVariant::Regular, 12); - for i in 0..1_000u64 { - w.update(&i.to_le_bytes()); - } - let est = w.estimate_cardinality(); - // Loose bounds — HLL with precision=12 has std error ~1.6%. - assert!(est > 800.0 && est < 1200.0, "est={est}"); + let est = filled(0..1_000).estimate_cardinality(); + assert!(est > 900.0 && est < 1100.0, "est={est}"); } #[test] - fn snapshot_roundtrip_preserves_registers() { - let mut w = HLLWrapper::new(RsHllVariant::Regular, 12); - for i in 0..100u64 { - w.update(&i.to_le_bytes()); - } + fn snapshot_round_trips_through_asapv1() { + let w = filled(0..100); let bytes = w.snapshot().unwrap(); - let decoded = HLLWrapper::decode_envelope(&bytes).unwrap(); - assert_eq!(decoded.registers, w.sk.registers); + let decoded = HLLWrapper::from_snapshot(&bytes).unwrap(); + assert_eq!( + decoded.inner().registers_as_slice(), + w.inner().registers_as_slice() + ); } #[test] fn merge_takes_register_max() { - let mut a = HLLWrapper::new(RsHllVariant::Regular, 12); - let mut b = HLLWrapper::new(RsHllVariant::Regular, 12); - for i in 0..500u64 { - a.update(&i.to_le_bytes()); - } - for i in 250..750u64 { - b.update(&i.to_le_bytes()); - } - let pre = a.estimate_cardinality(); - let other_bytes = b.snapshot().unwrap(); - a.apply_delta(&other_bytes).unwrap(); - let post = a.estimate_cardinality(); - assert!( - post >= pre, - "merged estimate decreased: pre={pre} post={post}" - ); + let mut a = filled(0..500); + a.merge(&filled(250..750)).unwrap(); + let est = a.estimate_cardinality(); + assert!(est > 675.0 && est < 825.0, "est={est}"); } } diff --git a/asap-precompute-rs/src/sketches/kll.rs b/asap-precompute-rs/src/sketches/kll.rs index 483bb4d..f6b298f 100644 --- a/asap-precompute-rs/src/sketches/kll.rs +++ b/asap-precompute-rs/src/sketches/kll.rs @@ -1,112 +1,51 @@ //! KLL wrapper over [`asap_sketchlib::KLL`]. //! -//! KLL uses random compaction so the wrapper takes an explicit -//! (optional) seed for deterministic byte-identical replay -//! (`init_with_seed` lands a deterministic compaction RNG). +//! Implements [`Sketch`] + [`QuantileSketch`]. Snapshots are sketchlib's +//! ASAPv1 envelope. KLL uses random compaction, so the wrapper takes an +//! optional seed for deterministic compaction. use asap_sketchlib::sketches::KLL; -use prost::Message; -use asap_sketchlib::proto::sketchlib::{ - sketch_envelope, CoinState, KllState, SketchEnvelope as ProtoEnvelope, -}; - -use crate::envelope::Encoding; use crate::observation::{KeyValue, Observation}; -use crate::precompute::{ - DeltaResult, EstimatePoint, PrecomputeError, QuantileSketch, Sketch, SketchObserver, -}; +use crate::precompute::{EstimatePoint, PrecomputeError, QuantileSketch, Sketch, SketchObserver}; -/// KLL wrapper. -/// -/// Owns one `asap_sketchlib::KLL`. Construction takes -/// `(k, optional seed)`; the seed is forwarded to -/// [`asap_sketchlib::KLL::init_kll_with_seed`] when provided so two -/// wrappers built with the same seed and fed the same input produce -/// byte-identical state. +/// KLL wrapper over `asap_sketchlib::KLL`. pub struct KLLWrapper { sk: KLL, - k: i32, - seed: Option, - /// Replay history used only by the legacy proto path. The ASAPv1 path - /// retains the compacted KLL state alone, keeping pane memory independent - /// of the number of incoming observations. - history: Vec, - /// Outbound wire format. KLL is full-only, so msgpack means the - /// self-describing ASAPv1 form; proto means the legacy `KllState` envelope. - /// Baked from `cfg.encoding` by the OTAP sketch factory. - wire_encoding: Encoding, } impl KLLWrapper { /// Construct an empty KLL with accuracy parameter `k`. /// `seed = Some(s)` enables deterministic compaction. pub fn new(k: i32, seed: Option) -> Self { - Self { - sk: build_kll(k, seed), - k, - seed, - history: Vec::new(), - wire_encoding: Encoding::ProtoFull, - } - } - - /// Set the outbound wire format (proto vs msgpack). Builder form used - /// by the OTAP sketch factory to bake in `cfg.encoding`. - pub fn with_wire_encoding(mut self, encoding: Encoding) -> Self { - self.wire_encoding = encoding; - self - } - - /// Canonical self-describing ASAPv1 KLL envelope. - /// - /// `asap_sketchlib` owns the framing, metadata, and payload encoding; - /// in particular these bytes start with the `b"ASAPv1"` magic number. - fn encode_msgpack(&self) -> Result, PrecomputeError> { - self.sk - .serialize_to_bytes() - .map_err(|e| PrecomputeError::Other(format!("KLLWrapper ASAPv1 serialize: {e}"))) + let sk = match seed { + Some(s) => KLL::init_kll_with_seed(k, s), + None => KLL::init_kll(k), + }; + Self { sk } } - /// Merge a self-describing ASAPv1 KLL frame in. - /// - /// Uses sketchlib's level-aware merge so each retained item keeps its - /// `2^level` weight after compaction. - fn merge_msgpack(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError> { - let other: KLL = KLL::deserialize_from_bytes(bytes) - .map_err(|e| PrecomputeError::Other(format!("KLLWrapper ASAPv1 decode: {e}")))?; - self.sk.merge(&other); - Ok(()) + /// Decode a full ASAPv1 snapshot. + pub fn from_snapshot(bytes: &[u8]) -> Result { + KLL::deserialize_from_bytes(bytes) + .map(|sk| Self { sk }) + .map_err(|e| PrecomputeError::Other(format!("KLLWrapper decode: {e}"))) } - /// Insert a single observation. + /// Insert a single observation. Non-finite values are ignored. pub fn update(&mut self, value: f64) { if value.is_finite() { - if !self.wire_encoding.is_msgpack() { - self.history.push(value); - } self.sk.update(&value); } } - /// Insert a batch without forcing callers to materialize a value vector. - /// The upstream compactor currently exposes scalar updates, so this keeps - /// the loop local and reserves the replay history once per batch. + /// Insert a batch of observations. pub fn update_batch(&mut self, values: I) where I: IntoIterator, { - let values = values.into_iter(); - if !self.wire_encoding.is_msgpack() { - self.history.reserve(values.size_hint().0); - } for value in values { - if value.is_finite() { - if !self.wire_encoding.is_msgpack() { - self.history.push(value); - } - self.sk.update(&value); - } + self.update(value); } } @@ -114,146 +53,30 @@ impl KLLWrapper { pub fn inner(&self) -> &KLL { &self.sk } - - fn build_state(&self) -> KllState { - // Wire-format-aligned: read directly from the compactor via - // the `wire_*` accessors added in asap_sketchlib so the - // emitted `KllState.levels` / `items` / `coin` bytes match the - // portable serialization output. - let (state, bit_cache, remaining_bits) = self.sk.wire_coin(); - KllState { - k: self.sk.wire_k(), - m: self.sk.wire_m(), - num_levels: self.sk.wire_num_levels(), - levels: self.sk.wire_levels(), - items: self.sk.wire_items(), - coin: Some(CoinState { - state, - bit_cache, - remaining_bits, - }), - // Emit the raw-f64 item representation (field 5); the - // value-offset fixed-point encoding (offset/value_scale/ - // residuals, fields 7-9) is left at its off defaults so the - // wire form is unchanged. - offset: 0.0, - value_scale: 0, - residuals: Vec::new(), - } - } - - fn encode_envelope(&self) -> Vec { - let env = ProtoEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(sketch_envelope::SketchState::Kll(self.build_state())), - }; - let mut buf = Vec::with_capacity(env.encoded_len()); - env.encode(&mut buf).expect("prost encode"); - buf - } - - fn decode_envelope_into_history(bytes: &[u8]) -> Result, PrecomputeError> { - let env = ProtoEnvelope::decode(bytes) - .map_err(|e| PrecomputeError::Other(format!("KLLWrapper decode: {e}")))?; - match env.sketch_state { - Some(sketch_envelope::SketchState::Kll(s)) => Ok(s.items), - _ => Err(PrecomputeError::Other( - "KLLWrapper: envelope did not carry KllState".into(), - )), - } - } -} - -fn build_kll(k: i32, seed: Option) -> KLL { - match seed { - Some(s) => KLL::init_kll_with_seed(k, s), - None => KLL::init_kll(k), - } } impl Sketch for KLLWrapper { fn snapshot(&self) -> Result, PrecomputeError> { - if self.wire_encoding.is_msgpack() { - // Msgpack reads the compactor directly (not the history vec). - if self.sk.count() == 0 { - return Ok(Vec::new()); - } - return self.encode_msgpack(); - } - if self.history.is_empty() { + if self.sk.count() == 0 { return Ok(Vec::new()); } - Ok(self.encode_envelope()) - } - - fn compute_delta_against( - &self, - _prev: &[u8], - _threshold: u64, - ) -> Result { - // KLL uses random compaction and is not additively mergeable - // in a delta sense. Always return the full snapshot. - let full = self.snapshot()?; - Ok(DeltaResult { - payload: full, - is_full: true, - }) - } - - fn apply_delta(&mut self, payload: &[u8]) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - let other_history = Self::decode_envelope_into_history(payload)?; - for &v in &other_history { - if v.is_finite() { - self.history.push(v); - self.sk.update(&v); - } - } - Ok(()) - } - - fn apply_delta_encoded( - &mut self, - payload: &[u8], - encoding: Encoding, - ) -> Result<(), PrecomputeError> { - if payload.is_empty() { - return Ok(()); - } - // KLL is full-only; a msgpack frame is a portable KLL full state — - // merge its compactor. Proto frames keep the history-replay path. - if encoding.is_msgpack() { - return self.merge_msgpack(payload); - } - self.apply_delta(payload) + self.sk + .serialize_to_bytes() + .map_err(|e| PrecomputeError::Other(format!("KLLWrapper snapshot: {e}"))) } - fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError> { - let bytes = other.snapshot()?; + /// Uses sketchlib's level-aware merge so each retained item keeps its + /// `2^level` weight after compaction. + fn merge_snapshot(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError> { if bytes.is_empty() { return Ok(()); } - if self.wire_encoding.is_msgpack() { - return self.merge_msgpack(&bytes); - } - let other_history = Self::decode_envelope_into_history(&bytes)?; - for &v in &other_history { - if v.is_finite() { - self.history.push(v); - self.sk.update(&v); - } - } + self.sk.merge(&Self::from_snapshot(bytes)?.sk); Ok(()) } fn reset(&mut self) { - self.sk = build_kll(self.k, self.seed); - self.history.clear(); + self.sk.clear(); } fn estimate(&self, quantiles: &[f64], _top_k: usize) -> Vec { @@ -309,68 +132,59 @@ impl SketchObserver for KLLObserver { #[cfg(test)] mod tests { - #[test] - fn asapv1_keeps_compacted_state_without_raw_replay_history() { - use super::*; - let mut sketch = KLLWrapper::new(200, Some(42)).with_wire_encoding(Encoding::Msgpack); - sketch.update_batch((0..20_000).map(|value| value as f64)); - assert!(sketch.history.is_empty()); - // The compactor reports retained weight, not the exact raw input count. - assert!(sketch.inner().count().abs_diff(20_000) < 400); - assert!(sketch.snapshot().unwrap().len() < 20_000); - } use super::*; #[test] fn new_wrapper_is_empty() { - let w = KLLWrapper::new(200, Some(0xDEAD)); - assert_eq!(w.snapshot().unwrap().len(), 0); + assert!(KLLWrapper::new(200, Some(0xDEAD)) + .snapshot() + .unwrap() + .is_empty()); } #[test] fn update_then_quantile() { let mut w = KLLWrapper::new(200, Some(0xDEAD)); - for i in 1..=100 { - w.update(i as f64); - } + w.update_batch((1..=100).map(f64::from)); let p50 = w.quantile(0.5); assert!((40.0..=60.0).contains(&p50), "p50={p50}"); } #[test] - fn snapshot_roundtrip_preserves_count() { + fn snapshot_round_trips_through_asapv1() { let mut w = KLLWrapper::new(200, Some(0xDEAD)); - for i in 1..=10 { - w.update(i as f64); - } + w.update_batch((1..=10).map(f64::from)); let bytes = w.snapshot().unwrap(); - let history = KLLWrapper::decode_envelope_into_history(&bytes).unwrap(); - assert_eq!(history.len(), 10); + let decoded = KLLWrapper::from_snapshot(&bytes).unwrap(); + assert_eq!(decoded.inner().count(), 10); + } + + #[test] + fn compacted_state_stays_small() { + let mut w = KLLWrapper::new(200, Some(42)); + w.update_batch((0..20_000).map(f64::from)); + // The compactor reports retained weight, not the exact raw input count. + assert!(w.inner().count().abs_diff(20_000) < 400); + assert!(w.snapshot().unwrap().len() < 20_000); } #[test] fn deterministic_seed_yields_identical_snapshot() { let mut a = KLLWrapper::new(200, Some(42)); let mut b = KLLWrapper::new(200, Some(42)); - for i in 1..=50 { - a.update(i as f64); - b.update(i as f64); - } + a.update_batch((1..=50).map(f64::from)); + b.update_batch((1..=50).map(f64::from)); assert_eq!(a.snapshot().unwrap(), b.snapshot().unwrap()); } - /// Merging a compacted ASAPv1 sketch must preserve the source's logical - /// sample weight, not reduce it to the number of retained items. + /// Merging a compacted sketch must preserve the source's logical sample + /// weight, not reduce it to the number of retained items. #[test] - fn asapv1_merge_preserves_compacted_sample_count() { - let mut source = KLLWrapper::new(200, Some(42)).with_wire_encoding(Encoding::Msgpack); - for value in 0..50_000 { - source.update(value as f64); - } - let mut target = KLLWrapper::new(200, Some(99)).with_wire_encoding(Encoding::Msgpack); - + fn merge_preserves_compacted_sample_count() { + let mut source = KLLWrapper::new(200, Some(42)); + source.update_batch((0..50_000).map(f64::from)); + let mut target = KLLWrapper::new(200, Some(99)); target.merge(&source).expect("merge compacted KLL"); - assert_eq!(target.inner().count(), source.inner().count()); let p50 = target.quantile(0.5); assert!((23_000.0..=27_000.0).contains(&p50), "p50={p50}"); diff --git a/asap-precompute-rs/src/sketches/mod.rs b/asap-precompute-rs/src/sketches/mod.rs index a539d4f..023b070 100644 --- a/asap-precompute-rs/src/sketches/mod.rs +++ b/asap-precompute-rs/src/sketches/mod.rs @@ -1,36 +1,10 @@ -//! Real sketch wrappers over [`asap_sketchlib`] types. +//! Sketch wrappers over [`asap_sketchlib`] types. //! -//! Each wrapper adapts a concrete sketch implementation from -//! [`asap_sketchlib`] to the host-neutral [`crate::precompute::Sketch`] -//! trait family so a [`crate::precompute::Precompute`] instance can own -//! it as a generic sketch. -//! -//! # Wire format -//! -//! Wrappers serialize via `asap_sketchlib::proto::sketchlib::SketchEnvelope` -//! (prost-encoded). The envelope shape comes from the shared proto -//! definitions (`asap_sketchlib/proto/*.proto`). Field-level byte -//! stability depends on: -//! -//! - Identical proto field tags / wire types (guaranteed by the shared -//! proto file). -//! - Identical numeric encoding (integer values map to fixed-tag -//! varints deterministically). -//! - Identical floating-point bit patterns (no platform divergence -//! for finite f64 in IEEE-754 round-trip). -//! -//! # API surface caveats -//! -//! - `asap_sketchlib` exposes per-sketch `compute_delta` / -//! `apply_delta_bytes` for DDSketch, CMS, CountSketch, and HLL. -//! Those four wrappers emit real `ProtoDelta` frames; KLL stays -//! full-only. -//! - `asap_sketchlib` does not expose -//! `DeserializeXxxFromProtoBytes` round-trip helpers for the -//! high-throughput sketch types. The wrappers decode the wire-format -//! `SketchEnvelope` envelope and reconstruct the wire-aligned -//! sketch struct (`DdSketch`, `HllSketch`, …) directly from the -//! inner state proto. +//! Each wrapper adapts one native sketchlib sketch to the host-neutral +//! [`crate::precompute::Sketch`] trait family. `snapshot()` returns the +//! sketch's self-describing ASAPv1 envelope from `serialize_to_bytes`, and +//! `merge_snapshot()` decodes one with `deserialize_from_bytes` and merges +//! it. Every payload is full state; there are no deltas. pub mod cms; pub mod cms_heap; @@ -40,7 +14,7 @@ pub mod hll; pub mod kll; pub use cms::{CMSObserver, CMSWrapper}; -pub use cms_heap::CMSHeapWrapper; +pub use cms_heap::{CMSHeapObserver, CMSHeapWrapper}; pub use countsketch::{CountSketchObserver, CountSketchWrapper}; pub use ddsketch::{DDSketchObserver, DDSketchWrapper}; pub use hll::{HLLObserver, HLLWrapper}; diff --git a/asap-precompute-rs/src/snapshot_cache.rs b/asap-precompute-rs/src/snapshot_cache.rs deleted file mode 100644 index 5ae5518..0000000 --- a/asap-precompute-rs/src/snapshot_cache.rs +++ /dev/null @@ -1,546 +0,0 @@ -//! Per-series sketch payload cache for delta encoding (outbound) and -//! inbound delta-apply against cached upstream snapshots. -//! -//! Mirrors today's per-processor `snapshots map[string][]byte` -//! (outbound) + `IngestState::sketch_snapshots` (inbound). - -use std::collections::HashMap; -use std::sync::RwLock; - -use crate::precompute::{DeltaResult, PrecomputeError, Sketch}; - -/// Stores per-series sketch payloads for delta encoding (outbound) -/// and inbound delta-apply against cached upstream snapshots. -/// -/// Two maps avoid cross-direction collisions: the same series key -/// may be both produced (outbound) and consumed (inbound) by the -/// same host when running as a forwarder, and the two byte streams -/// are not interchangeable (a remote sender's snapshot is not what -/// we'd emit locally). -/// -/// **Always-refresh policy**: every call to -/// [`Self::compute_delta`] updates the cached previous snapshot to -/// the current sketch state. Successive sub-threshold deltas are -/// therefore each computed against the immediately preceding -/// window, matching the established behavior of all five legacy -/// OTel sketch processors (DDSketch / KLL / HLL / CountSketch / -/// CountMinSketch). There is no configurable -/// "refresh-only-on-full" mode — that earlier design was a bug -/// because it forced downstream consumers to merge a chain of -/// deltas back to the original baseline rather than apply each -/// delta to the previous window's reconstructed state. -pub struct SnapshotCache { - inner: RwLock, -} - -struct SnapshotCacheInner { - outbound: HashMap>, - inbound: HashMap>, -} - -impl Default for SnapshotCache { - fn default() -> Self { - Self::new() - } -} - -impl SnapshotCache { - /// Constructs an empty cache. - pub fn new() -> Self { - Self { - inner: RwLock::new(SnapshotCacheInner { - outbound: HashMap::new(), - inbound: HashMap::new(), - }), - } - } - - /// Stores the latest full sketch payload, keyed by `series_key`. - /// Returns `true` if this is the first snapshot for that key - /// (caller can use this to force a [`crate::envelope::Encoding::ProtoFull`] - /// on next emit). - /// - /// Stores a defensive copy of `payload` so a caller mutating its - /// slice after caching does not race the cache reader. - pub fn cache_outbound(&self, series_key: &str, payload: &[u8]) -> bool { - let mut g = self.inner.write().expect("snapshot cache poisoned"); - let first_time = !g.outbound.contains_key(series_key); - g.outbound.insert(series_key.to_string(), payload.to_vec()); - first_time - } - - /// Returns the cached outbound payload, or `None`. - pub fn get_outbound(&self, series_key: &str) -> Option> { - self.inner - .read() - .expect("snapshot cache poisoned") - .outbound - .get(series_key) - .cloned() - } - - /// Stores an upstream snapshot for delta apply. - pub fn cache_inbound(&self, series_key: &str, payload: &[u8]) { - let mut g = self.inner.write().expect("snapshot cache poisoned"); - g.inbound.insert(series_key.to_string(), payload.to_vec()); - } - - /// Returns the cached upstream snapshot or `None`. - pub fn get_inbound(&self, series_key: &str) -> Option> { - self.inner - .read() - .expect("snapshot cache poisoned") - .inbound - .get(series_key) - .cloned() - } - - /// Diffs current sketch state against the cached outbound - /// snapshot for `series_key`. - /// - /// Returns a [`DeltaResult`] where `is_full` means the runtime - /// should emit `ProtoFull` (either no prior snapshot existed, - /// or the delta exceeded `threshold`). - /// - /// Always-refresh: every call updates the cached previous - /// snapshot to the current sketch state. When `is_full=true` - /// the wire payload IS the full snapshot, so it is reused for - /// the cache; otherwise a fresh full snapshot is serialized - /// for the cache. Both branches end with the cache holding - /// the latest full state. - pub fn compute_delta( - &self, - series_key: &str, - current: &dyn Sketch, - threshold: u64, - ) -> Result { - let prev = self - .inner - .read() - .expect("snapshot cache poisoned") - .outbound - .get(series_key) - .cloned(); - - let result = match prev { - None => { - // First time — emit full. - let full = current.snapshot()?; - DeltaResult { - payload: full, - is_full: true, - } - } - Some(prev_bytes) => current.compute_delta_against(&prev_bytes, threshold)?, - }; - - // Refresh the cached outbound base for the NEXT window's delta. - // - // Per-window deltas: - // families that opt in via `Sketch::delta_against_empty_base` - // (DDSketch / CMS / CountSketch / HLL) reset the cached base to the - // EMPTY-sketch snapshot after each window-close emit, so the next - // window diffs against empty and transmits its OWN per-window state - // as a delta (no cross-window subtraction). This is what makes the - // downstream per-window base rotation correct. - // - // Legacy always-refresh (KLL, and any family that returns the - // `None` default): - // update the cached base to the just-emitted full snapshot, so the - // next delta is computed against the immediately preceding window. - // When is_full=true the wire payload IS the snapshot; reuse it. - // Otherwise serialize a fresh snapshot. - match current.delta_against_empty_base()? { - Some(empty_base) => { - self.cache_outbound(series_key, &empty_base); - } - None => { - if result.is_full { - self.cache_outbound(series_key, &result.payload); - } else { - let full = current.snapshot()?; - self.cache_outbound(series_key, &full); - } - } - } - Ok(result) - } - - /// Clears all cached state. Used in tests and on shutdown. - pub fn reset(&self) { - let mut g = self.inner.write().expect("snapshot cache poisoned"); - g.outbound.clear(); - g.inbound.clear(); - } - - /// Returns the number of cached outbound snapshots. - pub fn len_outbound(&self) -> usize { - self.inner - .read() - .expect("snapshot cache poisoned") - .outbound - .len() - } - - /// Returns the number of cached inbound snapshots. - pub fn len_inbound(&self) -> usize { - self.inner - .read() - .expect("snapshot cache poisoned") - .inbound - .len() - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn outbound_first_time_true_then_false() { - let c = SnapshotCache::new(); - assert!(c.cache_outbound("series-1", &[1, 2, 3])); - assert!(!c.cache_outbound("series-1", &[4, 5, 6])); - assert_eq!(c.len_outbound(), 1); - assert_eq!(c.get_outbound("series-1"), Some(vec![4, 5, 6])); - } - - #[test] - fn inbound_independent_from_outbound() { - let c = SnapshotCache::new(); - c.cache_outbound("k", b"out"); - c.cache_inbound("k", b"in"); - assert_eq!(c.get_outbound("k"), Some(b"out".to_vec())); - assert_eq!(c.get_inbound("k"), Some(b"in".to_vec())); - assert_eq!(c.len_outbound(), 1); - assert_eq!(c.len_inbound(), 1); - } - - #[test] - fn reset_clears_all() { - let c = SnapshotCache::new(); - c.cache_outbound("k", b"v"); - c.cache_inbound("k", b"v"); - c.reset(); - assert_eq!(c.len_outbound(), 0); - assert_eq!(c.len_inbound(), 0); - } - - #[test] - fn cache_stores_defensive_copy() { - let c = SnapshotCache::new(); - let mut payload = vec![1, 2, 3]; - c.cache_outbound("k", &payload); - payload[0] = 99; - // Cache must not have mutated. - assert_eq!(c.get_outbound("k"), Some(vec![1, 2, 3])); - } - - // ---- per-window delta tests (DDSketch) ---- - - use crate::precompute::QuantileSketch; - use crate::sketches::ddsketch::DDSketchWrapper; - - /// Build a fresh per-window DDSketch wrapper from a value range. The - /// runtime resets per-series state every window, so each window is an - /// independent sketch — mirrored here by a fresh wrapper per window. - fn window_sketch(alpha: f64, values: impl IntoIterator) -> DDSketchWrapper { - let mut w = DDSketchWrapper::new(alpha); - for v in values { - w.update(v); - } - w - } - - /// With delta mode on, applying the emitted DDSketch delta - /// to an EMPTY base reconstructs the window's full state, and - /// quantiles match within the α relative-accuracy bound. - #[test] - fn ddsketch_delta_against_empty_round_trips() { - let alpha = 0.01; - let c = SnapshotCache::new(); - - // Window 1 — first emit is FULL (no prior base). - let w1 = window_sketch(alpha, (1..=200).map(|i| i as f64)); - let r1 = c.compute_delta("series", &w1, 1).unwrap(); - assert!(r1.is_full, "first window must emit a full frame"); - - // Window 2 — a fresh per-window sketch; emit must be a DELTA now - // (the cache reset the base to empty at window-1 close). - let w2 = window_sketch(alpha, (1..=200).map(|i| i as f64)); - let r2 = c.compute_delta("series", &w2, 1).unwrap(); - assert!(!r2.is_full, "window 2 must emit a delta, not a full frame"); - assert!(!r2.payload.is_empty(), "delta payload must be non-empty"); - - // Apply the delta to an EMPTY base → reconstructs window 2. - let mut recon = DDSketchWrapper::new(alpha); - recon.apply_delta(&r2.payload).unwrap(); - assert_eq!(recon.inner().total_count(), w2.inner().total_count()); - - // Quantiles match within α. - for q in [0.5, 0.9, 0.99] { - let got = recon.quantile(q); - let want = w2.quantile(q); - assert!( - (got / want - 1.0).abs() <= alpha, - "q={q}: recon={got} want={want}" - ); - } - } - - /// Two consecutive windows each emit their OWN state — - /// window 2's delta is NOT diffed against window 1 (no cross-window - /// subtraction). Even when window 1's events overlap window 2's, the - /// emitted delta reconstructs window 2 exactly from empty. - #[test] - fn ddsketch_consecutive_windows_no_cross_window_subtraction() { - let alpha = 0.01; - let c = SnapshotCache::new(); - - // Window 1: each value inserted TWICE (high counts). - let mut w1 = DDSketchWrapper::new(alpha); - for i in 1..=100 { - w1.update(i as f64); - w1.update(i as f64); - } - let r1 = c.compute_delta("s", &w1, 1).unwrap(); - assert!(r1.is_full); - - // Window 2: SAME value range but each inserted ONCE (lower counts). - // If the cache diffed window 2 against window 1 (cross-window), - // every bucket delta would saturate to 0 and the emitted delta - // would be empty — under-transmitting window 2 entirely. - let mut w2 = DDSketchWrapper::new(alpha); - for i in 1..=100 { - w2.update(i as f64); - } - let r2 = c.compute_delta("s", &w2, 1).unwrap(); - assert!(!r2.is_full); - - // Reconstruct window 2 from EMPTY — must equal window 2's own - // full state, proving no cross-window subtraction occurred. - let mut recon = DDSketchWrapper::new(alpha); - recon.apply_delta(&r2.payload).unwrap(); - assert_eq!( - recon.inner().total_count(), - w2.inner().total_count(), - "window 2 delta must carry window 2's full count" - ); - - // The reconstructed-from-empty state must match window 2's own - // full per-bucket counts (compared on absolute bucket index — - // the reconstructed store's backing-array padding differs from - // the chunk-grown original, but the logical distribution is - // identical: delta-against-empty == window 2's own state). - let w2_sk = w2.inner(); - let r_sk = recon.inner(); - for (i, &c) in w2_sk.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = w2_sk.store_offset + i as i32; - let r_idx = (k - r_sk.store_offset) as usize; - assert_eq!(r_sk.store_counts[r_idx], c, "bucket k={k}"); - } - - // And it must NOT equal window 1's state (which had double counts). - assert_ne!(recon.inner().total_count(), w1.inner().total_count()); - } - - // ---- per-window delta tests (CMS) ---- - - use crate::precompute::FrequencySketch; - use crate::sketches::cms::CMSWrapper; - - /// With delta mode on, applying the emitted CMS delta to an EMPTY base - /// reconstructs the window's per-cell matrix, and the estimated - /// frequency matches. - #[test] - fn cms_delta_against_empty_round_trips() { - let c = SnapshotCache::new(); - let (rows, cols) = (5, 1024); - - // Window 1 — first emit is FULL (no prior base). - let mut w1 = CMSWrapper::new(rows, cols); - for _ in 0..200 { - w1.update("hot", 1.0); - } - let r1 = c.compute_delta("series", &w1, 1).unwrap(); - assert!(r1.is_full, "first window must emit a full frame"); - - // Window 2 — fresh per-window sketch; emit must be a DELTA now. - let mut w2 = CMSWrapper::new(rows, cols); - for _ in 0..200 { - w2.update("hot", 1.0); - } - let r2 = c.compute_delta("series", &w2, 1).unwrap(); - assert!(!r2.is_full, "window 2 must emit a delta, not a full frame"); - assert!(!r2.payload.is_empty(), "delta payload must be non-empty"); - - // Apply the delta to an EMPTY base → reconstructs window 2. - let mut recon = CMSWrapper::new(rows, cols); - recon.apply_delta(&r2.payload).unwrap(); - let want = w2.estimate_count(b"hot"); - let got = recon.estimate_count(b"hot"); - assert!((got - want).abs() <= want * 0.1, "recon={got} want={want}"); - } - - /// Two consecutive windows each emit their OWN state — window 2's - /// delta is NOT diffed against window 1. Window 1 counts "k" 300×, - /// window 2 counts it 50×; reconstructing window 2 from EMPTY yields - /// ~50, proving no cross-window subtraction. - #[test] - fn cms_consecutive_windows_no_cross_window_subtraction() { - let c = SnapshotCache::new(); - let (rows, cols) = (5, 1024); - - let mut w1 = CMSWrapper::new(rows, cols); - for _ in 0..300 { - w1.update("k", 1.0); - } - assert!(c.compute_delta("s", &w1, 1).unwrap().is_full); - - let mut w2 = CMSWrapper::new(rows, cols); - for _ in 0..50 { - w2.update("k", 1.0); - } - let r2 = c.compute_delta("s", &w2, 1).unwrap(); - assert!(!r2.is_full); - assert!(!r2.payload.is_empty()); - - let mut recon = CMSWrapper::new(rows, cols); - recon.apply_delta(&r2.payload).unwrap(); - let got = recon.estimate_count(b"k"); - let want = w2.estimate_count(b"k"); - assert!((got - want).abs() <= want * 0.1, "got={got} want={want}"); - assert!(got < 150.0, "window 2 leaked window 1's mass: got={got}"); - } - - // ---- per-window delta tests (CountSketch) ---- - - use crate::sketches::countsketch::CountSketchWrapper; - - /// With delta mode on, applying the emitted CountSketch delta to an - /// EMPTY base reconstructs the window's per-cell matrix. - #[test] - fn countsketch_delta_against_empty_round_trips() { - let c = SnapshotCache::new(); - let (rows, cols) = (5, 1024); - - let mut w1 = CountSketchWrapper::new(rows, cols); - for _ in 0..200 { - w1.update("hot", 1.0); - } - assert!(c.compute_delta("series", &w1, 1).unwrap().is_full); - - let mut w2 = CountSketchWrapper::new(rows, cols); - for _ in 0..200 { - w2.update("hot", 1.0); - } - let r2 = c.compute_delta("series", &w2, 1).unwrap(); - assert!(!r2.is_full, "window 2 must emit a delta"); - assert!(!r2.payload.is_empty()); - - let mut recon = CountSketchWrapper::new(rows, cols); - recon.apply_delta(&r2.payload).unwrap(); - // CountSketch round-trips its cells exactly under - // delta-against-empty: matrices must be identical. - assert_eq!(recon.inner().matrix, w2.inner().matrix); - } - - /// CountSketch consecutive windows each emit their own state. - #[test] - fn countsketch_consecutive_windows_no_cross_window_subtraction() { - let c = SnapshotCache::new(); - let (rows, cols) = (5, 1024); - - let mut w1 = CountSketchWrapper::new(rows, cols); - for _ in 0..300 { - w1.update("k", 1.0); - } - assert!(c.compute_delta("s", &w1, 1).unwrap().is_full); - - let mut w2 = CountSketchWrapper::new(rows, cols); - for _ in 0..50 { - w2.update("k", 1.0); - } - let r2 = c.compute_delta("s", &w2, 1).unwrap(); - assert!(!r2.is_full); - - let mut recon = CountSketchWrapper::new(rows, cols); - recon.apply_delta(&r2.payload).unwrap(); - // Reconstructed-from-empty matrix equals window 2's own — not - // window 1's higher-count matrix. - assert_eq!(recon.inner().matrix, w2.inner().matrix); - assert_ne!(recon.inner().matrix, w1.inner().matrix); - } - - // ---- per-window delta tests (HLL) ---- - - use crate::precompute::CardinalitySketch; - use crate::sketches::hll::HLLWrapper; - use asap_sketchlib::HllVariant as RsHllVariant; - - /// With delta mode on, applying the emitted HLL register delta to an - /// EMPTY base reconstructs the window's registers exactly, and the - /// cardinality estimate matches. - #[test] - fn hll_delta_against_empty_round_trips() { - let c = SnapshotCache::new(); - - let mut w1 = HLLWrapper::new(RsHllVariant::Regular, 12); - for i in 0..2000u64 { - w1.update(&i.to_le_bytes()); - } - assert!(c.compute_delta("series", &w1, 1).unwrap().is_full); - - let mut w2 = HLLWrapper::new(RsHllVariant::Regular, 12); - for i in 0..2000u64 { - w2.update(&i.to_le_bytes()); - } - let r2 = c.compute_delta("series", &w2, 1).unwrap(); - assert!(!r2.is_full, "window 2 must emit a delta"); - assert!(!r2.payload.is_empty()); - - let mut recon = HLLWrapper::new(RsHllVariant::Regular, 12); - recon.apply_delta(&r2.payload).unwrap(); - // Register delta over an empty base carries every non-zero - // register, so the reconstructed registers are identical. - assert_eq!(recon.inner().registers, w2.inner().registers); - let want = w2.estimate_cardinality(); - let got = recon.estimate_cardinality(); - assert!((got - want).abs() <= want * 0.001, "got={got} want={want}"); - } - - /// HLL consecutive windows each emit their OWN register state. Window - /// 1 sees 5000 keys, window 2 a DISJOINT 200 keys. Reconstructing - /// window 2 from EMPTY must yield ~200, NOT the ~5000-element union — - /// proving the empty-base reset defeats register-MAX leakage. - #[test] - fn hll_consecutive_windows_no_cross_window_subtraction() { - let c = SnapshotCache::new(); - - let mut w1 = HLLWrapper::new(RsHllVariant::Regular, 12); - for i in 0..5000u64 { - w1.update(&i.to_le_bytes()); - } - assert!(c.compute_delta("s", &w1, 1).unwrap().is_full); - - let mut w2 = HLLWrapper::new(RsHllVariant::Regular, 12); - for i in 1_000_000u64..1_000_200u64 { - w2.update(&i.to_le_bytes()); - } - let r2 = c.compute_delta("s", &w2, 1).unwrap(); - assert!(!r2.is_full); - - let mut recon = HLLWrapper::new(RsHllVariant::Regular, 12); - recon.apply_delta(&r2.payload).unwrap(); - let got = recon.estimate_cardinality(); - let want = w2.estimate_cardinality(); - assert!((got - want).abs() <= want * 0.05, "got={got} want={want}"); - assert!( - got < 1000.0, - "window 2 cardinality leaked window 1's registers: got={got}" - ); - } -} diff --git a/asap-precompute-rs/src/window.rs b/asap-precompute-rs/src/window.rs index 0657f60..4fc05f8 100644 --- a/asap-precompute-rs/src/window.rs +++ b/asap-precompute-rs/src/window.rs @@ -4,10 +4,9 @@ use std::collections::HashMap; use std::time::Duration; use crate::config::{OnOverflow, PrecomputeConfig}; -use crate::envelope::{Encoding, SketchEnvelope}; +use crate::envelope::SketchEnvelope; use crate::observation::{KeyValue, Observation}; use crate::precompute::{BoxedObserver, PrecomputeError, Sketch, SketchFactory, StatsSnapshot}; -use crate::snapshot_cache::SnapshotCache; /// Per-series state held inside a window. /// @@ -205,18 +204,14 @@ impl WindowState { Ok(()) } - /// Applies an inbound envelope to the appropriate series via - /// the sketch's `apply_delta` or `merge` depending on encoding. - /// - /// Strategy A enforcement: inbound envelopes are merged - /// into the local sketch as sketches, never expanded to scalar + /// Merges an inbound envelope's ASAPv1 payload into the matching + /// series. Envelopes are merged as sketches, never expanded to scalar /// samples. pub fn observe_envelope( &mut self, env: &SketchEnvelope, cfg: &PrecomputeConfig, sketch_factory: &SketchFactory, - snapshot_cache: &SnapshotCache, stats: &mut StatsSnapshot, ) -> Result<(), PrecomputeError> { // Use the envelope's window-end as the reference timestamp; @@ -257,33 +252,7 @@ impl WindowState { .get_mut(&key) .expect("series entry must exist after insert"); - match env.encoding { - Encoding::ProtoDelta | Encoding::MsgpackDelta => { - // Delta apply path: feed the delta bytes directly - // into the sketch; `apply_delta_encoded` dispatches the - // proto vs msgpack delta decoder off the inbound tag. - entry - .sketch - .apply_delta_encoded(&env.payload, env.encoding)?; - // Reconstruct the new full snapshot for cached - // inbound use. - if let Ok(snap) = entry.sketch.snapshot() { - snapshot_cache.cache_inbound(&key, &snap); - } - } - Encoding::ProtoFull | Encoding::Msgpack | Encoding::Unspecified => { - // Full-state path: deserialize into a temporary - // sketch and merge. The Layer-3 runtime doesn't hold - // a deserialize hook (those are sketch-specific); we - // go through the SketchFactory + ApplyDelta-as-merge - // convention. `apply_delta_encoded` dispatches the - // proto vs msgpack full decoder off the inbound tag. - let mut other = sketch_factory(); - other.apply_delta_encoded(&env.payload, env.encoding)?; - entry.sketch.merge(other.as_ref())?; - snapshot_cache.cache_inbound(&key, &env.payload); - } - } + entry.sketch.merge_snapshot(&env.payload)?; // Carry the upstream envelope's observation count into our // running entry so the next emission reflects the merged // total. Envelopes with count==0 contribute zero. diff --git a/asap-precompute-rs/tests/kll_windows.rs b/asap-precompute-rs/tests/kll_windows.rs index 844049e..d38a55f 100644 --- a/asap-precompute-rs/tests/kll_windows.rs +++ b/asap-precompute-rs/tests/kll_windows.rs @@ -2,15 +2,14 @@ use asap_precompute_rs::kll_windows::{KllRollingPrecompute, KllWindowStore}; use asap_precompute_rs::precompute::PrecomputeError; use asap_precompute_rs::{ - AggregationMode, Encoding, KeyValue, Observation, ObservationValue, Precompute, - PrecomputeConfig, PrecomputeConfigSet, SketchType, WindowSpec, + AggregationMode, KeyValue, Observation, ObservationValue, Precompute, PrecomputeConfig, + PrecomputeConfigSet, SketchType, WindowSpec, }; use std::time::Duration; fn config() -> PrecomputeConfig { PrecomputeConfig { sketch_type: SketchType::KLLSketch, - encoding: Encoding::Msgpack, mode: AggregationMode::Sliding, transmit_sketch: false, metric_name: "request.duration.rolling_estimate".into(), @@ -267,7 +266,7 @@ fn disjoint_mode_transmits_full_panes_for_downstream_rolling_queries() { creator.observe(&observation(1, 42., "checkout")).unwrap(); let pane = creator.tick(10); assert_eq!(pane.len(), 1); - assert_eq!(pane[0].encoding, Encoding::Msgpack); + assert!(pane[0].payload.starts_with(b"ASAPv1")); let mut retained = store(); retained.merge_pane(&pane[0]).unwrap(); let answer = retained.quantile_over_time(0, 10, &[0.25, 0.75]).unwrap(); diff --git a/asap-precompute-rs/tests/otap_pipeline_e2e.rs b/asap-precompute-rs/tests/otap_pipeline_e2e.rs index 7201239..3aa616d 100644 --- a/asap-precompute-rs/tests/otap_pipeline_e2e.rs +++ b/asap-precompute-rs/tests/otap_pipeline_e2e.rs @@ -5,7 +5,7 @@ use std::sync::{Arc, Mutex, OnceLock}; use std::time::{Duration, Instant}; -use asap_precompute_rs::envelope::{Encoding, SketchEnvelope, SketchType}; +use asap_precompute_rs::envelope::{SketchEnvelope, SketchType}; use asap_precompute_rs::observation::KeyValue; use asap_precompute_rs::otap::codec::{decode_pdata_to_observations, encode_envelopes_to_pdata}; use async_trait::async_trait; @@ -57,9 +57,7 @@ fn scalar_input(start: i32, end: i32) -> OtapPdata { labels: vec![KeyValue::new("route", "/checkout")], window_start_ms: 1_000, window_end_ms: 2_000, - encoding: Encoding::Unspecified, payload: vec![], - hash_spec: None, metric_name: "request.duration".to_owned(), count: 0, aggregation_temporality: 0, @@ -200,7 +198,6 @@ nodes: type: "urn:asap:processor:asap_sketches" config: sketch_type: "kll" - encoding: "Msgpack" window_size: "20ms" output_metric_name: "request.duration.sketch" agg_id: 7 @@ -211,7 +208,6 @@ nodes: type: "urn:asap:processor:asap_sketches" config: sketch_type: "kll" - encoding: "Msgpack" window_size: "20ms" output_metric_name: "request.duration.sketch" agg_id: 7 @@ -222,7 +218,6 @@ nodes: type: "urn:asap:processor:asap_sketches" config: sketch_type: "kll" - encoding: "Msgpack" window_size: "20ms" output_metric_name: "request.duration.merged_sketch" agg_id: 7 @@ -233,7 +228,6 @@ nodes: type: "urn:asap:processor:asap_sketches" config: sketch_type: "kll" - encoding: "Msgpack" window_size: "20ms" output_metric_name: "request.duration.estimate" agg_id: 7 diff --git a/asap-precompute-rs/tests/runtime.rs b/asap-precompute-rs/tests/runtime.rs index 3f34de8..6607e59 100644 --- a/asap-precompute-rs/tests/runtime.rs +++ b/asap-precompute-rs/tests/runtime.rs @@ -5,114 +5,42 @@ //! //! Coverage: //! - `series_key` byte-equivalence (golden literal strings). -//! - `SnapshotCache::compute_delta` always-refresh policy. //! - End-to-end observe → tick cycle. //! - `Drain` rotates partial-window state. -use std::sync::{Arc, Mutex}; use std::time::Duration; use asap_precompute_rs::config::WindowSpec; use asap_precompute_rs::matchers::{attributes_key, series_key}; use asap_precompute_rs::precompute::{ - DeltaResult, Precompute, PrecomputeError, PrecomputeImpl, Sketch, SketchObserver, + Precompute, PrecomputeError, PrecomputeImpl, Sketch, SketchObserver, }; -use asap_precompute_rs::snapshot_cache::SnapshotCache; use asap_precompute_rs::{ - AggregationMode, Encoding, KeyValue, Observation, ObservationValue, PrecomputeConfig, + AggregationMode, KeyValue, Observation, ObservationValue, PrecomputeConfig, PrecomputeConfigSet, SketchType, }; // ---------------------------------------------------------------- -// Fake Sketch + SketchObserver test doubles. The byte-level test -// fixtures (delta:"" prefix) are pinned so they stay identical. +// Fake Sketch + SketchObserver test doubles: the "snapshot" is the raw +// state bytes and merging concatenates them. -#[derive(Clone, Default)] -struct FakeSketchInner { - state: Vec, - delta_force: bool, -} - -#[derive(Clone, Default)] +#[derive(Default)] struct FakeSketch { - inner: Arc>, -} - -impl FakeSketch { - fn new() -> Self { - Self::default() - } - - fn with_state(state: &[u8]) -> Self { - Self { - inner: Arc::new(Mutex::new(FakeSketchInner { - state: state.to_vec(), - delta_force: false, - })), - } - } - - fn set_state(&self, state: &[u8]) { - self.inner.lock().unwrap().state = state.to_vec(); - } - - fn force_full_next(&self) { - self.inner.lock().unwrap().delta_force = true; - } + state: Vec, } impl Sketch for FakeSketch { fn snapshot(&self) -> Result, PrecomputeError> { - Ok(self.inner.lock().unwrap().state.clone()) + Ok(self.state.clone()) } - fn compute_delta_against( - &self, - prev: &[u8], - threshold: u64, - ) -> Result { - let inner = self.inner.lock().unwrap(); - if inner.delta_force { - return Ok(DeltaResult { - payload: inner.state.clone(), - is_full: true, - }); - } - // delta = "delta:" + state (ignoring prev for the fake). - let _ = prev; - let mut delta = Vec::with_capacity(inner.state.len() + 6); - delta.extend_from_slice(b"delta:"); - delta.extend_from_slice(&inner.state); - if delta.len() as u64 > threshold { - return Ok(DeltaResult { - payload: inner.state.clone(), - is_full: true, - }); - } - Ok(DeltaResult { - payload: delta, - is_full: false, - }) - } - - fn apply_delta(&mut self, delta: &[u8]) -> Result<(), PrecomputeError> { - self.inner.lock().unwrap().state.extend_from_slice(delta); - Ok(()) - } - - fn merge(&mut self, other: &dyn Sketch) -> Result<(), PrecomputeError> { - // Snapshot the other sketch and concatenate. - let other_state = other.snapshot()?; - self.inner - .lock() - .unwrap() - .state - .extend_from_slice(&other_state); + fn merge_snapshot(&mut self, bytes: &[u8]) -> Result<(), PrecomputeError> { + self.state.extend_from_slice(bytes); Ok(()) } fn reset(&mut self) { - self.inner.lock().unwrap().state.clear(); + self.state.clear(); } fn as_any_mut(&mut self) -> &mut dyn std::any::Any { @@ -131,12 +59,12 @@ impl SketchObserver for FakeObserver { asap_precompute_rs::ObservationValueKind::Bytes => &obs.value.bytes, asap_precompute_rs::ObservationValueKind::Envelope => b"", }; - sketch.apply_delta(tag) + sketch.merge_snapshot(tag) } } fn make_factory() -> Box Box + Send + Sync> { - Box::new(|| Box::new(FakeSketch::new()) as Box) + Box::new(|| Box::new(FakeSketch::default()) as Box) } // ---------------------------------------------------------------- @@ -235,65 +163,7 @@ fn attributes_key_sorts_by_key() { } // ---------------------------------------------------------------- -// 2. SnapshotCache always-refresh policy. - -#[test] -fn snapshot_cache_first_call_returns_full() { - let c = SnapshotCache::new(); - let s = FakeSketch::with_state(b"v1"); - let r = c.compute_delta("k1", &s, 1024).expect("compute"); - assert!(r.is_full); - assert_eq!(r.payload, b"v1"); - // Cache populated. - assert_eq!(c.get_outbound("k1").as_deref(), Some(b"v1".as_ref())); -} - -#[test] -fn snapshot_cache_subsequent_returns_delta_and_refreshes() { - let c = SnapshotCache::new(); - let s = FakeSketch::with_state(b"v1"); - c.compute_delta("k1", &s, 1024).expect("first"); - s.set_state(b"v1v2"); - let r = c.compute_delta("k1", &s, 1024).expect("second"); - assert!(!r.is_full); - assert_eq!(r.payload, b"delta:v1v2"); - // Always-refresh: cache advances to the latest full state. - assert_eq!(c.get_outbound("k1").as_deref(), Some(b"v1v2".as_ref())); -} - -#[test] -fn snapshot_cache_always_refresh_chain() { - // Each delta is computed against the immediately preceding - // window's state — not the original baseline. - let c = SnapshotCache::new(); - let s = FakeSketch::with_state(b"a"); - c.compute_delta("k", &s, 1024).expect("w0"); - assert_eq!(c.get_outbound("k").as_deref(), Some(b"a".as_ref())); - - s.set_state(b"ab"); - c.compute_delta("k", &s, 1024).expect("w1"); - assert_eq!(c.get_outbound("k").as_deref(), Some(b"ab".as_ref())); - - s.set_state(b"abc"); - c.compute_delta("k", &s, 1024).expect("w2"); - assert_eq!(c.get_outbound("k").as_deref(), Some(b"abc".as_ref())); -} - -#[test] -fn snapshot_cache_above_threshold_refreshes() { - let c = SnapshotCache::new(); - let s = FakeSketch::with_state(b"v1"); - c.compute_delta("k1", &s, 1024).expect("first"); - s.set_state(b"BIG"); - s.force_full_next(); - let r = c.compute_delta("k1", &s, 1024).expect("forced"); - assert!(r.is_full); - assert_eq!(r.payload, b"BIG"); - assert_eq!(c.get_outbound("k1").as_deref(), Some(b"BIG".as_ref())); -} - -// ---------------------------------------------------------------- -// 3. Observe → tick cycle produces expected envelope count + shape. +// 2. Observe → tick cycle produces expected envelope count + shape. #[test] fn observe_then_tick_emits_one_envelope_per_series() { @@ -385,49 +255,8 @@ fn stats_account_observations() { assert_eq!(s.active_series, 1); } -#[test] -fn delta_transmission_first_full_then_delta() { - let cfg = PrecomputeConfig { - agg_id: 1, - sketch_type: SketchType::DDSketch, - mode: AggregationMode::Tumbling, - window: WindowSpec { - size: Duration::from_secs(10), - ..Default::default() - }, - delta_transmission: true, - delta_threshold: 1024, - ..Default::default() - }; - let p = PrecomputeImpl::new( - Some(cfg), - Some(make_factory()), - Some(Box::new(FakeObserver)), - ); - - let obs = |ts: u64| Observation { - timestamp_ms: ts, - metric: "m".into(), - labels: vec![KeyValue::new("k", "a")], - value: ObservationValue::float(1.0), - ..Default::default() - }; - - // Window 0 — first emission is full. - p.observe(&obs(1_000)).expect("o1"); - let envs = p.tick(10_000); - assert_eq!(envs.len(), 1); - assert_eq!(envs[0].encoding, Encoding::ProtoFull); - - // Window 1 — small delta. - p.observe(&obs(11_000)).expect("o2"); - let envs = p.tick(20_000); - assert_eq!(envs.len(), 1); - assert_eq!(envs[0].encoding, Encoding::ProtoDelta); -} - // ---------------------------------------------------------------- -// 4. Drain rotates partial-window state. +// 3. Drain rotates partial-window state. #[test] fn drain_flushes_mid_window_observations() { @@ -576,7 +405,6 @@ fn observe_envelope_merges_into_active_window() { labels: vec![KeyValue::new("k", "a")], window_start_ms: 0, window_end_ms: 10_000, - encoding: Encoding::ProtoFull, payload: b"upstream-state".to_vec(), count: 17, ..Default::default() @@ -617,7 +445,6 @@ fn observe_envelope_rejects_agg_id_mismatch() { schema_version: 1, sketch_type: SketchType::DDSketch, agg_id: 99, - encoding: Encoding::ProtoFull, payload: b"x".to_vec(), ..Default::default() }; @@ -800,12 +627,13 @@ fn update_config_swaps_active() { } // ---------------------------------------------------------------- -// 8. Real-sketch wrappers (asap_sketchlib-backed) drive the runtime +// 4. Real-sketch wrappers (asap_sketchlib-backed) drive the runtime // end-to-end. One test per wrapper exercising // observe → tick → envelope output. mod real_sketch { use super::*; + use asap_precompute_rs::precompute::{FrequencySketch, QuantileSketch}; use asap_precompute_rs::sketches::{ CMSObserver, CMSWrapper, CountSketchObserver, CountSketchWrapper, DDSketchObserver, DDSketchWrapper, HLLObserver, HLLWrapper, KLLObserver, KLLWrapper, @@ -821,9 +649,7 @@ mod real_sketch { } fn hll_factory() -> Box Box + Send + Sync> { - Box::new(|| { - Box::new(HLLWrapper::new(asap_sketchlib::HllVariant::Regular, 12)) as Box - }) + Box::new(|| Box::new(HLLWrapper::new()) as Box) } fn count_sketch_factory() -> Box Box + Send + Sync> { @@ -879,63 +705,8 @@ mod real_sketch { assert_eq!(envs.len(), 1); let env = &envs[0]; assert_eq!(env.sketch_type, SketchType::DDSketch); - assert_eq!(env.encoding, Encoding::ProtoFull); - assert!(!env.payload.is_empty()); - } - - #[test] - fn ddsketch_msgpack_delta_first_full_then_delta() { - // End-to-end Stage-3 wiring: a msgpack-configured DDSketch emits a - // full `Msgpack` frame on the first window and a sparse - // `MsgpackDelta` frame on the next, each reconstructable via - // sketchlib's msgpack codec. - use asap_sketchlib::MessagePackCodec; - - let cfg = PrecomputeConfig { - agg_id: 1, - sketch_type: SketchType::DDSketch, - mode: AggregationMode::Tumbling, - window: WindowSpec { - size: Duration::from_secs(10), - ..Default::default() - }, - delta_transmission: true, - delta_threshold: 1, - encoding: Encoding::MsgpackDelta, - ..Default::default() - }; - let factory: Box Box + Send + Sync> = Box::new(|| { - Box::new(DDSketchWrapper::new(0.01).with_wire_encoding(Encoding::MsgpackDelta)) - as Box - }); - let p = PrecomputeImpl::new(Some(cfg), Some(factory), Some(Box::new(DDSketchObserver))); - - // Window 0 — first emission is a full msgpack frame. - for i in 1..=50 { - p.observe(&float_obs("latency_ms", 1_000, "GET", i as f64)) - .expect("observe"); - } - let envs = p.tick(10_000); - assert_eq!(envs.len(), 1); - assert_eq!(envs[0].encoding, Encoding::Msgpack); - let full = asap_sketchlib::DdSketch::from_msgpack(&envs[0].payload).expect("decode full"); - assert_eq!(full.total_count(), 50); - - // Window 1 — a sparse msgpack delta frame; applying it to an empty - // base reconstructs this window's own count (no cross-window leak). - for i in 1..=50 { - p.observe(&float_obs("latency_ms", 11_000, "GET", i as f64)) - .expect("observe"); - } - let envs = p.tick(20_000); - assert_eq!(envs.len(), 1); - assert_eq!(envs[0].encoding, Encoding::MsgpackDelta); - assert!(!envs[0].payload.is_empty()); - let mut recon = asap_sketchlib::DdSketch::new(0.01); - recon - .apply_delta_msgpack_bytes(&envs[0].payload) - .expect("apply msgpack delta"); - assert_eq!(recon.total_count(), 50); + let decoded = DDSketchWrapper::from_snapshot(&env.payload).expect("ASAPv1 DDSketch"); + assert_eq!(decoded.inner().get_count(), 20); } #[test] @@ -975,7 +746,6 @@ mod real_sketch { .unwrap_or_else(|| panic!("missing quantile {q}")) }; for env in &envs { - assert_eq!(env.encoding, Encoding::Unspecified); assert!( env.payload.is_empty(), "estimate rows carry no sketch bytes" @@ -1007,7 +777,6 @@ mod real_sketch { } let envs = p.tick(10_000); assert_eq!(envs.len(), 1, "one cardinality gauge"); - assert_eq!(envs[0].encoding, Encoding::Unspecified); assert!(envs[0].payload.is_empty()); assert!( (envs[0].value - 1000.0).abs() / 1000.0 < 0.1, @@ -1016,13 +785,10 @@ mod real_sketch { ); } + /// A sketch emitted by one node is re-ingested by a second + /// (envelope-input) node via `observe_envelope` and re-emitted. #[test] - fn cms_msgpack_full_roundtrips_through_ingest() { - // A msgpack full frame emitted by one node is re-ingested by a - // second (envelope-input) node via observe_envelope → the - // encoding-aware full-merge path, and the merged estimate matches. - use asap_sketchlib::MessagePackCodec; - + fn cms_asapv1_roundtrips_through_ingest() { let cfg = PrecomputeConfig { agg_id: 1, sketch_type: SketchType::CountMinSketch, @@ -1031,16 +797,11 @@ mod real_sketch { size: Duration::from_secs(10), ..Default::default() }, - encoding: Encoding::Msgpack, ..Default::default() }; - let factory: Box Box + Send + Sync> = Box::new(|| { - Box::new(CMSWrapper::new(4, 32).with_wire_encoding(Encoding::Msgpack)) - as Box - }); let producer = PrecomputeImpl::new( Some(cfg.clone()), - Some(factory), + Some(cms_factory()), Some(Box::new(CMSObserver)), ); for _ in 0..100 { @@ -1050,25 +811,19 @@ mod real_sketch { } let envs = producer.tick(10_000); assert_eq!(envs.len(), 1); - assert_eq!(envs[0].encoding, Encoding::Msgpack); - // Second node ingests the emitted msgpack envelope and re-emits. - let factory2: Box Box + Send + Sync> = Box::new(|| { - Box::new(CMSWrapper::new(4, 32).with_wire_encoding(Encoding::Msgpack)) - as Box - }); - let merger = PrecomputeImpl::new(Some(cfg), Some(factory2), Some(Box::new(CMSObserver))); + let merger = + PrecomputeImpl::new(Some(cfg), Some(cms_factory()), Some(Box::new(CMSObserver))); + merger.observe_envelope(&envs[0]).expect("observe_envelope"); merger.observe_envelope(&envs[0]).expect("observe_envelope"); let out = merger.tick(20_000); assert_eq!(out.len(), 1); - assert_eq!(out[0].encoding, Encoding::Msgpack); - let merged = - asap_sketchlib::CountMinSketch::from_msgpack(&out[0].payload).expect("decode merged"); - assert!(merged.estimate("hot") >= 90.0); + let merged = CMSWrapper::from_snapshot(&out[0].payload).expect("decode merged"); + assert_eq!(merged.estimate_count(b"hot"), 200.0); } #[test] - fn kll_msgpack_full_roundtrips_through_ingest() { + fn kll_asapv1_roundtrips_through_ingest() { let cfg = PrecomputeConfig { agg_id: 1, sketch_type: SketchType::KLLSketch, @@ -1077,16 +832,11 @@ mod real_sketch { size: Duration::from_secs(10), ..Default::default() }, - encoding: Encoding::Msgpack, ..Default::default() }; - let factory: Box Box + Send + Sync> = Box::new(|| { - Box::new(KLLWrapper::new(200, None).with_wire_encoding(Encoding::Msgpack)) - as Box - }); let producer = PrecomputeImpl::new( Some(cfg.clone()), - Some(factory), + Some(kll_factory()), Some(Box::new(KLLObserver)), ); for i in 1..=1000 { @@ -1096,34 +846,16 @@ mod real_sketch { } let envs = producer.tick(10_000); assert_eq!(envs.len(), 1); - assert_eq!(envs[0].encoding, Encoding::Msgpack); - assert!(!envs[0].payload.is_empty()); - let psk = asap_sketchlib::KLL::::deserialize_from_bytes(&envs[0].payload) - .expect("decode producer ASAPv1 envelope"); - assert!( - (psk.quantile(0.5) - 500.0).abs() < 60.0, - "producer p50={}", - psk.quantile(0.5) - ); + assert!(envs[0].payload.starts_with(b"ASAPv1")); - // Second node ingests the msgpack full frame and re-emits a valid - // msgpack frame (smoke). NOTE: cross-host KLL *merge* accuracy is a - // known, pre-existing limitation shared by the proto path — KLL's - // retained items carry level weights that item-replay drops — so we - // assert only that ingest + re-emit round-trips structurally, not - // that the merged median is preserved. - let factory2: Box Box + Send + Sync> = Box::new(|| { - Box::new(KLLWrapper::new(200, None).with_wire_encoding(Encoding::Msgpack)) - as Box - }); - let merger = PrecomputeImpl::new(Some(cfg), Some(factory2), Some(Box::new(KLLObserver))); + let merger = + PrecomputeImpl::new(Some(cfg), Some(kll_factory()), Some(Box::new(KLLObserver))); merger.observe_envelope(&envs[0]).expect("observe_envelope"); let out = merger.tick(20_000); assert_eq!(out.len(), 1); - assert_eq!(out[0].encoding, Encoding::Msgpack); - let sk = asap_sketchlib::KLL::::deserialize_from_bytes(&out[0].payload) - .expect("decode merged ASAPv1 envelope"); - assert!(sk.count() > 0, "re-emitted merged sketch must be non-empty"); + let merged = KLLWrapper::from_snapshot(&out[0].payload).expect("decode merged"); + let p50 = merged.quantile(0.5); + assert!((p50 - 500.0).abs() < 60.0, "merged p50={p50}"); } #[test] @@ -1146,7 +878,6 @@ mod real_sketch { let envs = p.tick(10_000); assert_eq!(envs.len(), 1); assert_eq!(envs[0].sketch_type, SketchType::KLLSketch); - assert_eq!(envs[0].encoding, Encoding::ProtoFull); assert!(!envs[0].payload.is_empty()); } diff --git a/docs/data_model.md b/docs/data_model.md index 61e74ec..2476914 100644 --- a/docs/data_model.md +++ b/docs/data_model.md @@ -1,11 +1,13 @@ # Sketch Envelope Data Model This document describes the information carried when sketch state -(DDSketch, KLL, HLL, Count Sketch, Count-Min Sketch) crosses a node or +(DDSketch, KLL, HLL, Count Sketch, Count-Min Sketch, Count-Min Sketch with +heap) crosses a node or network boundary between `asap_sketches` processor instances. A `RECORD` carries one of two things for a given series and window: sketch -state (an `envelope` — a full snapshot or a delta) or an estimate (a +state (an `envelope` — the full ASAPv1 snapshot from `asap_sketchlib`) or an +estimate (a quantile or cardinality `value`), never both. ## The OTAP analogy: Schema / Dictionary / Record @@ -40,7 +42,7 @@ series" as the separating question, at each of three distinct timescales: | Timescale | Common across... | Fields | OTAP-equivalent tier | |---|---|---|---| -| **Config-time** (changes only on redeploy/reconfig) | every sketch instance, and every summarized series | `sketch_type`; sketch configuration/parameters (sketch size, hash seed, hash function); `encoding`; `schema_version` | **Schema** — this is the contract for decoding *any* envelope byte, not data about one. | +| **Config-time** (changes only on redeploy/reconfig) | every sketch instance, and every summarized series | `sketch_type`; sketch configuration/parameters (sketch size); `schema_version` | **Schema** — this is the contract for decoding *any* envelope byte, not data about one. | | **Series-lifetime** (fixed for one series, differs series to series) | every instance of *one* summarized series, but not across different series | `metric` name + `labels` (the aggregated attributes identifying the series) | **Dictionary** — a bounded, slowly-growing set of distinct (name, label-set) identities, each reused across every window that series produces. | | **Sketch-instance-lifetime** (unique per sketch instance — the OTAP analogy to a row) | nothing — genuinely one-off | `window_start_ms` / `window_end_ms`; `envelope` bytes; `value` (estimate scalar) | **Record** — the only tier whose cost should scale with row count. | @@ -84,9 +86,6 @@ erDiagram agg_id u32 sketch_type string sketch_size u32 "optional" - hash_seed u64 "optional" - hash_function string "optional" - encoding string schema_version u32 } DICTIONARY{ @@ -221,7 +220,7 @@ fully independent stream that each repeats `attrs`: `(attrs, timestamp)` pair. The waste is re-sending that identical pair once per sibling metric. - **Schema/Dictionary/Record's axis is temporal, at two different rates**: - sketch configuration (`sketch_type`, size/hash params, `encoding`) recurs + sketch configuration (`sketch_type`, size params) recurs across *every series and every window* an `agg_id` ever produces — config-time reuse, deduplicated once into `SCHEMA`; series identity (`metric` name + label combination) recurs across *every window that one @@ -256,19 +255,18 @@ bearing on that design; the two axes are orthogonal. | `agg_id` | u32 | Primary key. Identifies this recipe — the aggregation plan/config. | | `sketch_type` | string | Which algorithm: DDSketch / KLL / HLL / Count Sketch / Count-Min Sketch. | | `sketch_size` | u32, optional | The algorithm's size/accuracy parameter (relative accuracy, buffer size, width × depth — whichever applies). | -| `hash_seed` | u64, optional | Determinism contract for hash-based sketches (HLL, Count Sketch, Count-Min Sketch), so independently-produced sketches for the same series can merge correctly. | -| `hash_function` | string, optional | Which hash function, for algorithms that need one. | -| `encoding` | string | Wire layout of every `RECORD.envelope` under this schema: proto or msgpack, full or delta. | | `schema_version` | u32 | Wire-schema version, for forward/backward compatibility across deploys. | -`sketch_type` decides what `sketch_size` (and whether `hash_seed` / -`hash_function` apply) actually means: +`sketch_type` decides what `sketch_size` actually means. Hashing (seed and +function) is not a SCHEMA field: each ASAPv1 envelope's metadata records how +its sketch was hashed, and sketchlib refuses to decode bytes whose hashing +does not match. -| `sketch_type` | What `sketch_size` holds | Example | `hash_seed` / `hash_function` | +| `sketch_type` | What `sketch_size` holds | Example | Hashing | |---|---|---|---| | DDSketch | Relative accuracy (α) | `0.01` | not used — bucket boundaries are computed directly from the value, no hashing involved | | KLL | Buffer size (`k`) | `200` | not used — KLL stores/compacts items directly; it has no hashing step (a separate, optional compaction seed exists to make its internal randomized compaction reproducible, but that's not this field) | -| HLL | Precision — register-count exponent, 4–18 | `12` | used — every value must hash to the same register on every host, or merging two sketches (max-per-register) overcounts the true union | +| HLL | Precision — fixed at 14 (sketchlib's HyperLogLog) | `14` | used — every value must hash to the same register on every host, or merging two sketches (max-per-register) overcounts the true union | | Count Sketch | Width × depth | `2048 × 4` | used — sketch is hash-based, so both sides need the same seed/function to merge correctly | | Count-Min Sketch | Width × depth | `2048 × 4` | used — same reason as Count Sketch | @@ -319,7 +317,7 @@ keys (`path`, `region`): |---|---|---| | `series_id` | u32 | Which `DICTIONARY` entry (and, transitively, which `SCHEMA`) this record belongs to. | | `window_start_ms` / `window_end_ms` | u64 | The time window this record summarizes. | -| `envelope` | bytes, optional | The serialized sketch or sketch-delta. Opaque without the series' `SCHEMA` to interpret it. | +| `envelope` | bytes, optional | The sketch's full state as the self-describing ASAPv1 envelope from `asap_sketchlib`. | | `value` | f64, optional | An estimate-mode result (a quantile or cardinality estimate), carried instead of `envelope` when the series emits estimates rather than sketch state. | Continuing the `agg_id = 7` example above, across two successive 10s @@ -346,7 +344,7 @@ lookup key, never re-derived from `metric`/`labels`. Everything above describes SCHEMA/DICTIONARY/RECORD as if they ride the same stream. In practice SCHEMA is not negotiated on the data path at all — the aggregation plan behind an `agg_id` (which `sketch_type`, -`sketch_size`, `encoding`) is decided out of band by whatever configures +`sketch_size`) is decided out of band by whatever configures the sketch processor. That's a control-plane concern, not a data-plane one. @@ -384,9 +382,9 @@ canonical sketchlib ASAPv1 bytes. The `b"ASAPv1"` magic number, envelope version, kind ID, and length-prefixed metadata and payload make the binary self-describing; the surrounding `sketch.*` Resource/Scope attributes provide OTAP routing and indexing rather than being required to interpret the sketch. -Native OTAP sketch RECORDs require this full MessagePack envelope. Protobuf, -legacy unframed MessagePack, and bare delta frames are rejected at the carrier -boundary. +Native OTAP sketch RECORDs require this full envelope, and the codec rejects +bytes that do not decode as the declared `sketch.type`. There are no delta +frames. OTAP's `resource_id`, `scope_id`, metric `id`, and attribute `parent_id` columns express the joins instead of repeating SCHEMA and series facts on diff --git a/docs/df-processor-topology.md b/docs/df-processor-topology.md index 33ea38d..05316e9 100644 --- a/docs/df-processor-topology.md +++ b/docs/df-processor-topology.md @@ -8,7 +8,7 @@ crosses each edge. The whole design turns on one fact: > hop is a pointer + refcount bump, not a memcpy. But a **sketch is not a > column**: the moment a value lands in a `DdSketch`/`Hll`/… it becomes opaque > single-owner state that never rides a channel. Crossing a node boundary with -> a sketch therefore *always* costs one serialize (proto or **msgpack**). +> a sketch therefore *always* costs one serialize (the ASAPv1 envelope). Design rule: **keep every sketch inside the one node that owns its window state; only ever let SUMMARIES (estimated datapoints) or SERIALIZED envelopes @@ -39,41 +39,39 @@ The sketch state lives inside `P`, owned, never on a channel. At `Wakeup` ### Topology B — edge → central (ship the sketch, merge & estimate downstream) -`transmit_sketch = true`, `encoding = Msgpack | MsgpackDelta`: the edge ships -sketch bytes; a **second instance of the same processor** on the central tier +`transmit_sketch = true`: the edge ships each window's full sketch as the +self-describing ASAPv1 envelope from `asap_sketchlib`; a **second instance of the same processor** on the central tier decodes, merges across edges, and estimates. ```mermaid flowchart LR subgraph EDGE["Edge host"] R1["receiver"] -->|"raw"| P1["asap_sketches
observe · window
output = sketch bytes"] - P1 -->|"Arrow _asap_envelope Binary
msgpack full / delta
serialize once"| E1["OTAP exporter / gRPC"] + P1 -->|"sketch.envelope Binary
ASAPv1 full state
serialize once"| E1["OTAP exporter / gRPC"] end subgraph CENTRAL["Central host"] R2["OTAP receiver"] -->|"Arrow envelope col"| P2["asap_sketches
input = envelope
merge · window · estimate"] P2 -->|"Arrow typed datapoints"| X2["exporter → backend"] end - E1 ==>|"network — msgpack full / delta"| R2 + E1 ==>|"network — ASAPv1 full state"| R2 style P1 fill:#2b6cb0,color:#fff style P2 fill:#2b6cb0,color:#fff ``` -The only real network copy is the `==>` hop. Delta-encoding -(`MsgpackDelta`, per-window against-empty) shrinks it. +The only real network copy is the `==>` hop. --- -## The four cases +## The cases -All four are the **same processor**, parameterized by two knobs — -`transmit_sketch` (sketch vs. estimate out) and `encoding` (proto vs. msgpack, -full vs. delta) — deployed once (terminal) or twice (edge + central). +All are the **same processor**, parameterized by `transmit_sketch` (sketch +vs. estimate out) — deployed once (terminal) or twice (edge + central). | Case | Topology | Config | Compute in node | What crosses the boundary | |---|---|---|---|---| | **1** — insert → merge → estimate, ship results | A | `transmit_sketch=false`, `quantiles=[…]` | insert **+ merge inbound** + estimate | typed OTAP datapoints (gauges) | -| **2** — insert only, ship msgpack full | B (edge) | `transmit_sketch=true`, `encoding=Msgpack` | insert + window | `_asap_envelope` Binary — **msgpack full** | -| **3** — insert-merge, ship msgpack delta | B (edge **+** central) | `encoding=MsgpackDelta`, `delta_transmission=true` | insert (edge); merge + estimate (central) | `_asap_envelope` Binary — **msgpack delta** (edge); gauges (central) | +| **2** — insert only, ship the sketch | B (edge) | `transmit_sketch=true` | insert + window | `sketch.envelope` Binary — **ASAPv1 full state** | +| **3** — insert, then central merge | B (edge **+** central) | `transmit_sketch=true` (edge), `false` (central) | insert (edge); merge + estimate (central) | `sketch.envelope` Binary (edge); gauges (central) | | **4** — insert + estimate, ship via metric types | A | `transmit_sketch=false` | insert + estimate | OTAP Gauge / Sum datapoints | **Case 1 ⊇ Case 4:** same terminal node; Case 1 also folds *inbound* sketch @@ -83,10 +81,7 @@ envelopes (`observe_envelope`), Case 4 is raw-in only. flowchart TD C{"cfg.transmit_sketch?"} C -->|"false → estimate mode"| A["Topology A
Cases 1 and 4
emit gauges"] - C -->|"true → sketch on wire"| D{"cfg.encoding?"} - D -->|"Msgpack (full)"| B2["Topology B · Case 2
ship full sketch"] - D -->|"MsgpackDelta
(+ delta_transmission)"| B3["Topology B · Case 3
ship sparse delta
+ central merger"] - D -->|"ProtoFull / ProtoDelta"| B4["Topology B (proto variant)"] + C -->|"true → sketch on wire"| B2["Topology B · Cases 2 and 3
ship ASAPv1 sketch
(+ central merger)"] ``` --- @@ -110,8 +105,7 @@ flush(): # serialize_series per closed series for p in sketch.estimate(quantiles): # DDSketch/KLL: 1 gauge per quantile emit Gauge{ value=p.value, labels+=p.labels } # HLL: 1 cardinality gauge else: # SKETCH mode (Topology B) - payload,enc = delta_transmission ? compute_delta(...) : snapshot() - emit Envelope{ _asap_envelope=payload, encoding=enc } # proto | msgpack (full/delta) + emit Envelope{ sketch.envelope=snapshot() } # ASAPv1 full state effect_handler.send_message(batch) # zero-copy hop downstream ``` @@ -126,10 +120,10 @@ Every boundary and its true cost: | receiver → proc | raw metrics (OTAP Arrow) | **zero-copy** (`Arc` buffers, channel move) | | decode inside proc | Arrow columns → `f64`/`bytes` → `sketch.update()` | **read-in-place**, no intermediate object | | the windowed sketches | owned `asap_sketchlib` structs | **resident, never serialized while live** | -| inbound sketch (merge) | Binary col → prost/msgpack decode → struct → `merge()` | **one deserialize** per envelope | +| inbound sketch (merge) | Binary col → ASAPv1 decode → struct → `merge()` | **one deserialize** per envelope | | proc → next, **estimate mode** | small Arrow gauge batch (series × quantiles) | **one small alloc**, then zero-copy | -| proc → next, **sketch mode** | `to_msgpack()` / delta → bytes in Binary col | **one serialize** (sketch isn't columnar) | -| edge → central (gRPC) | the envelope batch | the **only real network copy**; delta shrinks it | +| proc → next, **sketch mode** | `serialize_to_bytes()` → bytes in Binary col | **one serialize** (sketch isn't columnar) | +| edge → central (gRPC) | the envelope batch | the **only real network copy** | | control (`Wakeup`/`Config`/…) | tiny `NodeControlMsg` | negligible | --- @@ -144,24 +138,24 @@ Every boundary and its true cost: the *same* processor in `input=Envelope, output=Estimate` mode. - **Stock OTAP nodes** around it — OTLP/OTAP receiver, OTAP/gRPC exporter, batch, retry, signal-router. Sketches ride the standard metrics batch in the - `_asap_envelope` Binary carrier column, untouched by stock nodes. + `sketch.envelope` Binary attribute, untouched by stock nodes. --- ## Implemented status (this repo) -- **Sketch-on-wire**: KLL in sketchlib's canonical self-describing ASAPv1 - MessagePack envelope. Private proto and legacy unframed MessagePack payloads - are rejected at configuration time. Other algorithms remain usable in - scalar estimate mode where implemented. +- **Sketch-on-wire**: every sketch type, as sketchlib's self-describing ASAPv1 + envelope (full state; no deltas). The codec rejects bytes that do not + decode as the declared sketch type. - **Estimate mode** (`transmit_sketch=false`): DDSketch / KLL emit one Gauge per - configured quantile (`quantile` label); HLL emits a cardinality Gauge. - CMS / CountSketch top-k output needs a heavy-hitter tracker (not yet built). + configured quantile (`quantile` label); HLL emits a cardinality Gauge; + Count-Min with heap emits one Gauge per heavy hitter (`key`, `rank` labels). + Plain CMS / CountSketch track no keys, so they emit no estimates. - **Cross-host merge**: KLL uses sketchlib's level-aware merge, preserving the logical weights of retained items after compaction. Config knobs live on `PluginConfig` / `PrecomputeConfig`: -`encoding`, `delta_transmission`, `transmit_sketch`, `quantiles`. +`transmit_sketch`, `quantiles`. --- @@ -169,7 +163,10 @@ Config knobs live on `PluginConfig` / `PrecomputeConfig`: Each seam in the diagrams/pseudocode above, linked to source. Permalinks are pinned to the merged commits (`ASAPCollector-public@1be24ef`, -`asap_sketchlib@0a2ac37`) so the line numbers don't drift. +`asap_sketchlib@0a2ac37`) so the line numbers don't drift. Those commits +predate the ASAPv1-only wire format: the `Encoding`, delta, and +`decode_batch`/`encode_batch` rows below describe code that has since been +removed. ### The OTAP node