Add telemetry-function experiment results (inline cost, exact baseline, accuracy vs size)

docs/results/accuracy-size-cms.csv

@@ -0,0 +1,5 @@
+use_case,variant,offered_per_source,branch_cpu_cores,branch_rss_mib,exact_cpu_cores,delivered_points_per_s,sketch_answer_delay_s,exact_answer_delay_s,param,recall_at_10,count_rel_error_mean,merge_bytes_per_window
+topk,with,20000,0.13017390846217503,38.1953125,0.22413272861818173,40000.0,2.8564,5.0762,cms_columns=256,1.0,0.059888626820811815,7075.871347793676
+topk,with,20000,0.13129619608181814,37.91796875,0.22298338972326304,40000.0,2.985333333333333,5.8839999999999995,cms_columns=512,1.0,0.028160700726932348,11018.363636363636
+topk,with,20000,0.13096587330909093,37.8984375,0.2276244130256913,40000.0,2.8488333333333333,5.4864,cms_columns=1024,1.0,0.01253763218771096,11659.3028964163
+topk,with,20000,0.1310888823363636,37.97265625,0.2216046038799294,40000.0,2.49,5.6916,cms_columns=2048,1.0,0.004619899204047916,18537.06431026019

docs/results/accuracy-size-kll.csv

@@ -0,0 +1,5 @@
+use_case,variant,offered_per_source,branch_cpu_cores,branch_rss_mib,exact_cpu_cores,delivered_points_per_s,sketch_answer_delay_s,exact_answer_delay_s,param,p99_rel_error_median,p99_rel_error_max,merge_bytes_per_window,recall_at_10,count_rel_error_mean
+latency,with,20000,0.09914191567272729,37.0390625,0.2092542768413969,33333.333333333336,2.6765999999999996,5.511,kll_k=50,0.1330302060067105,0.27856753406962903,39080.56217159688,,
+latency,with,20000,0.09827948084545457,37.2578125,0.21181707490909082,33333.333333333336,2.5806,6.0404,kll_k=100,0.06974517878446874,0.3029436095130493,58343.48466391521,,
+latency,with,20000,0.1003896176727273,37.39453125,0.20892087658181807,33333.333333333336,2.7338,5.2328,kll_k=200,0.07542263925915717,0.14752570499183654,96150.90909090909,,
+latency,with,20000,0.09849296526872751,37.26953125,0.20987031667272726,33333.333333333336,2.9702,5.8506,kll_k=400,0.020337662632778274,0.06450106158736299,181082.74695903563,,

docs/results/exact-baseline.csv

@@ -0,0 +1,25 @@
+use_case,variant,offered_per_source,branch_cpu_cores,branch_rss_mib,exact_cpu_cores,delivered_points_per_s,sketch_answer_delay_s,exact_answer_delay_s
+latency,without,20000,0.009161495090909091,32.3359375,0.1738272694412625,40000.0,nan,5.908166666666666
+latency,with,20000,0.09781009715877434,37.55078125,0.19620479176742237,33333.333333333336,2.4765,5.396800000000001
+latency,without,50000,0.01977753126363636,33.59765625,0.2905816402610857,83333.33333333333,nan,5.308999999999999
+latency,with,50000,0.2057437769595813,38.59375,0.461242614520264,100000.0,2.1461666666666663,6.3165000000000004
+latency,without,100000,0.03893441778773481,36.01171875,0.7708123132909094,200000.0,nan,6.0255
+latency,with,100000,0.3874080315812437,40.12890625,0.7799896220181812,200000.0,2.9143333333333334,6.562666666666667
+latency,without,200000,0.07929270387924721,39.2578125,0.9753325697636357,395910.4,nan,6.458333333333333
+latency,with,200000,0.6970759094471008,53.25,0.9963856179090906,392696.5333333333,2.994666666666667,6.226666666666667
+cardinality,without,20000,0.01497283973927747,34.33984375,0.5903903819705812,40000.0,nan,5.7044999999999995
+cardinality,with,20000,0.31779410332622304,42.921875,0.4853736946489934,40000.0,2.6901666666666664,5.699
+cardinality,without,50000,0.02953429046194209,45.171875,0.9093562941401065,83333.33333333333,nan,6.8186
+cardinality,with,50000,0.6353617936259432,85.21875,0.9181438344757369,83333.33333333333,2.8605,6.487399999999999
+cardinality,without,100000,0.03483813425141277,52.8515625,0.9984272099454549,98066.13333333333,nan,7.4366
+cardinality,with,100000,0.6049840947727274,106.5390625,0.9693291110181812,94635.73333333334,2.519833333333333,7.1532
+cardinality,without,200000,0.03334013094858605,52.2578125,0.9955248798159969,117008.0,nan,7.319333333333333
+cardinality,with,200000,0.6182521825132712,106.58203125,0.9675288886161569,93606.4,2.6548,7.2228
+topk,without,20000,0.00929132681420305,32.12109375,0.1544458227850403,33333.333333333336,nan,5.8102
+topk,with,20000,0.12957542427272728,38.09765625,0.16892617122129475,33333.333333333336,2.6591666666666662,5.019
+topk,without,50000,0.02018486886446431,33.58203125,0.35918850440901057,83333.33333333333,nan,5.401000000000001
+topk,with,50000,0.28545476954684756,38.796875,0.37081463252727254,83333.33333333333,2.5073333333333334,5.5732
+topk,without,100000,0.039291462927586125,36.1484375,0.6095100602,166666.66666666666,nan,5.9116
+topk,with,100000,0.5424276417533571,39.484375,0.7120886668727273,166666.66666666666,2.3085999999999998,5.7638
+topk,without,200000,0.0781240751077253,38.64453125,0.8842406513454546,333333.3333333333,nan,5.316
+topk,with,200000,0.7685498032411674,51.8046875,0.8950591008545447,333316.26666666666,2.079,5.905399999999999

docs/results/experiments.md

@@ -0,0 +1,78 @@
+# Telemetry-function experiments
+
+Two identical 56-core x86 servers (Ubuntu 22.04), one for inline cost and one for the exact baseline and accuracy, 2026-10-12. Every stage is an `asap-collector` container pinned to its own CPU (deploy/telemetry-functions), with 10-second windows (`experiments.py`); each point is one 60-second measurement after a 40-second warm-up.
+
+## Findings
+
+- **Inline cost per point** (extra branch CPU over plain forwarding, which costs about 0.3 µs): KLL 2.7–4.1 µs, Count-Min + heap 3.7–5.3 µs, HLL over four attribute keys 9–11 µs (about 2.3 µs per key). At 200,000 points/s per branch that is 0.60 cores for KLL and 0.81 for Count-Min + heap; the four-key HLL branch reaches one core at about 100,000 points/s (0.96 cores at 100,000). At 200,000 points/s per source the cardinality generator itself tops out at about 130,000 points/s, which is why that use case's *without* line flattens. These rates are below the ~1M points/s per core reference, so the next step is a profile of the per-point path (decoding OTAP into observations and the per-attribute split are the likely costs; not yet measured).
+- **The global stages cost almost nothing.** Merge and detector receive a few KB to 180 KB per 10-second window and use well under 0.01 cores. The exact baseline, which receives every raw point, needs one core at about 95,000 points/s for cardinality (it holds every distinct value) and at about 400,000 points/s for latency quantiles.
+- **Answers arrive 2–3 s after each window ends** (these runs used a 2-second merge grace; deployments use 5 s).
+- **Accuracy follows sketch size.** KLL p99 error drops from 13% (k = 50) to 2% (k = 400) for routes with at least 20,000 points per window, while merged bytes grow from 39 KB to 181 KB per window. Count-Min + heap over 100,000 keys keeps recall@10 at 1.0 at every width; the mean count error of the top 10 falls from 6.0% (256 columns) to 0.46% (2,048 columns). Each setting is one window, so neighbouring settings can cross (k = 100 vs 200).
+- **Not covered yet:** sampling plus sketches, the Arrow-native layout, and multi-machine runs.
+
+## Inline cost
+
+Each branch receives one source. *Without* forwards raw points only; *with* adds the fanout and the sketch. CPU is the branch pipeline's own CPU time (average of the two branches). The exact stage only counts points here, so it never slows the branches; *added per point* divides the extra branch CPU by the points each branch delivered.
+
+### Latency quantiles (KLL)
+
+| Offered per source | Delivered per branch | Branch CPU without | Branch CPU with | Added per point | Branch memory with | Sketch answer delay |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 20,000 | 20,000 | 0.007 cores | 0.089 cores | 4.1 µs | 37 MiB | 2.1 s |
+| 50,000 | 49,846 | 0.015 cores | 0.171 cores | 3.1 µs | 39 MiB | 2.9 s |
+| 100,000 | 100,000 | 0.030 cores | 0.310 cores | 2.8 µs | 38 MiB | 2.2 s |
+| 200,000 | 200,000 | 0.060 cores | 0.603 cores | 2.7 µs | 50 MiB | 2.7 s |
+
+### Attribute cardinality (HLL)
+
+| Offered per source | Delivered per branch | Branch CPU without | Branch CPU with | Added per point | Branch memory with | Sketch answer delay |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 20,000 | 20,000 | 0.011 cores | 0.225 cores | 10.7 µs | 42 MiB | 2.2 s |
+| 50,000 | 50,000 | 0.026 cores | 0.513 cores | 9.7 µs | 42 MiB | 2.6 s |
+| 100,000 | 99,990 | 0.054 cores | 0.961 cores | 9.1 µs | 73 MiB | 2.7 s |
+| 200,000 | 102,989 | 0.055 cores | 0.993 cores | 9.1 µs | 73 MiB | 2.1 s |
+
+### Heavy hitters (Count-Min + heap)
+
+| Offered per source | Delivered per branch | Branch CPU without | Branch CPU with | Added per point | Branch memory with | Sketch answer delay |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 20,000 | 20,000 | 0.007 cores | 0.113 cores | 5.3 µs | 38 MiB | 2.9 s |
+| 50,000 | 49,980 | 0.015 cores | 0.232 cores | 4.3 µs | 38 MiB | 2.3 s |
+| 100,000 | 100,000 | 0.030 cores | 0.434 cores | 4.0 µs | 38 MiB | 2.8 s |
+| 200,000 | 200,000 | 0.059 cores | 0.808 cores | 3.7 µs | 51 MiB | 2.6 s |
+
+![Branch CPU with and without each function](inline-cost.png)
+
+## Exact baseline cost
+
+The alternative to inline sketches: ship every raw point to one place and compute exact answers. Here the exact stage (one pinned CPU) holds each window's raw values or distinct values; when it saturates, fanout back-pressure lowers the delivered rate. Delivered rates are approximate (5-6 ten-second windows per measurement).
+
+| Use case | Offered (both sources) | Exact stage CPU | Points delivered |
+| --- | ---: | ---: | ---: |
+| Latency quantiles (KLL) | 40,000/s | 0.20 cores | 33,333/s |
+| Latency quantiles (KLL) | 100,000/s | 0.46 cores | 100,000/s |
+| Latency quantiles (KLL) | 200,000/s | 0.78 cores | 200,000/s |
+| Latency quantiles (KLL) | 400,000/s | 1.00 cores | 392,697/s |
+| Attribute cardinality (HLL) | 40,000/s | 0.49 cores | 40,000/s |
+| Attribute cardinality (HLL) | 100,000/s | 0.92 cores | 83,333/s |
+| Attribute cardinality (HLL) | 200,000/s | 0.97 cores | 94,636/s |
+| Attribute cardinality (HLL) | 400,000/s | 0.97 cores | 93,606/s |
+| Heavy hitters (Count-Min + heap) | 40,000/s | 0.17 cores | 33,333/s |
+| Heavy hitters (Count-Min + heap) | 100,000/s | 0.37 cores | 83,333/s |
+| Heavy hitters (Count-Min + heap) | 200,000/s | 0.71 cores | 166,667/s |
+| Heavy hitters (Count-Min + heap) | 400,000/s | 0.90 cores | 333,316/s |
+
+## Accuracy versus sketch size
+
+One window per setting, at 20,000 points/s per source. KLL: the latency scenario (20 routes). Count-Min + heap: 100,000 Zipf routes so collisions matter. Merge bytes per window is what the merge stage receives from both branches.
+
+| Setting | p99 error, median / max (routes >= 20,000 points) | Recall@10 | Mean count error, top 10 | Merge bytes per window |
+| --- | ---: | ---: | ---: | ---: |
+| cms_columns=256 |  | 1.00 | 5.99% | 7,076 B |
+| cms_columns=512 |  | 1.00 | 2.82% | 11,018 B |
+| cms_columns=1024 |  | 1.00 | 1.25% | 11,659 B |
+| cms_columns=2048 |  | 1.00 | 0.46% | 18,537 B |
+| kll_k=50 | 13.30% / 27.86% |  |  | 39,081 B |
+| kll_k=100 | 6.97% / 30.29% |  |  | 58,343 B |
+| kll_k=200 | 7.54% / 14.75% |  |  | 96,151 B |
+| kll_k=400 | 2.03% / 6.45% |  |  | 181,083 B |

docs/results/inline-cost.csv

@@ -0,0 +1,25 @@
+use_case,variant,offered_per_source,branch_cpu_cores,branch_rss_mib,exact_cpu_cores,delivered_points_per_s,sketch_answer_delay_s,exact_answer_delay_s
+latency,without,20000,0.006514463804584545,32.02734375,0.13390540290909092,40000.0,nan,5.731833333333334
+latency,with,20000,0.08886799941973197,37.30859375,0.18230652136363626,40000.0,2.126,5.6122
+latency,without,50000,0.014891407707498484,32.875,0.2768239844184651,100000.0,nan,5.915666666666667
+latency,with,50000,0.17091746561818177,38.66015625,0.3473995991818185,99692.8,2.8523333333333336,5.6646
+latency,without,100000,0.030296921337542176,35.28515625,0.5023240544909093,200000.0,nan,5.4322
+latency,with,100000,0.3097622100363636,37.56640625,0.5853916397818184,200000.0,2.2468333333333335,5.679333333333333
+latency,without,200000,0.05964040422363783,37.93359375,0.9762311017109417,400020.48,nan,5.1158
+latency,with,200000,0.6025151805200473,49.97265625,0.982679691363636,400000.0,2.6931666666666665,5.202666666666667
+cardinality,without,20000,0.010867961180807921,31.41015625,0.18719079880000006,40000.0,nan,5.1552
+cardinality,with,20000,0.22526361957630564,42.06640625,0.27273168492727273,40000.0,2.178,5.072
+cardinality,without,50000,0.026328344951726392,37.6015625,0.38504750582716674,100051.2,nan,5.535833333333333
+cardinality,with,50000,0.5132392756991413,42.12890625,0.5207149674181819,100000.0,2.6315999999999997,5.506399999999999
+cardinality,without,100000,0.0544423877434423,31.0859375,0.7687145079090896,200000.0,nan,5.463333333333334
+cardinality,with,100000,0.9610451166727276,73.24609375,0.7776510720545452,199979.52,2.7383333333333333,5.5686
+cardinality,without,200000,0.05458064601829739,51.6328125,0.9990883847888139,258478.08,nan,5.5446
+cardinality,with,200000,0.9926015046454546,72.93359375,0.800502983981817,205978.66666666666,2.1376666666666666,5.0089999999999995
+topk,without,20000,0.00654013174784996,31.6953125,0.12934254928271424,40000.0,nan,5.159166666666667
+topk,with,20000,0.1126095087217051,37.67578125,0.18262559174545445,40000.0,2.9118,5.4892
+topk,without,50000,0.015008962902369046,33.0234375,0.2785002468682388,100000.0,nan,5.458833333333334
+topk,with,50000,0.23216660726278482,38.47265625,0.35610990427272754,99959.04,2.314833333333333,5.92
+topk,without,100000,0.02985230547636159,35.40625,0.5120796644909097,200000.0,nan,5.0878
+topk,with,100000,0.4335742304478749,38.41015625,0.6294457554909091,200000.0,2.754,5.310666666666666
+topk,without,200000,0.05854284119090907,33.78515625,0.9948354450000007,400000.0,nan,5.5274
+topk,with,200000,0.8075293743454555,51.37109375,0.9784585726026392,400000.0,2.581,5.4685