Skip to content

Commit 8ced299

Browse files
feat(planner): --planner saved reads RQE trees and writes rqe_configs (#888)
* feat(asap-common): RQE expression tree type and validation Add `asap_types::rqe_expr`: a serde type for RQE expression trees (rqe leaves, agg, binary, scalar nodes), a `validate` check run at load, and `leaves` / `output_labels` helpers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * feat(planner): --planner saved reads RQE trees and writes rqe_configs An e2e query's optional `expr` (an RQE expression tree, validated at load) now names the plan RQEs it reads. Each tree leaf becomes one more optimizer item and Raqe, served by its plan pick like a text leaf, and gets an entry in a new `rqe_configs` section of the inference config: RQE id -> the aggregations it reads and, for an exact roll-up, the query-time stage that regroups to the RQE's grouping. `query_configs` are unchanged; a table without trees converts as before and writes no `rqe_configs`. A query may have only a tree: its language is that of the metrics its leaves read (records -> SQL). The engine's `InferenceConfig` loads `rqe_configs` (empty when absent); nothing reads it yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * feat(asap-common): per-bucket top-k (`topk_by`) on RQE leaves An RQE leaf may name `topk_by` labels: like PromQL `topk by (...)`, a top-k leaf then ranks its input independently within each value combination of those labels, while `grouping` stays its output labels. Absent, the whole input is one bucket. Validation allows `topk_by` only on `topk_by_value` leaves, non-empty and within `grouping`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * feat(planner): plan a top-k RQE leaf by its `topk_by` buckets A top-k leaf's optimizer item now carries its `topk_by` labels as `topk_by_labels`, as a PromQL `topk by (...)` query's does, so its Raqe is grouped by the buckets (the plan's top-k RQE grouping) and its heap is one per bucket, keyed by the remaining labels. Without `topk_by` the Raqe is a single bucket and can't share the deployment of a `topk by (...)` query. Tests run the pilot random workload (seed-862/workload-01-pilot) and its saved plans: `topk by (region, service)` leaves read the same heaps as their PromQL queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * fix(asap-common): tighten RQE tree validation Require every subtree that feeds the result (root, agg operand, one side of each binary op) to read at least one RQE leaf; allow a repeated leaf id only for identical leaves; reject duplicate labels in grouping, by and topk_by; reject an empty id or metric and a zero lookback. Document that label lists are sets matched by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * fix(planner): saved-plan RQE checks per review; add --expr-only - Compare a query's `rqes` with its tree's leaves as sets. - Allow one RQE leaf at several repeat intervals: each interval is a Raqe, and the interval-independent rqe_configs entry is written once. Different leaves under one id are still refused. - Name a query in errors by its text, else its tree's RQE ids. - Compute and check a query's RQE ids once. - Check a tree's leaves read one kind of metric without dedup(). - Describe the e2e-tables fixture instead of citing its PR. - --expr-only (with --planner saved) plans only the e2e queries that have an RQE tree; the others (e.g. rate) get no deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * style(asap-common): state behaviour in duplicate-label test comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * fix(planner): require rqes with text; --expr-only plans tree leaves only - A query with `promql`/`sql` must name its `rqes`: text leaves map to plan RQEs by `rqes`, never by the tree's leaf order. Only a tree-only query may omit them. - `--expr-only` no longer plans query text: it writes only the tree leaves' deployments and `rqe_configs`, and no `query_configs`. - `leaf_item` drops the lookback check `RqeExpr::validate` owns, and shares one metric lookup (`leaf_metric`) with `is_sql`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent dd574c7 commit 8ced299

30 files changed

Lines changed: 2325 additions & 173 deletions

‎asap-common/dependencies/rs/asap_types/src/inference_config.rs‎

Lines changed: 179 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
use anyhow::Result;
22
use serde_yaml::Value;
3-
use std::collections::HashSet;
3+
use std::collections::{HashMap, HashSet};
44
use std::fs::File;
55
use std::io::BufReader;
66

@@ -21,11 +21,21 @@ pub enum SchemaConfig {
2121
ElasticSQL(SQLSchema),
2222
}
2323

24+
/// How the engine answers one RQE leaf directly: the aggregations it reads
25+
/// and the query-time stages that regroup them to the RQE's grouping.
26+
#[derive(Debug, Clone)]
27+
pub struct RqeConfig {
28+
pub aggregations: Vec<AggregationReference>,
29+
pub query_time_aggregations: Vec<QueryTimeAggregation>,
30+
}
31+
2432
#[derive(Debug, Clone)]
2533
pub struct InferenceConfig {
2634
pub schema: SchemaConfig,
2735
pub query_configs: Vec<QueryConfig>,
2836
pub cleanup_policy: CleanupPolicy,
37+
/// RQE id -> how to answer that RQE leaf; empty unless planned from RQE trees.
38+
pub rqe_configs: HashMap<String, RqeConfig>,
2939
}
3040

3141
impl InferenceConfig {
@@ -42,6 +52,7 @@ impl InferenceConfig {
4252
schema,
4353
query_configs: Vec::new(),
4454
cleanup_policy,
55+
rqe_configs: HashMap::new(),
4556
}
4657
}
4758

@@ -75,11 +86,13 @@ impl InferenceConfig {
7586

7687
let cleanup_policy = Self::parse_cleanup_policy(data)?;
7788
let query_configs = Self::parse_query_configs(data, cleanup_policy)?;
89+
let rqe_configs = Self::parse_rqe_configs(data, cleanup_policy)?;
7890

7991
Ok(Self {
8092
schema,
8193
query_configs,
8294
cleanup_policy,
95+
rqe_configs,
8396
})
8497
}
8598

@@ -263,43 +276,11 @@ impl InferenceConfig {
263276
.map_err(anyhow::Error::from)
264277
})?;
265278

266-
let aggregations = if let Some(aggregations_data) =
267-
query_data.get("aggregations").and_then(|v| v.as_sequence())
268-
{
269-
let mut agg_refs = Vec::new();
270-
for agg_data in aggregations_data {
271-
let aggregation_id = agg_data
272-
.get("aggregation_id")
273-
.and_then(|v| v.as_u64())
274-
.ok_or_else(|| {
275-
anyhow::anyhow!("Missing aggregation_id in aggregation")
276-
})?;
277-
278-
let agg_ref = match cleanup_policy {
279-
CleanupPolicy::CircularBuffer => {
280-
let num_aggregates_to_retain = agg_data
281-
.get("num_aggregates_to_retain")
282-
.and_then(|v| v.as_u64());
283-
AggregationReference::new(aggregation_id, num_aggregates_to_retain)
284-
}
285-
CleanupPolicy::ReadBased => {
286-
let read_count_threshold = agg_data
287-
.get("read_count_threshold")
288-
.and_then(|v| v.as_u64());
289-
AggregationReference::with_read_count_threshold(
290-
aggregation_id,
291-
read_count_threshold,
292-
)
293-
}
294-
CleanupPolicy::NoCleanup => {
295-
AggregationReference::new(aggregation_id, None)
296-
}
297-
};
298-
agg_refs.push(agg_ref);
279+
let aggregations = match query_data.get("aggregations") {
280+
Some(aggregations) => {
281+
Self::parse_aggregation_refs(aggregations, cleanup_policy)?
299282
}
300-
agg_refs
301-
} else {
302-
Vec::new()
283+
None => Vec::new(),
303284
};
304285

305286
let config =
@@ -316,11 +297,97 @@ impl InferenceConfig {
316297
};
317298
Ok(query_configs)
318299
}
300+
301+
/// A sequence of `{aggregation_id, num_aggregates_to_retain | read_count_threshold}`.
302+
fn parse_aggregation_refs(
303+
aggregations_data: &Value,
304+
cleanup_policy: CleanupPolicy,
305+
) -> Result<Vec<AggregationReference>> {
306+
let Some(aggregations_data) = aggregations_data.as_sequence() else {
307+
return Ok(Vec::new());
308+
};
309+
let mut agg_refs = Vec::new();
310+
for agg_data in aggregations_data {
311+
let aggregation_id = agg_data
312+
.get("aggregation_id")
313+
.and_then(|v| v.as_u64())
314+
.ok_or_else(|| anyhow::anyhow!("Missing aggregation_id in aggregation"))?;
315+
316+
let agg_ref = match cleanup_policy {
317+
CleanupPolicy::CircularBuffer => {
318+
let num_aggregates_to_retain = agg_data
319+
.get("num_aggregates_to_retain")
320+
.and_then(|v| v.as_u64());
321+
AggregationReference::new(aggregation_id, num_aggregates_to_retain)
322+
}
323+
CleanupPolicy::ReadBased => {
324+
let read_count_threshold = agg_data
325+
.get("read_count_threshold")
326+
.and_then(|v| v.as_u64());
327+
AggregationReference::with_read_count_threshold(
328+
aggregation_id,
329+
read_count_threshold,
330+
)
331+
}
332+
CleanupPolicy::NoCleanup => AggregationReference::new(aggregation_id, None),
333+
};
334+
agg_refs.push(agg_ref);
335+
}
336+
Ok(agg_refs)
337+
}
338+
339+
/// The optional `rqe_configs` mapping: RQE id -> `{aggregations,
340+
/// query_time_aggregations}`, both required.
341+
fn parse_rqe_configs(
342+
data: &Value,
343+
cleanup_policy: CleanupPolicy,
344+
) -> Result<HashMap<String, RqeConfig>> {
345+
let Some(rqe_configs) = data.get("rqe_configs") else {
346+
return Ok(HashMap::new());
347+
};
348+
let rqe_configs = rqe_configs
349+
.as_mapping()
350+
.ok_or_else(|| anyhow::anyhow!("rqe_configs must map RQE ids to configs"))?;
351+
let mut configs = HashMap::new();
352+
for (id, config) in rqe_configs {
353+
let id = id
354+
.as_str()
355+
.ok_or_else(|| anyhow::anyhow!("rqe_configs key {id:?} is not a string"))?;
356+
let aggregations = config
357+
.get("aggregations")
358+
.ok_or_else(|| anyhow::anyhow!("RQE {id:?}: missing aggregations"))
359+
.and_then(|value| Self::parse_aggregation_refs(value, cleanup_policy))?;
360+
if aggregations.is_empty() {
361+
anyhow::bail!("RQE {id:?}: aggregations is empty");
362+
}
363+
let query_time_aggregations = config
364+
.get("query_time_aggregations")
365+
.ok_or_else(|| anyhow::anyhow!("RQE {id:?}: missing query_time_aggregations"))
366+
.and_then(|value| {
367+
serde_yaml::from_value::<Vec<QueryTimeAggregation>>(value.clone())
368+
.map_err(anyhow::Error::from)
369+
})?;
370+
for stage in &query_time_aggregations {
371+
stage
372+
.validate()
373+
.map_err(|error| anyhow::anyhow!("RQE {id:?}: {error}"))?;
374+
}
375+
configs.insert(
376+
id.to_string(),
377+
RqeConfig {
378+
aggregations,
379+
query_time_aggregations,
380+
},
381+
);
382+
}
383+
Ok(configs)
384+
}
319385
}
320386

321387
#[cfg(test)]
322388
mod tests {
323389
use super::*;
390+
use crate::query_config::QueryTimeAggregationOperator;
324391

325392
#[test]
326393
fn rejects_a_query_without_a_planned_subquery() {
@@ -390,4 +457,79 @@ queries:
390457

391458
assert!(error.to_string().contains("quantile"));
392459
}
460+
461+
#[test]
462+
fn loads_rqe_configs_with_their_roll_up() {
463+
let data: Value = serde_yaml::from_str(
464+
r#"
465+
cleanup_policy:
466+
name: circular_buffer
467+
metrics: {}
468+
queries: []
469+
rqe_configs:
470+
sum_by_job:
471+
aggregations:
472+
- aggregation_id: 2
473+
num_aggregates_to_retain: 5
474+
query_time_aggregations:
475+
- operator: sum
476+
grouping:
477+
mode: by
478+
labels: [job]
479+
parameter: null
480+
"#,
481+
)
482+
.unwrap();
483+
484+
let config = InferenceConfig::from_yaml_data(&data, QueryLanguage::promql).unwrap();
485+
486+
let rqe = &config.rqe_configs["sum_by_job"];
487+
let [aggregation] = rqe.aggregations.as_slice() else {
488+
panic!("{:?}", rqe.aggregations);
489+
};
490+
assert_eq!(aggregation.aggregation_id, 2);
491+
assert_eq!(aggregation.num_aggregates_to_retain, Some(5));
492+
let [stage] = rqe.query_time_aggregations.as_slice() else {
493+
panic!("{:?}", rqe.query_time_aggregations);
494+
};
495+
assert!(matches!(stage.operator, QueryTimeAggregationOperator::Sum));
496+
assert_eq!(stage.grouping.labels, ["job"]);
497+
}
498+
499+
#[test]
500+
fn a_config_without_rqe_configs_has_none() {
501+
let data: Value =
502+
serde_yaml::from_str("cleanup_policy: {name: no_cleanup}\nmetrics: {}\n").unwrap();
503+
let config = InferenceConfig::from_yaml_data(&data, QueryLanguage::promql).unwrap();
504+
assert!(config.rqe_configs.is_empty());
505+
}
506+
507+
#[test]
508+
fn rejects_an_rqe_config_missing_a_field_or_with_an_invalid_stage() {
509+
for (rqe, reason) in [
510+
("{query_time_aggregations: []}", "missing aggregations"),
511+
(
512+
"{aggregations: [], query_time_aggregations: []}",
513+
"aggregations is empty",
514+
),
515+
(
516+
"{aggregations: [{aggregation_id: 1}]}",
517+
"missing query_time_aggregations",
518+
),
519+
(
520+
"{aggregations: [{aggregation_id: 1}], query_time_aggregations: \
521+
[{operator: sum, grouping: {mode: by, labels: []}, parameter: null}]}",
522+
"by grouping must name at least one label",
523+
),
524+
] {
525+
let yaml = format!(
526+
"cleanup_policy: {{name: no_cleanup}}\nmetrics: {{}}\nrqe_configs: {{r: {rqe}}}\n"
527+
);
528+
let data: Value = serde_yaml::from_str(&yaml).unwrap();
529+
let error =
530+
InferenceConfig::from_yaml_data(&data, QueryLanguage::promql).expect_err("refused");
531+
assert!(error.to_string().contains(reason), "{error}");
532+
assert!(error.to_string().contains("\"r\""), "{error}");
533+
}
534+
}
393535
}

‎asap-common/dependencies/rs/asap_types/src/streaming_config.rs‎

Lines changed: 77 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,8 +55,19 @@ impl StreamingConfig {
5555
let mut read_count_threshold_map: HashMap<u64, u64> = HashMap::new();
5656

5757
if let Some(inference_config) = inference_config {
58-
for query_config in &inference_config.query_configs {
59-
for aggregation in &query_config.aggregations {
58+
// Every reader of an aggregation: text queries and RQE leaves.
59+
let references = inference_config
60+
.query_configs
61+
.iter()
62+
.map(|q| &q.aggregations)
63+
.chain(
64+
inference_config
65+
.rqe_configs
66+
.values()
67+
.map(|r| &r.aggregations),
68+
);
69+
for aggregations in references {
70+
for aggregation in aggregations {
6071
let aggregation_id = aggregation.aggregation_id;
6172
let num_aggregates = aggregation.num_aggregates_to_retain;
6273
retention_map
@@ -152,7 +163,7 @@ mod tests {
152163
use crate::aggregation_config::AggregationConfigError;
153164
use crate::aggregation_reference::AggregationReference;
154165
use crate::enums::{CleanupPolicy, QueryLanguage};
155-
use crate::inference_config::InferenceConfig;
166+
use crate::inference_config::{InferenceConfig, RqeConfig};
156167
use crate::query_config::QueryConfig;
157168
use promql_utilities::query_logics::enums::AggregationType;
158169

@@ -523,4 +534,67 @@ aggregations:
523534
.expect_err("an overflowing threshold must not load");
524535
assert!(error.to_string().contains("overflowed"), "{error}");
525536
}
537+
538+
fn rqe_config(reference: AggregationReference) -> RqeConfig {
539+
RqeConfig {
540+
aggregations: vec![reference],
541+
query_time_aggregations: Vec::new(),
542+
}
543+
}
544+
545+
#[test]
546+
fn rqe_only_reference_bounds_retention() {
547+
// An aggregation only an RQE leaf reads keeps that leaf's windows,
548+
// not every window ever built.
549+
let yaml = minmax_yaml("MinMax", "max");
550+
let mut inference =
551+
InferenceConfig::new(QueryLanguage::promql, CleanupPolicy::CircularBuffer);
552+
inference.rqe_configs.insert(
553+
"q0".into(),
554+
rqe_config(AggregationReference::new(1, Some(4))),
555+
);
556+
557+
let config = StreamingConfig::from_yaml_data(&yaml, Some(&inference))
558+
.expect("an RQE-only aggregation should load");
559+
560+
assert_eq!(config[1].num_aggregates_to_retain, Some(4));
561+
}
562+
563+
#[test]
564+
fn query_and_rqe_references_keep_the_largest_retention() {
565+
// A shared aggregation keeps the longer lookback, whichever reader has it.
566+
let yaml = minmax_yaml("MinMax", "max");
567+
for (query_retain, rqe_retain) in [(7, 2), (2, 7)] {
568+
let mut inference =
569+
InferenceConfig::new(QueryLanguage::promql, CleanupPolicy::CircularBuffer);
570+
inference.query_configs = vec![QueryConfig::new("max(metric[1m])".into())
571+
.add_aggregation(AggregationReference::new(1, Some(query_retain)))];
572+
inference.rqe_configs.insert(
573+
"q0".into(),
574+
rqe_config(AggregationReference::new(1, Some(rqe_retain))),
575+
);
576+
577+
let config = StreamingConfig::from_yaml_data(&yaml, Some(&inference))
578+
.expect("a shared aggregation should load");
579+
580+
assert_eq!(config[1].num_aggregates_to_retain, Some(7));
581+
}
582+
}
583+
584+
#[test]
585+
fn query_and_rqe_references_sum_read_based_thresholds() {
586+
let yaml = minmax_yaml("MinMax", "max");
587+
let mut inference = InferenceConfig::new(QueryLanguage::promql, CleanupPolicy::ReadBased);
588+
inference.query_configs = vec![QueryConfig::new("max(metric[1m])".into())
589+
.add_aggregation(AggregationReference::with_read_count_threshold(1, Some(2)))];
590+
inference.rqe_configs.insert(
591+
"q0".into(),
592+
rqe_config(AggregationReference::with_read_count_threshold(1, Some(3))),
593+
);
594+
595+
let config = StreamingConfig::from_yaml_data(&yaml, Some(&inference))
596+
.expect("a shared aggregation should load");
597+
598+
assert_eq!(config[1].read_count_threshold, Some(5));
599+
}
526600
}

0 commit comments

Comments
 (0)