Skip to content

Commit 90cd86e

Browse files
committed
feat(precompute): event-time barrier for Remote Write producers
Ports #814 to the plan-executing engine. POST /api/v1/precompute/watermark?time_ms=T: the producer states that every sample at or before T has been written. The request queues an AdvanceWatermark command behind every write already acknowledged and answers once it has run: every pane with end <= T is closed, keeping the left-open (end - width, end] convention, so the last pane of a series that stopped sending is built and its window served from the plan without a later sample. - T is capped at the wall clock, like the sample watermark. - A repeated or lower T does nothing; the answer is the barrier in force. - The closing reuses the expired-pane skip and checked arithmetic, so T = i64::MAX terminates. - Input stays open; samples at or before T arriving later count as late. - With several producers, the barrier must follow all of their writes through T; the README says so.
1 parent d1feb90 commit 90cd86e

6 files changed

Lines changed: 294 additions & 8 deletions

File tree

‎README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,15 @@ are dropped. Remote Write gets 503 when admitting it would leave more than
5353
stopped, which also makes `/-/ready` fail. `/api/v1/status/asap` reports the
5454
engine's sample and pane counters as JSON.
5555

56+
A producer can close panes without waiting for later samples:
57+
`POST /api/v1/precompute/watermark?time_ms=T` states that every sample at or
58+
before `T` has been written. After the writes acknowledged before it have
59+
been ingested, every pane ending at or before `T` (capped at the wall clock)
60+
is closed, so the last pane of a series that stopped sending is built and
61+
served. A repeated or lower `T` does nothing, input stays open, and samples
62+
at or before `T` that arrive later count as late. With several producers,
63+
send the barrier only after all of them have written through `T`.
64+
5665
An instant query whose text is one of the plan's queries is answered by the
5766
plan (`x-asap-execution: plan`). Its stored inputs read the panes ending at
5867
`T - offset`, so `T` must be on the pane grid and those panes stored; its

‎data_plane/src/engine.rs‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,10 @@ enum Command {
5050
raw: BTreeMap<u64, Vec<RawSample>>,
5151
reply: Reply<Vec<ResultSample>>,
5252
},
53+
AdvanceWatermark {
54+
barrier_ms: i64,
55+
reply: tokio::sync::oneshot::Sender<i64>,
56+
},
5357
Stop,
5458
}
5559

@@ -205,6 +209,21 @@ impl EngineHandle {
205209
*self.stats.lock().unwrap_or_else(|e| e.into_inner())
206210
}
207211

212+
/// Queues an event-time barrier behind every write already acknowledged
213+
/// and waits until it has run: every pane ending at or before
214+
/// `barrier_ms` (capped at the wall clock) is then closed. Returns the
215+
/// barrier in force; see [`Precompute::advance_watermark_at`].
216+
pub async fn advance_watermark(&self, barrier_ms: i64) -> Result<i64, IngestError> {
217+
if !self.is_running() {
218+
return Err(IngestError::Stopped);
219+
}
220+
let (reply, done) = tokio::sync::oneshot::channel();
221+
self.commands
222+
.send(Command::AdvanceWatermark { barrier_ms, reply })
223+
.map_err(|_| IngestError::Stopped)?;
224+
done.await.map_err(|_| IngestError::Stopped)
225+
}
226+
208227
/// For tests: stops the engine thread after the commands queued before.
209228
/// Samples queued between this call and the stop are acknowledged and
210229
/// then lost; ingestion after the stop fails with
@@ -294,6 +313,11 @@ impl Engine {
294313
&raw,
295314
));
296315
}
316+
Command::AdvanceWatermark { barrier_ms, reply } => {
317+
let barrier_ms = self.precompute.advance_watermark(barrier_ms);
318+
*shared.stats.lock().unwrap_or_else(|e| e.into_inner()) = self.precompute.stats;
319+
let _ = reply.send(barrier_ms);
320+
}
297321
Command::Stop => break,
298322
}
299323
}

‎data_plane/src/lib.rs‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ pub fn router(state: Arc<DataPlane>) -> Router {
106106
.route("/api/v1/query", get(query).post(query))
107107
.route("/api/v1/query_range", get(query_range).post(query_range))
108108
.route("/api/v1/status/asap", get(status))
109+
.route("/api/v1/precompute/watermark", post(watermark))
109110
.route("/-/ready", get(ready))
110111
.with_state(state)
111112
}
@@ -159,6 +160,37 @@ async fn write(State(state): State<Arc<DataPlane>>, body: Bytes) -> Response {
159160
}
160161
}
161162

163+
/// `POST /api/v1/precompute/watermark?time_ms=T`: the producer states that
164+
/// every sample at or before `T` has been written. Once the engine has run
165+
/// every write acknowledged before this request, every pane with
166+
/// `end <= T` is closed, so the last pane of a series that stopped sending
167+
/// is built without a later sample. `T` is capped at the wall clock; a
168+
/// repeated or lower `T` does nothing; samples at or before it that arrive
169+
/// later count as late. Input stays open. With several producers, call it
170+
/// only after all of them have written through `T`. Answers
171+
/// `{"watermark_ms": barrier in force}`.
172+
async fn watermark(State(state): State<Arc<DataPlane>>, RawQuery(query): RawQuery) -> Response {
173+
let time_ms = form_urlencoded::parse(query.unwrap_or_default().as_bytes())
174+
.find(|(name, _)| name == "time_ms")
175+
.and_then(|(_, value)| value.parse::<i64>().ok());
176+
let Some(time_ms) = time_ms else {
177+
return (
178+
StatusCode::BAD_REQUEST,
179+
"time_ms (integer milliseconds) is required",
180+
)
181+
.into_response();
182+
};
183+
match state.engine.advance_watermark(time_ms).await {
184+
Ok(barrier_ms) => {
185+
axum::Json(serde_json::json!({ "watermark_ms": barrier_ms })).into_response()
186+
}
187+
Err(error) => {
188+
tracing::error!(%error, "event-time barrier refused");
189+
(StatusCode::INTERNAL_SERVER_ERROR, error.to_string()).into_response()
190+
}
191+
}
192+
}
193+
162194
/// Ready while the engine thread runs; a stopped engine never ingests
163195
/// again, so the data plane must be restarted.
164196
async fn ready(State(state): State<Arc<DataPlane>>) -> StatusCode {

‎data_plane/src/precompute.rs‎

Lines changed: 40 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,9 @@ pub struct Precompute {
207207
/// passes it, so a series stamped ahead of the others cannot close
208208
/// panes early.
209209
newest_clock_ms: Option<i64>,
210+
/// The newest event-time barrier a producer declared: every pane ending
211+
/// at or before it is closed.
212+
barrier_ms: Option<i64>,
210213
pub store: SummaryStore<Batch>,
211214
pub stats: PrecomputeStats,
212215
}
@@ -250,28 +253,50 @@ impl Precompute {
250253
max_future_ms: millis(options.max_future_ms),
251254
newest_sample_ms: None,
252255
newest_clock_ms: None,
256+
barrier_ms: None,
253257
store: SummaryStore::new(),
254258
stats: PrecomputeStats::default(),
255259
})
256260
}
257261

258262
/// The watermark: no sample older than this is expected any more. It is
259263
/// the newest routed sample, capped at the wall clock, minus the allowed
260-
/// lateness.
264+
/// lateness, or the newest barrier if that is later.
261265
pub fn watermark_ms(&self) -> Option<i64> {
262266
let newest = match self.newest_clock_ms {
263-
Some(clock) => self.newest_sample_ms?.min(clock),
264-
None => self.newest_sample_ms?,
267+
Some(clock) => self.newest_sample_ms.map(|newest| newest.min(clock)),
268+
None => self.newest_sample_ms,
265269
};
266-
Some(newest.saturating_sub(self.allowed_lateness_ms))
270+
let from_samples = newest.map(|newest| newest.saturating_sub(self.allowed_lateness_ms));
271+
from_samples.max(self.barrier_ms)
272+
}
273+
274+
/// [`Self::advance_watermark_at`] the current wall-clock time.
275+
pub fn advance_watermark(&mut self, barrier_ms: i64) -> i64 {
276+
self.advance_watermark_at(barrier_ms, wall_clock_ms())
277+
}
278+
279+
/// An event-time barrier: the producer states that every sample at or
280+
/// before `barrier_ms` has been written. Closes every pane with
281+
/// `end <= barrier_ms` (panes are `(end - width, end]`), so a pane whose
282+
/// series stopped sending closes without a later sample. The barrier is
283+
/// capped at `now_ms`; a repeated or lower barrier does nothing. Samples
284+
/// at or before it that arrive later count as late. Returns the barrier
285+
/// in force.
286+
pub fn advance_watermark_at(&mut self, barrier_ms: i64, now_ms: i64) -> i64 {
287+
self.newest_clock_ms = self.newest_clock_ms.max(Some(now_ms));
288+
let barrier_ms = barrier_ms.min(now_ms);
289+
if self.barrier_ms.is_some_and(|current| current >= barrier_ms) {
290+
return self.barrier_ms.unwrap_or(barrier_ms);
291+
}
292+
self.barrier_ms = Some(barrier_ms);
293+
self.close_panes();
294+
barrier_ms
267295
}
268296

269297
/// [`Self::ingest_at`] the current wall-clock time.
270298
pub fn ingest(&mut self, samples: &[RawSample]) {
271-
let now_ms = std::time::SystemTime::now()
272-
.duration_since(std::time::UNIX_EPOCH)
273-
.map_or(0, |d| i64::try_from(d.as_millis()).unwrap_or(i64::MAX));
274-
self.ingest_at(samples, now_ms);
299+
self.ingest_at(samples, wall_clock_ms());
275300
}
276301

277302
/// Buffers `samples` into their panes, then builds every pane the
@@ -395,6 +420,13 @@ impl Precompute {
395420
}
396421
}
397422

423+
/// Milliseconds since the Unix epoch.
424+
fn wall_clock_ms() -> i64 {
425+
std::time::SystemTime::now()
426+
.duration_since(std::time::UNIX_EPOCH)
427+
.map_or(0, |d| i64::try_from(d.as_millis()).unwrap_or(i64::MAX))
428+
}
429+
398430
/// End of the tumbling pane of width `pane_ms` that contains `timestamp_ms`,
399431
/// or `None` when it does not fit in an `i64`. Panes are
400432
/// `(end - pane_ms, end]`, like PromQL ranges: a sample at a pane boundary

‎data_plane/tests/answers.rs‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -356,3 +356,93 @@ async fn raw_range_reads_forward_auth_and_tenancy_headers() {
356356
.unwrap()
357357
.contains("fetching raw samples"));
358358
}
359+
360+
/// After a producer's event-time barrier, the last pane of a series that
361+
/// stopped sending is closed, and the planned query is served from the
362+
/// plan instead of forwarded. Input stays open.
363+
#[tokio::test]
364+
async fn barrier_serves_the_window_of_a_stopped_series() {
365+
let state = Arc::new(
366+
DataPlane::new(
367+
common::plan_json("example4b", false),
368+
fake_prometheus(vec![]).await,
369+
15_000,
370+
)
371+
.unwrap(),
372+
);
373+
let app = router(state.clone());
374+
let write = |from: i64, to: i64| {
375+
let request = WriteRequest {
376+
timeseries: vec![TimeSeries {
377+
labels: vec![
378+
Label {
379+
name: "__name__".into(),
380+
value: "latency_ms".into(),
381+
},
382+
Label {
383+
name: "job".into(),
384+
value: "a".into(),
385+
},
386+
],
387+
samples: (from..to)
388+
.step_by(15_000)
389+
.map(|t| Sample {
390+
timestamp: t,
391+
value: (t / 15_000 % 240) as f64,
392+
})
393+
.collect(),
394+
}],
395+
};
396+
Request::post("/api/v1/write")
397+
.body(Body::from(encode(&request)))
398+
.unwrap()
399+
};
400+
let written = app
401+
.clone()
402+
.oneshot(write(MINUTE, 70 * MINUTE))
403+
.await
404+
.unwrap();
405+
assert_eq!(written.status(), StatusCode::NO_CONTENT);
406+
let query = "quantile_over_time(0.99, latency_ms[60m])";
407+
// The series stopped at 69:45; with 15 s lateness (60, 70 min] is open.
408+
let response = instant(&app, query, 70 * MINUTE).await;
409+
assert_eq!(response.execution.as_deref(), Some("forwarded"));
410+
411+
let barrier = app
412+
.clone()
413+
.oneshot(
414+
Request::post(format!(
415+
"/api/v1/precompute/watermark?time_ms={}",
416+
70 * MINUTE
417+
))
418+
.body(Body::empty())
419+
.unwrap(),
420+
)
421+
.await
422+
.unwrap();
423+
assert_eq!(barrier.status(), StatusCode::OK);
424+
let response = instant(&app, query, 70 * MINUTE).await;
425+
assert_eq!(
426+
response.execution.as_deref(),
427+
Some("plan"),
428+
"{:?}",
429+
response.body
430+
);
431+
432+
let written = app
433+
.clone()
434+
.oneshot(write(70 * MINUTE + 15_000, 72 * MINUTE))
435+
.await
436+
.unwrap();
437+
assert_eq!(written.status(), StatusCode::NO_CONTENT, "input stays open");
438+
let bad = app
439+
.clone()
440+
.oneshot(
441+
Request::post("/api/v1/precompute/watermark")
442+
.body(Body::empty())
443+
.unwrap(),
444+
)
445+
.await
446+
.unwrap();
447+
assert_eq!(bad.status(), StatusCode::BAD_REQUEST);
448+
}

‎data_plane/tests/precompute.rs‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -501,3 +501,102 @@ fn query_run_metrics_are_bounded_like_routed_samples() {
501501
assert_eq!(engine.stats.samples_query_run_only, 1);
502502
assert_eq!(engine.watermark_ms(), Some(now));
503503
}
504+
505+
/// An event-time barrier closes the last pane of a series that stopped
506+
/// sending, without a later sample.
507+
#[test]
508+
fn barrier_closes_the_last_pane_of_a_stopped_series() {
509+
let (mut engine, sds) = precompute(common::plan("example4b", false));
510+
let id = sds.definitions.keys().next().unwrap().clone();
511+
engine.ingest_at(
512+
&samples("latency_ms", &["a"], MINUTE, 25 * MINUTE),
513+
31 * MINUTE,
514+
);
515+
assert!(engine
516+
.store
517+
.get(&id, Selection::ending_at(2 * PANE))
518+
.is_some());
519+
assert!(
520+
engine
521+
.store
522+
.get(&id, Selection::ending_at(3 * PANE))
523+
.is_none(),
524+
"(20, 30 min] open"
525+
);
526+
assert_eq!(engine.advance_watermark_at(3 * PANE, 31 * MINUTE), 3 * PANE);
527+
assert_eq!(
528+
engine
529+
.store
530+
.get(&id, Selection::ending_at(3 * PANE))
531+
.unwrap()
532+
.num_rows(),
533+
1
534+
);
535+
}
536+
537+
/// A repeated or lower barrier does nothing, and a barrier ahead of the
538+
/// wall clock is capped at it.
539+
#[test]
540+
fn barrier_is_monotonic_and_capped_at_the_wall_clock() {
541+
let (mut engine, sds) = precompute(common::plan("example4b", false));
542+
let id = sds.definitions.keys().next().unwrap().clone();
543+
engine.ingest_at(
544+
&samples("latency_ms", &["a"], MINUTE, 25 * MINUTE),
545+
31 * MINUTE,
546+
);
547+
assert_eq!(
548+
engine.advance_watermark_at(4 * PANE, 31 * MINUTE),
549+
31 * MINUTE
550+
);
551+
assert!(engine
552+
.store
553+
.get(&id, Selection::ending_at(3 * PANE))
554+
.is_some());
555+
assert!(
556+
engine
557+
.store
558+
.get(&id, Selection::ending_at(4 * PANE))
559+
.is_none(),
560+
"capped at 31 min"
561+
);
562+
let stats = engine.stats;
563+
assert_eq!(
564+
engine.advance_watermark_at(31 * MINUTE, 31 * MINUTE),
565+
31 * MINUTE
566+
);
567+
assert_eq!(
568+
engine.advance_watermark_at(2 * PANE, 31 * MINUTE),
569+
31 * MINUTE
570+
);
571+
assert_eq!(engine.stats, stats);
572+
}
573+
574+
/// A barrier at `i64::MAX` terminates and builds only retained panes.
575+
#[test]
576+
fn barrier_at_i64_max_terminates() {
577+
let (mut engine, _) = precompute(common::plan("example4b", false));
578+
engine.ingest_at(
579+
&samples("latency_ms", &["a"], MINUTE, 25 * MINUTE),
580+
i64::MAX,
581+
);
582+
let built = engine.stats.panes_built;
583+
assert_eq!(engine.advance_watermark_at(i64::MAX, i64::MAX), i64::MAX);
584+
assert!(engine.stats.panes_built - built <= 7);
585+
}
586+
587+
/// Input stays open after a barrier: a later sample is accepted, and one at
588+
/// or before the barrier counts as late.
589+
#[test]
590+
fn input_stays_open_after_a_barrier() {
591+
let (mut engine, _) = precompute(common::plan("example4b", false));
592+
engine.ingest_at(
593+
&samples("latency_ms", &["a"], MINUTE, 25 * MINUTE),
594+
31 * MINUTE,
595+
);
596+
engine.advance_watermark_at(3 * PANE, 31 * MINUTE);
597+
let accepted = engine.stats.samples_accepted;
598+
engine.ingest_at(&[sample("latency_ms", "a", 35 * MINUTE, 1.0)], 36 * MINUTE);
599+
assert_eq!(engine.stats.samples_accepted, accepted + 1);
600+
engine.ingest_at(&[sample("latency_ms", "a", 29 * MINUTE, 1.0)], 36 * MINUTE);
601+
assert_eq!(engine.stats.samples_dropped_late, 1);
602+
}

0 commit comments

Comments
 (0)