From 4f1dcafaf715c1694e50e08658a6ef935f2a1df9 Mon Sep 17 00:00:00 2001 From: "tom.bonfert" Date: Fri, 23 Oct 2026 17:32:50 +0200 Subject: [PATCH 1/4] docs(tsal): add UDF rules for empty inputs, gaps, synchronization, and return types Document the rules users must follow when writing TSAL UDFs that accept `SampleSeries`: handle empty planning-time inputs, return a consistent type on every branch, respect gaps via `continuous_interval_indices`, and synchronize series before element-wise numpy operations. Include a `magnitude` example using `synchronized()`. Update both the defining expressions reference and the impulse-tsal skill. --- .../query_engine/tsal/defining_expressions.md | 35 +++++++++++++++++++ skills/impulse-tsal/SKILL.md | 28 +++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/docs/impulse/docs/references/query_engine/tsal/defining_expressions.md b/docs/impulse/docs/references/query_engine/tsal/defining_expressions.md index f57b9756..fd596df8 100644 --- a/docs/impulse/docs/references/query_engine/tsal/defining_expressions.md +++ b/docs/impulse/docs/references/query_engine/tsal/defining_expressions.md @@ -310,6 +310,41 @@ still return a valid object of the declared type. For example, an empty `SampleS UDF should return a valid, empty `SampleSeries`. ::: +#### Rules for correct results + +When a UDF receives one or more `SampleSeries`, follow these rules: + +- **Handle empty inputs.** On the planning call above, every series argument is empty. Indexing + (`series.values[0]`, `series.tstarts[-1]`), `np.max(series.values)` or + `SampleSeries.from_timestamps(...)` raise on empty input. Guard with + `if len(series) == 0: return SampleSeries.empty()`; the built-in reducers (`sum()`, `min()`, + `max()`, `mean()`) return NaN for an empty series. +- **Return the same type on every path.** The type returned on the planning call determines the + Spark column type of the result. A UDF whose return type depends on the data (for example, a + `SampleSeries` on one branch and a float on another) produces values that do not fit that column. +- **Respect gaps.** A series may contain gaps. Two consecutive samples are contiguous only if + `tstarts[i + 1] == tends[i]` holds exactly. Compute neighbour-based logic (differences, gradients, + cumulative sums) per contiguous segment using `series.continuous_interval_indices`, a list of + inclusive `(start, end)` index pairs. The built-in `diff()`, `cumtrapz()` and `resample()` work this + way. +- **Synchronize before combining the raw arrays of several series.** Different channels have + different timestamps, so element-wise numpy operations on `a.values` and `b.values` pair unrelated + samples (or raise on a shape mismatch). TSAL operators (`a + b`, `a > b`) synchronize + automatically. Inside a UDF, call `a.synchronized(b)` or `a.synchronized_all([b, c])`: they return + the series (the calling series first) on shared intervals covering only their overlap, so the + result may be shorter than the inputs, or empty. + +```python +import numpy as np + +@TimeSeriesExpression.udf +def magnitude(x, y): + xs, ys = x.synchronized(y) + return SampleSeries(xs.tstarts, xs.tends, np.hypot(xs.values, ys.values)) + +speed = magnitude(vel_x, vel_y) +``` + --- ## Virtual signals diff --git a/skills/impulse-tsal/SKILL.md b/skills/impulse-tsal/SKILL.md index 69888d25..d1ff16f5 100644 --- a/skills/impulse-tsal/SKILL.md +++ b/skills/impulse-tsal/SKILL.md @@ -161,6 +161,34 @@ derated = derate(power) - Values are constant per container. The declaration **propagates** when the UDF is wrapped in an aggregation, event, or calculated channel — the wrapper does not redeclare anything. +**Rules for correct UDF results** (when an input is a `SampleSeries`): + +- **Handle empty inputs.** Every UDF runs once at planning time, with every series input empty (and + every declared container tag/metric `None`), so the engine can infer the result type. Indexing + (`values[0]`, `tstarts[-1]`), `np.max(values)` or `SampleSeries.from_timestamps(...)` fail on empty + input; guard with `if len(series) == 0: return SampleSeries.empty()`. Built-in reducers return NaN. +- **Return the same type on every path.** The type returned on the planning call fixes the output + column type; a return type that depends on the data (a `SampleSeries` on one branch, a float on + another) produces values that do not fit that column. +- **Respect gaps.** A series may contain gaps; two samples are contiguous only if + `tstarts[i+1] == tends[i]` exactly. Compute neighbour-based logic (differences, gradients, + cumulative sums) per segment using `series.continuous_interval_indices` (inclusive `(start, end)` + index pairs), as `diff()`, `cumtrapz()` and `resample()` do. +- **Synchronize before combining raw arrays of several series.** Channels have different timestamps, + so element-wise numpy on `a.values` and `b.values` pairs unrelated samples (or raises on a shape + mismatch). Operators (`a + b`, `a > b`) synchronize automatically; inside a UDF call + `a.synchronized(b)` or `a.synchronized_all([b, c])`, which return the series (self first) on shared + intervals covering only their overlap, so the result may be shorter or empty. + +```python +import numpy as np + +@TimeSeriesExpression.udf +def magnitude(x, y): + xs, ys = x.synchronized(y) + return SampleSeries(xs.tstarts, xs.tends, np.hypot(xs.values, ys.values)) +``` + ## What an expression evaluates to (the core data model) Every expression resolves — per container — to one of four in-memory types. Knowing the type tells you From 3ab9720683c8855bc2b8cf9e2c651b5039074600 Mon Sep 17 00:00:00 2001 From: "tom.bonfert" Date: Fri, 23 Oct 2026 17:44:03 +0200 Subject: [PATCH 2/4] docs(impulse, tsal): recommend native TSAL building blocks before UDFs Add a "Prefer native functionality" section to the impulse and impulse-tsal skills advising users to exhaust native operators, signal methods, Intervals methods, event types, and aggregation classes before falling back to UDFs. Highlight that native operations handle empty inputs, gaps, and unaligned channels and remain optimizable by the engine. --- skills/impulse-tsal/SKILL.md | 8 ++++++++ skills/impulse/SKILL.md | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/skills/impulse-tsal/SKILL.md b/skills/impulse-tsal/SKILL.md index d1ff16f5..9d5a1e18 100644 --- a/skills/impulse-tsal/SKILL.md +++ b/skills/impulse-tsal/SKILL.md @@ -129,6 +129,14 @@ three `SampleSeries` (min, max, average). **User-defined functions** +**Prefer native functionality.** Before writing a UDF, check whether the request can be expressed +with native building blocks: the operators and signal methods above, the `Intervals` methods below, +the event types (`impulse-events`) and the aggregation classes (`impulse-aggregations`). Write a UDF +only for the part native functionality cannot express, and keep everything else native. Native +operations already handle empty inputs, gaps and unaligned channels (see the rules below). Future +Impulse versions may also optimize the execution of native functionality, while logic inside a UDF +stays opaque to the engine. + ```python from impulse_query_engine.model.series import SampleSeries from impulse_query_engine.analyze.metadata.time_series_expression import TimeSeriesExpression diff --git a/skills/impulse/SKILL.md b/skills/impulse/SKILL.md index 0b5064b6..d1f03f4e 100644 --- a/skills/impulse/SKILL.md +++ b/skills/impulse/SKILL.md @@ -132,6 +132,14 @@ report.add_page(page) report.run() # compute and write the gold star schema ``` +## Prefer native functionality + +Build solutions from Impulse's native building blocks (TSAL operators and methods, event types, +aggregation classes, calculated channels) and fall back to a user-defined function only for logic +they cannot express. Native operations handle edge cases such as empty inputs, gaps and unaligned +channels, and future Impulse versions may also optimize their execution, while logic inside a UDF +stays opaque to the engine. See **`impulse-tsal`** → User-defined functions. + ## Where to go next - Building signal expressions → **`impulse-tsal`** From 925b5e1209171ba988a67809ef9cb8d01d8adba3 Mon Sep 17 00:00:00 2001 From: "tom.bonfert" Date: Fri, 23 Oct 2026 17:55:32 +0200 Subject: [PATCH 3/4] docs(tsal): clarify empty-input handling for UDFs returning scalars Update the TSAL UDF empty-input guidance in both the defining expressions reference and the impulse-tsal skill to say that scalar-returning UDFs should return `np.nan` (not `SampleSeries.empty()`), and that built-in reducers already return NaN for empty series. --- .../references/query_engine/tsal/defining_expressions.md | 8 ++++---- skills/impulse-tsal/SKILL.md | 5 +++-- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/impulse/docs/references/query_engine/tsal/defining_expressions.md b/docs/impulse/docs/references/query_engine/tsal/defining_expressions.md index fd596df8..dda8ed93 100644 --- a/docs/impulse/docs/references/query_engine/tsal/defining_expressions.md +++ b/docs/impulse/docs/references/query_engine/tsal/defining_expressions.md @@ -315,10 +315,10 @@ UDF should return a valid, empty `SampleSeries`. When a UDF receives one or more `SampleSeries`, follow these rules: - **Handle empty inputs.** On the planning call above, every series argument is empty. Indexing - (`series.values[0]`, `series.tstarts[-1]`), `np.max(series.values)` or - `SampleSeries.from_timestamps(...)` raise on empty input. Guard with - `if len(series) == 0: return SampleSeries.empty()`; the built-in reducers (`sum()`, `min()`, - `max()`, `mean()`) return NaN for an empty series. + (`series.values[0]`, `series.tstarts[-1]`) raises on empty input. Guard with + `if len(series) == 0:` and return an empty result of the UDF's return type: `SampleSeries.empty()` + for a series, `np.nan` for a number. The built-in reducers (`sum()`, `min()`, `max()`, `mean()`) + already return NaN for an empty series. - **Return the same type on every path.** The type returned on the planning call determines the Spark column type of the result. A UDF whose return type depends on the data (for example, a `SampleSeries` on one branch and a float on another) produces values that do not fit that column. diff --git a/skills/impulse-tsal/SKILL.md b/skills/impulse-tsal/SKILL.md index 9d5a1e18..4576da50 100644 --- a/skills/impulse-tsal/SKILL.md +++ b/skills/impulse-tsal/SKILL.md @@ -173,8 +173,9 @@ derated = derate(power) - **Handle empty inputs.** Every UDF runs once at planning time, with every series input empty (and every declared container tag/metric `None`), so the engine can infer the result type. Indexing - (`values[0]`, `tstarts[-1]`), `np.max(values)` or `SampleSeries.from_timestamps(...)` fail on empty - input; guard with `if len(series) == 0: return SampleSeries.empty()`. Built-in reducers return NaN. + (`series.values[0]`, `series.tstarts[-1]`) fails on empty input. Guard with `if len(series) == 0:` + and return an empty result of the UDF's return type: `SampleSeries.empty()` for a series, `np.nan` + for a number. The built-in reducers (`sum()`, `min()`, `max()`, `mean()`) already return NaN. - **Return the same type on every path.** The type returned on the planning call fixes the output column type; a return type that depends on the data (a `SampleSeries` on one branch, a float on another) produces values that do not fit that column. From 7f25044011a47a85d0d3c8ad7285a8138791804f Mon Sep 17 00:00:00 2001 From: "tom.bonfert" Date: Fri, 23 Oct 2026 17:59:51 +0200 Subject: [PATCH 4/4] docs(impulse-tsal): mention calculated channels as native building blocks Update the "Prefer native functionality" guidance in the impulse-tsal skill to include calculated channels (`impulse-channels`) alongside existing native options. --- skills/impulse-tsal/SKILL.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/skills/impulse-tsal/SKILL.md b/skills/impulse-tsal/SKILL.md index 4576da50..5e469f56 100644 --- a/skills/impulse-tsal/SKILL.md +++ b/skills/impulse-tsal/SKILL.md @@ -131,11 +131,11 @@ three `SampleSeries` (min, max, average). **Prefer native functionality.** Before writing a UDF, check whether the request can be expressed with native building blocks: the operators and signal methods above, the `Intervals` methods below, -the event types (`impulse-events`) and the aggregation classes (`impulse-aggregations`). Write a UDF -only for the part native functionality cannot express, and keep everything else native. Native -operations already handle empty inputs, gaps and unaligned channels (see the rules below). Future -Impulse versions may also optimize the execution of native functionality, while logic inside a UDF -stays opaque to the engine. +the event types (`impulse-events`), the aggregation classes (`impulse-aggregations`) and calculated +channels (`impulse-channels`). Write a UDF only for the part native functionality cannot express, +and keep everything else native. Native operations already handle empty inputs, gaps and unaligned +channels (see the rules below). Future Impulse versions may also optimize the execution of native +functionality, while logic inside a UDF stays opaque to the engine. ```python from impulse_query_engine.model.series import SampleSeries