From 335267ce66e64ce995cf459b89ca5f9686019b5a Mon Sep 17 00:00:00 2001 From: Oleksii Sholik Date: Tue, 21 Apr 2026 13:05:36 +0200 Subject: [PATCH 1/3] chore(agent-tasks): add sv1465 task notes --- .../progress.md | 23 ++++++++++++++++ .../task.md | 26 +++++++++++++++++++ 2 files changed, 49 insertions(+) create mode 100644 .agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/progress.md create mode 100644 .agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/task.md diff --git a/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/progress.md b/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/progress.md new file mode 100644 index 0000000000..5ad8bb68f9 --- /dev/null +++ b/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/progress.md @@ -0,0 +1,23 @@ +# Progress log + +## Session: 2026-04-21--sv1465--populate-telemetry-columns + +### 2026-04-21 + +- Cloned worktree `sv1465-telemetry-columns` off electric main (`b32d2128e`). +- Audited the four columns called out in stratovolt#1465: + - `num_bytes`: missing span attribute in `ServeShapePlug.end_telemetry_span/2`. Need to add. + - `chunk_size`: already emitted as span attribute in `Api.Response.send_stream/2` (child span `shape_get.plug.stream_chunk`). No change needed. + - `shape_snapshot.query.duration_us`: landed in electric#4110 as `shape_snapshot.query.duration_µs` (Unicode µ). Leaving as-is per the issue note (PR already merged). + - `electric.subqueries.subset_result.bytes`: `PartialModes.record_subset_metrics/4` only calls `:telemetry.execute`, never sets a span attribute. Need to add span attributes. +- Decided to mirror the span-attribute naming that Honeycomb already expects: dotted `electric.subqueries.subset_result.*` paths. + +### Implementation + +- Added `num_bytes` attribute to the ServeShapePlug root span alongside the existing `bytes` telemetry measurement. +- Added span attributes `electric.subqueries.subset_result.bytes`, `.rows`, `.duration_µs` on the subset materialisation span. +- `chunk_size` left untouched; already wired. + +### Operational issues + +- `gclone` is a fish function; had to invoke it via `fish -c` from bash sessions (noted in instructions). diff --git a/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/task.md b/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/task.md new file mode 100644 index 0000000000..b58f205582 --- /dev/null +++ b/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/task.md @@ -0,0 +1,26 @@ +# Populate unpopulated telemetry columns + +Upstream issue: electric-sql/stratovolt#1465 + +## Summary + +Several Honeycomb columns are defined but never populated: + +- `num_bytes` — should be emitted on `ServeShapePlug` streaming completion (alongside `http.response_size`). +- `chunk_size` — should be emitted per chunk as a histogram at the shape chunk buffer flush point. +- `shape_snapshot.query.duration_us` — already landed via electric#4110; no code change needed, only confirm wiring. +- `electric.subqueries.subset_result.bytes` — should be emitted after subset materialisation. + +## Investigation findings (worktree main @ b32d2128e) + +- `Electric.Shapes.Api.Response.send_stream/2` already wraps each chunk with a `shape_get.plug.stream_chunk` span and sets `chunk_size` as an attribute. The total `streaming_bytes_sent` assign is used for `http.response_size`. +- `Electric.Plug.ServeShapePlug.end_telemetry_span/2` already emits the `[:electric, :plug, :serve_shape]` telemetry event with a `bytes` measurement but never sets `num_bytes` as a span attribute — hence the empty Honeycomb column. +- `Electric.Shapes.PartialModes.record_subset_metrics/4` emits `[:electric, :subqueries, :subset_result]` telemetry with `bytes` but does not set any span attribute, so the Honeycomb column `electric.subqueries.subset_result.bytes` remains empty. +- `shape_snapshot.query.duration_µs` is set as a span attribute via `OpenTelemetry.stop_and_save_intervals` in `SnapshotQuery.execute_for_shape/4` (PR #4110). The issue uses `_us` but the column in Honeycomb is the literal attribute name (Unicode `µ`). Not touching. + +## Plan + +1. In `ServeShapePlug.end_telemetry_span`, add `num_bytes` span attribute mirroring the existing `bytes` measurement. +2. In `PartialModes.record_subset_metrics`, add span attributes `electric.subqueries.subset_result.bytes`, `electric.subqueries.subset_result.rows`, `electric.subqueries.subset_result.duration_µs` to the current span. +3. Confirm `chunk_size` already set on `shape_get.plug.stream_chunk` child span; no change needed. +4. Add a changeset and a lightweight test if feasible. From 4e75fdc5e4d55ac9be7f80aaf2cc54f3ef968916 Mon Sep 17 00:00:00 2001 From: Oleksii Sholik Date: Tue, 21 Apr 2026 13:05:43 +0200 Subject: [PATCH 2/3] feat(sync-service): populate num_bytes and subset_result span attributes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expose previously-empty telemetry span attributes so the corresponding Honeycomb columns can be queried: - `num_bytes` on the ServeShapePlug root span, mirroring the existing `bytes` telemetry measurement emitted alongside `http.response_size`. - `electric.subqueries.subset_result.{bytes,rows,duration_µs}` on the subset materialisation path in `PartialModes.record_subset_metrics/4`. `chunk_size` is already emitted as a span attribute on the `shape_get.plug.stream_chunk` child span; no change required. `shape_snapshot.query.duration_µs` landed in #4110. Refs electric-sql/stratovolt#1465 --- .changeset/populate-telemetry-columns.md | 5 +++++ .../lib/electric/plug/serve_shape_plug.ex | 10 +++++++++- .../lib/electric/shapes/partial_modes.ex | 13 ++++++++++++- 3 files changed, 26 insertions(+), 2 deletions(-) create mode 100644 .changeset/populate-telemetry-columns.md diff --git a/.changeset/populate-telemetry-columns.md b/.changeset/populate-telemetry-columns.md new file mode 100644 index 0000000000..05891bab77 --- /dev/null +++ b/.changeset/populate-telemetry-columns.md @@ -0,0 +1,5 @@ +--- +'@core/sync-service': patch +--- + +Populate previously-empty telemetry span attributes for shape requests: `num_bytes` on the root shape-get span and `electric.subqueries.subset_result.{bytes,rows,duration_µs}` on subset materialisation spans. This makes the corresponding Honeycomb columns queryable. diff --git a/packages/sync-service/lib/electric/plug/serve_shape_plug.ex b/packages/sync-service/lib/electric/plug/serve_shape_plug.ex index ba4311bd2f..c57afea19a 100644 --- a/packages/sync-service/lib/electric/plug/serve_shape_plug.ex +++ b/packages/sync-service/lib/electric/plug/serve_shape_plug.ex @@ -291,11 +291,13 @@ defmodule Electric.Plug.ServeShapePlug do # is the place to assign them because we keep this plug last in the "plug pipeline" defined # in this module. defp end_telemetry_span(%Conn{assigns: assigns} = conn, _ \\ nil) do + bytes_sent = assigns[:streaming_bytes_sent] || 0 + OpenTelemetry.execute( [:electric, :plug, :serve_shape], %{ count: 1, - bytes: assigns[:streaming_bytes_sent] || 0, + bytes: bytes_sent, monotonic_time: System.monotonic_time(), duration: System.monotonic_time() - conn.private[:electric_telemetry_span][:start_time] }, @@ -308,6 +310,12 @@ defmodule Electric.Plug.ServeShapePlug do } ) + # Expose the total response body size as a dedicated span attribute + # alongside the semantic `http.response_size` set by + # `Electric.Plug.Utils.common_open_telemetry_attrs/1`. This populates the + # `num_bytes` Honeycomb column for shape-get requests. + OpenTelemetry.add_span_attributes(%{num_bytes: bytes_sent}) + add_span_attrs_from_conn(conn) OpentelemetryTelemetry.end_telemetry_span(OpenTelemetry, %{}) conn diff --git a/packages/sync-service/lib/electric/shapes/partial_modes.ex b/packages/sync-service/lib/electric/shapes/partial_modes.ex index 53482d7d41..962b2fcd6f 100644 --- a/packages/sync-service/lib/electric/shapes/partial_modes.ex +++ b/packages/sync-service/lib/electric/shapes/partial_modes.ex @@ -63,10 +63,12 @@ defmodule Electric.Shapes.PartialModes do {[row], {start_time, bytes + IO.iodata_length(row), rows + 1}} end, fn {start_time, bytes, rows} -> + duration = System.monotonic_time(:microsecond) - start_time + OpenTelemetry.execute( [:electric, :subqueries, :subset_result], %{ - duration: System.monotonic_time(:microsecond) - start_time, + duration: duration, bytes: bytes, count: 1, rows: rows @@ -77,6 +79,15 @@ defmodule Electric.Shapes.PartialModes do "shape.root_table": shape.root_table } ) + + # Expose subset materialisation metrics as span attributes so they + # become queryable Honeycomb columns on the enclosing shape request + # span. Mirrors the telemetry event measurements above. + OpenTelemetry.add_span_attributes(%{ + "electric.subqueries.subset_result.bytes" => bytes, + "electric.subqueries.subset_result.rows" => rows, + "electric.subqueries.subset_result.duration_µs" => duration + }) end ) end From 757cf4b11f0fdc6bc876c1ef0c4362f6ba3981f6 Mon Sep 17 00:00:00 2001 From: Oleksii Sholik Date: Wed, 22 Apr 2026 01:58:41 +0200 Subject: [PATCH 3/3] fix comment and stop tracking .agent-tasks/ - Clarify that the span attributes added in PartialModes.record_subset_metrics/4 land on the active shape_snapshot.query_fn child span (not the enclosing shape request span), addressing review feedback on PR #4143. - Stop tracking .agent-tasks/ in Git and add it to .gitignore. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../progress.md | 23 ---------------- .../task.md | 26 ------------------- .gitignore | 3 ++- .../lib/electric/shapes/partial_modes.ex | 6 ++--- 4 files changed, 5 insertions(+), 53 deletions(-) delete mode 100644 .agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/progress.md delete mode 100644 .agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/task.md diff --git a/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/progress.md b/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/progress.md deleted file mode 100644 index 5ad8bb68f9..0000000000 --- a/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/progress.md +++ /dev/null @@ -1,23 +0,0 @@ -# Progress log - -## Session: 2026-04-21--sv1465--populate-telemetry-columns - -### 2026-04-21 - -- Cloned worktree `sv1465-telemetry-columns` off electric main (`b32d2128e`). -- Audited the four columns called out in stratovolt#1465: - - `num_bytes`: missing span attribute in `ServeShapePlug.end_telemetry_span/2`. Need to add. - - `chunk_size`: already emitted as span attribute in `Api.Response.send_stream/2` (child span `shape_get.plug.stream_chunk`). No change needed. - - `shape_snapshot.query.duration_us`: landed in electric#4110 as `shape_snapshot.query.duration_µs` (Unicode µ). Leaving as-is per the issue note (PR already merged). - - `electric.subqueries.subset_result.bytes`: `PartialModes.record_subset_metrics/4` only calls `:telemetry.execute`, never sets a span attribute. Need to add span attributes. -- Decided to mirror the span-attribute naming that Honeycomb already expects: dotted `electric.subqueries.subset_result.*` paths. - -### Implementation - -- Added `num_bytes` attribute to the ServeShapePlug root span alongside the existing `bytes` telemetry measurement. -- Added span attributes `electric.subqueries.subset_result.bytes`, `.rows`, `.duration_µs` on the subset materialisation span. -- `chunk_size` left untouched; already wired. - -### Operational issues - -- `gclone` is a fish function; had to invoke it via `fish -c` from bash sessions (noted in instructions). diff --git a/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/task.md b/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/task.md deleted file mode 100644 index b58f205582..0000000000 --- a/.agent-tasks/2026-04-21--sv1465--populate-telemetry-columns/task.md +++ /dev/null @@ -1,26 +0,0 @@ -# Populate unpopulated telemetry columns - -Upstream issue: electric-sql/stratovolt#1465 - -## Summary - -Several Honeycomb columns are defined but never populated: - -- `num_bytes` — should be emitted on `ServeShapePlug` streaming completion (alongside `http.response_size`). -- `chunk_size` — should be emitted per chunk as a histogram at the shape chunk buffer flush point. -- `shape_snapshot.query.duration_us` — already landed via electric#4110; no code change needed, only confirm wiring. -- `electric.subqueries.subset_result.bytes` — should be emitted after subset materialisation. - -## Investigation findings (worktree main @ b32d2128e) - -- `Electric.Shapes.Api.Response.send_stream/2` already wraps each chunk with a `shape_get.plug.stream_chunk` span and sets `chunk_size` as an attribute. The total `streaming_bytes_sent` assign is used for `http.response_size`. -- `Electric.Plug.ServeShapePlug.end_telemetry_span/2` already emits the `[:electric, :plug, :serve_shape]` telemetry event with a `bytes` measurement but never sets `num_bytes` as a span attribute — hence the empty Honeycomb column. -- `Electric.Shapes.PartialModes.record_subset_metrics/4` emits `[:electric, :subqueries, :subset_result]` telemetry with `bytes` but does not set any span attribute, so the Honeycomb column `electric.subqueries.subset_result.bytes` remains empty. -- `shape_snapshot.query.duration_µs` is set as a span attribute via `OpenTelemetry.stop_and_save_intervals` in `SnapshotQuery.execute_for_shape/4` (PR #4110). The issue uses `_us` but the column in Honeycomb is the literal attribute name (Unicode `µ`). Not touching. - -## Plan - -1. In `ServeShapePlug.end_telemetry_span`, add `num_bytes` span attribute mirroring the existing `bytes` measurement. -2. In `PartialModes.record_subset_metrics`, add span attributes `electric.subqueries.subset_result.bytes`, `electric.subqueries.subset_result.rows`, `electric.subqueries.subset_result.duration_µs` to the current span. -3. Confirm `chunk_size` already set on `shape_get.plug.stream_chunk` child span; no change needed. -4. Add a changeset and a lightweight test if feasible. diff --git a/.gitignore b/.gitignore index 9d4bcc8d04..a89b4d42bc 100644 --- a/.gitignore +++ b/.gitignore @@ -30,4 +30,5 @@ response.tmp .claude !website/.claude/commands !website/.claude/skills -_artifacts \ No newline at end of file +_artifacts +.agent-tasks \ No newline at end of file diff --git a/packages/sync-service/lib/electric/shapes/partial_modes.ex b/packages/sync-service/lib/electric/shapes/partial_modes.ex index 962b2fcd6f..a188cbf57c 100644 --- a/packages/sync-service/lib/electric/shapes/partial_modes.ex +++ b/packages/sync-service/lib/electric/shapes/partial_modes.ex @@ -80,9 +80,9 @@ defmodule Electric.Shapes.PartialModes do } ) - # Expose subset materialisation metrics as span attributes so they - # become queryable Honeycomb columns on the enclosing shape request - # span. Mirrors the telemetry event measurements above. + # Expose subset materialisation metrics as span attributes on the + # active shape_snapshot.query_fn child span so they become queryable + # Honeycomb columns. Mirrors the telemetry event measurements above. OpenTelemetry.add_span_attributes(%{ "electric.subqueries.subset_result.bytes" => bytes, "electric.subqueries.subset_result.rows" => rows,