diff --git a/website/blog/posts/2026-04-22-subqueries.md b/website/blog/posts/2026-04-22-subqueries.md new file mode 100644 index 0000000000..121a4ab36a --- /dev/null +++ b/website/blog/posts/2026-04-22-subqueries.md @@ -0,0 +1,176 @@ +--- +title: Subqueries — making sync work in practice +description: >- + Subqueries let Electric shapes express relational filtering in SQL. + Electric 1.6 keeps complex AND/OR/NOT expressions incremental too, so + large shapes stay fast. +excerpt: >- + Sync only works in real apps if it can follow relationships. + Subqueries let Electric sync the right rows for each user using SQL, + and Electric 1.6 keeps complex expressions incremental too. +authors: [rob] +image: /img/blog/subqueries/header.jpg +tags: [shapes, postgres-sync, release] +outline: [2, 3] +post: true +published: true +--- + +Sync is what makes apps feel instant. The data is already there when a screen renders. Another user changes something and your UI stays current. You can refresh, reconnect, switch devices, and keep going. + +That is the broad pitch. We have written more about how [sync replaces data fetching](/blog/2025/04/22/untangling-llm-spaghetti) and why it is the right foundation for [collaborative, real-time apps](/blog/2025/04/09/building-ai-apps-on-sync). + +But there is a more practical question underneath all of it: + +Which rows should this client actually receive? + +In simple demos, a column filter is enough. In real systems, the rule usually lives in other tables. A document is visible because you own it, or because it was shared with you, or because you belong to the workspace that contains it. Comments sync because their issue belongs to a project you can access. Invoice line items sync because their parent invoice does. + +This is where subqueries matter. + +## Sync the right rows + +Shapes are Electric's primitive for partial replication: a table and a `WHERE` clause. Define the subset once and Electric keeps that subset synced. + +For flat cases, the filter is simple: + +```sql +owner_id = $1 +``` + +And here is how that looks in TanStack DB: + +```ts +const documentsCollection = createCollection( + electricCollectionOptions({ + id: 'my-documents', + shapeOptions: { + url: `${ELECTRIC_URL}/v1/shape`, + params: { + table: 'documents', + where: 'owner_id = $1', + params: { '1': currentUserId }, + }, + }, + }) +) +``` + +Parameters (`$1`) are bound per client, so the same shape definition can serve different data to different users. + +But real apps do not stay flat for long. Access control, tenant membership, and parent-child data all pull in related tables. Subqueries let you express those rules directly in SQL. + +Sync documents for workspaces this user belongs to: + +```sql +workspace_id IN ( + SELECT workspace_id FROM workspace_members + WHERE user_id = $1 +) +``` + + +You can combine relational checks with ordinary predicates. For example, sync documents that I own, plus documents shared with me: + +```sql +owner_id = $1 +OR id IN ( + SELECT document_id FROM document_shares + WHERE shared_with = $1 +) +``` + +You can also traverse multiple hops. Sync comments for a project by walking from comments to issues to tasks: + +```sql +issue_id IN ( + SELECT id FROM issues WHERE task_id IN ( + SELECT id FROM tasks WHERE project_id = $1 + ) +) +``` + +This is mundane SQL. That is the point. + +The rule stays close to the data, where you already reason about memberships, shares, and relationships. Electric evaluates it server-side and keeps only the matching rows on each client. + +See the [WHERE clause docs](/docs/guides/shapes#where-clause) for the full reference on supported operators and subquery patterns. + +:::info +Subqueries are available on [Electric Cloud](/cloud) and are included in the [Pro, Scale, and Enterprise plans](/pricing). +::: + +## Why this matters + +The interesting part of sync is not a nicer `fetch()`. It is what you get once the right data is already local: live UIs, collaboration, resilient apps, instant navigation, fewer loading states. + +But none of that survives contact with production unless sync can follow your actual data model. The moment you have shared documents, org membership, private projects, child records, or exclusions, a simple column filter stops being enough. + +Subqueries are what make shapes fit real applications. They let you describe who can see a row, which child rows come along with a parent, how multiple access paths compose, and how exclusions or overrides work. + +## What changed in Electric 1.6 + +Subqueries are not new. We have supported them for a while, behind feature flags, and they have already been battle tested by customers in production. + +Electric 1.6 is the release that closes one of the last awkward cases. + +Subqueries already supported incremental sync for simple expressions. Complex expressions using `AND`, `OR`, and `NOT` also worked, but when the subquery result changed Electric could fall back to a full resync. On small shapes you might never notice. On large ones you would feel it as lag between a write and the UI catching up. + +With 1.6, those complex expressions stay incremental too. When memberships change, shares are granted, or related rows move in or out of scope, Electric now syncs only the affected rows. Large shapes keep the low-latency behavior that makes sync useful in the first place. + +That is why we now consider subqueries suitable for general use. + +This release also includes a client protocol update needed for the new incremental behavior. The feature flags are unchanged for now and we will remove them once we are confident clients have moved onto the newer protocol. + +## Using it now + +Here is the shared-documents example wired into a TanStack DB collection: + +```ts +import { electricCollectionOptions } from '@tanstack/electric-db-collection' +import { createCollection } from '@tanstack/react-db' + +const documentsCollection = createCollection( + electricCollectionOptions({ + id: 'my-documents', + shapeOptions: { + url: `${ELECTRIC_URL}/v1/shape`, + params: { + table: 'documents', + where: ` + owner_id = $1 + OR id IN ( + SELECT document_id FROM document_shares + WHERE shared_with = $1 + ) + `, + params: { '1': currentUserId }, + }, + }, + }) +) +``` + +Update to the latest packages: + +```sh +npm install @tanstack/db@latest @tanstack/electric-db-collection@latest +``` + +Subqueries remain behind the same feature flags as before: + +```sh +ELECTRIC_FEATURE_FLAGS=allow_subqueries,tagged_subqueries +``` + +:::warning +Incremental sync for complex subquery expressions in Electric 1.6 requires a client protocol update. Make sure all your clients are on `@tanstack/db >= 0.6.2` and `@tanstack/electric-db-collection >= 0.3.0` before upgrading the server. Those versions have been available since April 3, 2026. +::: + +If you were waiting for shapes to handle more realistic access-control logic without giving up the fast path, this is the point to try it. + +See the [WHERE clause docs](/docs/guides/shapes#where-clause) for the full reference. + +*** + +[Docs](/docs/guides/shapes#where-clause) · [Cloud](/cloud) · [Discord](https://discord.electric-sql.com) diff --git a/website/data/blog/authors.yaml b/website/data/blog/authors.yaml index 703cca5d39..a9c865fe2d 100644 --- a/website/data/blog/authors.yaml +++ b/website/data/blog/authors.yaml @@ -51,3 +51,9 @@ tdrz: title: Founding Engineer image: /img/team/tudor.jpg url: /about/team#tudor + +rob: + name: Rob A'Court + title: Founding Engineer + image: /img/team/rob.jpg + url: /about/team#rob diff --git a/website/docs/api/config.md b/website/docs/api/config.md index 3226912189..8569c72291 100644 --- a/website/docs/api/config.md +++ b/website/docs/api/config.md @@ -453,7 +453,7 @@ Consumer processes are partitioned across some number of supervisors to improve ## Feature Flags -Feature flags enable experimental or advanced features that are not yet enabled by default in production. +Feature flags enable advanced features and staged rollouts for capabilities that are not yet enabled by default in production. ### ELECTRIC_FEATURE_FLAGS @@ -465,10 +465,14 @@ Feature flags enable experimental or advanced features that are not yet enabled **Available flags:** - `allow_subqueries` - Enables subquery support in shape WHERE clauses -- `tagged_subqueries` - Enables improved multi-level dependency handling +- `tagged_subqueries` - Enables incremental subquery move handling, including compound boolean expressions with compatible clients +:::warning Client compatibility +Electric 1.6's incremental handling for compound subquery expressions changes the client protocol. Upgrade clients before enabling the server rollout. TanStack DB clients need `@tanstack/db >= 0.6.2` and `@tanstack/electric-db-collection >= 0.3.0`. +::: + ### allow_subqueries Enables support for subqueries in the WHERE clause of [shape](/docs/guides/shapes) definitions. When enabled, you can use queries in the form: @@ -479,15 +483,17 @@ WHERE id IN (SELECT user_id FROM memberships WHERE org_id = 'org_123') This allows creating shapes that filter based on related data in other tables, enabling more complex data synchronization patterns. -**Status:** Experimental. Disabled by default in production. +**Status:** General use. Disabled by default in production until enabled with `ELECTRIC_FEATURE_FLAGS`. ### tagged_subqueries -Subqueries create dependency trees between shapes. Without this flag, when data moves into or out of a dependent shape, the shape is invalidated (returning a 409). With this flag enabled, move operations are handled correctly without invalidation. +Subqueries create dependency trees between shapes. This flag enables incremental move handling when dependency rows change, including compound `WHERE` expressions that combine subqueries with `AND`, `OR`, and `NOT`. + +Before Electric 1.6, complex boolean combinations around subqueries could still invalidate the shape and return a `409` on a move. With this flag enabled and compatible clients, those changes are reconciled in-stream instead. See [discussion #2931](https://github.com/electric-sql/electric/discussions/2931) for more details about this feature. -**Status:** Experimental. Disabled by default in production. Requires `allow_subqueries` to be enabled. +**Status:** Rollout flag for subquery move handling. Disabled by default in production. Requires `allow_subqueries` to be enabled. ## Caching diff --git a/website/docs/guides/shapes.md b/website/docs/guides/shapes.md index e6b053e6b3..f9cb31ed1b 100644 --- a/website/docs/guides/shapes.md +++ b/website/docs/guides/shapes.md @@ -51,7 +51,7 @@ Shapes are defined by: A shape contains all of the rows in the table that match the where clause, if provided. If a columns clause is provided, the synced rows will only contain those selected columns. > [!Warning] Limitations -> Shapes are currently [single table](#single-table), though you can use [subqueries](#subqueries-experimental) to filter based on related data. Shape definitions are [immutable](#immutable). +> Shapes are currently [single table](#single-table), though you can use [subqueries](#subqueries) to filter based on related data. Shape definitions are [immutable](#immutable). > [!Warning] Security > Production apps should request shapes through your backend API for authorization and security. See the [auth guide](/docs/guides/auth). @@ -116,9 +116,12 @@ Where clauses have the following constraints: 1. can't use non-deterministic SQL functions like `count()` or `now()` -#### Subqueries (experimental) + +#### Subqueries -Electric supports subqueries in where clauses, allowing you to filter rows based on data in other tables. This enables cross-table filtering patterns—for example, syncing only users who belong to a specific organization: +Electric supports subqueries in where clauses, allowing you to filter rows based on data in other tables. This makes shapes suitable for general-use relational filtering, such as memberships, sharing rules, parent-child traversal, and exclusions. + +For example, you can sync only users who belong to a specific organization: ```ts import { electricCollectionOptions } from '@tanstack/electric-db-collection' @@ -139,6 +142,30 @@ const usersCollection = createCollection( ) ``` +Or combine subqueries with boolean logic to express more realistic access rules: + +```ts +const documentsCollection = createCollection( + electricCollectionOptions({ + id: 'visible-documents', + shapeOptions: { + url: 'http://localhost:3000/v1/shape', + params: { + table: 'documents', + where: ` + owner_id = $1 + OR id IN ( + SELECT document_id FROM document_shares + WHERE shared_with = $1 + ) + `, + params: { '1': 'user_123' }, + }, + }, + }) +) +``` + Or with `ShapeStream` directly: ```ts @@ -154,10 +181,16 @@ const stream = new ShapeStream({ const shape = new Shape(stream) ``` -When a shape uses a subquery, Electric tracks the dependency between tables. If the data in the subquery changes (e.g., a project becomes archived), rows will automatically move in or out of the shape without the row itself being modified. +When a shape uses a subquery, Electric tracks the dependency between tables. If the data in the subquery changes (for example, a project becomes archived or a membership row is added), rows will automatically move in or out of the shape without the root row itself being modified. + +Electric 1.6 keeps these moves incremental even for compound expressions that use `AND`, `OR`, and `NOT` around subqueries. In older releases those cases could return `409` and force a full resync of the shape. + +:::info Feature flags +Subqueries are currently enabled using `ELECTRIC_FEATURE_FLAGS=allow_subqueries,tagged_subqueries`. The flags are unchanged; they gate rollout rather than a separate syntax or API. See the [configuration docs](/docs/api/config#allow_subqueries) for details. +::: -:::warning Experimental feature -Subqueries require enabling feature flags. Set `ELECTRIC_FEATURE_FLAGS=allow_subqueries,tagged_subqueries` to enable. See the [configuration docs](/docs/api/config#allow_subqueries) for details. +:::warning Client compatibility +The incremental behavior for compound subquery expressions in Electric 1.6 requires updated clients. TanStack DB clients need `@tanstack/db >= 0.6.2` and `@tanstack/electric-db-collection >= 0.3.0`. Upgrade clients before rolling out the server update. ::: When constructing a where clause with user input as a filter, it's recommended to use a positional placeholder (`$1`) to avoid @@ -399,8 +432,9 @@ With non-optimized where clauses, throughput is inversely proportional to the nu With optimized where clauses, Electric can evaluate millions of clauses at once and maintain a consistent throughput of ~5,000 row changes per second **no matter how many shapes you have**. If you have 10 shapes, Electric can process 5,000 changes per second. If you have 1,000 shapes, throughput remains at 5,000 changes per second. -For more details see the [benchmarks](/docs/reference/benchmarks#_7-write-throughput-with-optimized-where-clauses) and [this blog post](/blog/2025/08/13/electricsql-v1.1-released) about our storage engine. +For more details see the [benchmarks](/docs/reference/benchmarks#_7-write-throughput-with-optimised-where-clauses) and [this blog post](/blog/2025/08/13/electricsql-v1.1-released) about our storage engine. + ### Optimized where clauses We currently optimize the evaluation of the following clauses: @@ -411,6 +445,7 @@ We currently optimize the evaluation of the following clauses: Note that this index is internal to Electric and unrelated to Postgres indexes. - `field = constant AND another_condition` - the `field = constant` part of the where clause is optimized as above, and any shapes that match are iterated through to check the other condition. Providing the first condition is enough to filter out most of the shapes, the write processing will be fast. If however `field = const` matches for a large number of shapes, then the write processing will be slower since each of the shapes will need to be iterated through. - `a_non_optimized_condition AND field = constant` - as above. The order of the clauses is not important (Electric will filter by optimized clauses first). +- `field = constant OR field = other_constant` - if both `OR` branches are individually indexable, Electric indexes both sides and unions the matching shapes instead of evaluating every shape. If one side of the `OR` is not optimizable, Electric falls back to normal per-shape evaluation for that clause. > [!Warning] Need additional where clause optimization? > We plan to optimize a much larger subset of Postgres where clauses. If you need a particular clause optimized, please [raise an issue on GitHub](https://github.com/electric-sql/electric) or [let us know on Discord](https://discord.electric-sql.com). @@ -419,7 +454,7 @@ We currently optimize the evaluation of the following clauses: ### Single table -Shapes sync data from a single table. While you can use [subqueries](#subqueries-experimental) to filter rows based on data in other tables, the shape only contains rows from the root table—not the related data itself. +Shapes sync data from a single table. While you can use [subqueries](#subqueries) to filter rows based on data in other tables, the shape only contains rows from the root table—not the related data itself. For syncing related data across tables, you currently need to use multiple shapes. In the [old version of Electric](https://legacy.electric-sql.com/docs/usage/data-access/shapes), Shapes had an include tree that allowed you to sync nested relations. The new Electric has not yet implemented support for include trees. diff --git a/website/docs/reference/benchmarks.md b/website/docs/reference/benchmarks.md index fd975fd6c0..23084293b4 100644 --- a/website/docs/reference/benchmarks.md +++ b/website/docs/reference/benchmarks.md @@ -124,7 +124,7 @@ Each shape in this benchmark is independent, ensuring that a write operation aff The two graphs differ based on the type of where clause used for the shapes: - **Top Graph:** The where clause is in the form `field = constant`, where each shape is assigned a unique constant. These types of where clause, along with - [other patterns](/docs/guides/shapes#optimised-where-clauses), + [other patterns](/docs/guides/shapes#optimized-where-clauses), are optimised for high performance regardless of the number of shapes — analogous to having an index on the field. As shown in the graph, the latency remains consistently flat at 6ms as the number of shapes increases. This 6ms latency includes 3ms for PostgreSQL to process the write operation and 3ms for Electric to propagate it. We are actively working to optimise additional where clause types in the future. @@ -182,6 +182,7 @@ In this benchmark there are a varying number of shapes with just one client subs Latency and peak memory use rises linearly. Average memory use is flat. + #### 7. Write throughput with optimised where clauses
@@ -202,10 +203,10 @@ is using an optimised where clause, specifically `field = constant`. > so that we can evaluate millions of where clauses at once, providing the where clauses follow various patterns, which we call optimised where clauses. > `field = constant` is one of the patterns we optimise, we can evaluate millions of these where clauses at once by indexing the shapes based on the constant > value for each shape. This index is internal to Electric, and nothing to do with Postgres indexes. It's a hashmap if you're interested. -> `field = const AND another_condition` is another pattern we optimise. We aim to optimise a large subset of Postgres where clauses in the future. +> `field = const AND another_condition` is another pattern we optimise, and some indexable `OR` combinations are now optimised too. This benchmark still measures the specific `field = constant` case shown above. > Optimised where clauses mean that we can process writes in a quarter of a millisecond, regardless of how many shapes there are. > -> For more information on optimised where clauses, see the [shape API](/docs/guides/shapes#optimised-where-clauses). +> For more information on optimised where clauses, see the [shape API](/docs/guides/shapes#optimized-where-clauses). The top graph shows throughput for Postgres 14, the bottom graph for Postgres 15. diff --git a/website/public/img/blog/subqueries/header.jpg b/website/public/img/blog/subqueries/header.jpg new file mode 100644 index 0000000000..2d26f9c146 Binary files /dev/null and b/website/public/img/blog/subqueries/header.jpg differ