server: batch dialog work and qualify prepared queries

server/BENCHMARKING.md

@@ -36,6 +36,10 @@ bun run bench:backend --compare .test-results/before.json --output .test-results
 # Explore sensitivity to additional DB transport latency on one path.
 bun run bench:backend --scenario send.dm.open --rtt 0,3,5
 
+# Compare safe default preparation with qualified broad preparation.
+bun run bench:backend --preparation access --scenario send.dm.open --rtt 0,3,5 --output .test-results/access.json
+bun run bench:backend --preparation all-custom --scenario send.dm.open --rtt 0,3,5 --compare .test-results/access.json --output .test-results/all-custom.json
+
 # Check the entire benchmark workflow quickly; this is not a tail-latency study.
 bun run bench:backend --all --samples 1 --warmup 1
 ```
@@ -47,6 +51,20 @@ never overwritten. Unknown selections, invalid arguments, failed assertions,
 background failures, failed cleanup and incomplete runs exit unsuccessfully.
 Interrupted worker artifacts are candidates, not accepted comparison baselines.
 
+`--preparation access` is the default; shell rollout flags are deliberately not
+inherited. `--preparation all-custom` sets `force_custom_plan` only as a database
+default on the lifecycle's disposable local database. It then recreates the
+application pools and validates startup before measuring. The provisioning
+database and role defaults are never changed. Use at least three warmups when
+studying retained broad statements, which require two observations for admission.
+
+Reports record the requested mode, the actual `plan_cache_mode`, and statement/
+execution counters from one sampled backend after the matrix. These counters
+include driver-owned statements and are not an admission count or pool total.
+Access-to-all-custom comparisons display both treatments and planning settings.
+Legacy reports without preparation metadata remain readable but cannot compare
+to new qualified reports; rerun those baselines with an explicit mode.
+
 Run timing experiments alone, on the same otherwise idle machine, database
 version and power configuration. Avoid concurrent builds and tests. Repeat A/B
 measurements before claiming a small improvement. Keep reports with the change;
@@ -77,6 +95,9 @@ Each iteration follows this sequence:
 | `commandsBeforeReturn` | Commands submitted by return time; background work can overlap, so this is **not** a foreground-only count |
 | `wire.exchangeBoundaries` | Frontend `Query`, `Flush`, and `Sync` frames observed at the proxy |
 | `calibrationMs` | Ten real `SELECT 1` timings at the requested delay, outside the operation samples |
+| `preparation.mode` / `planCacheMode` | Explicit benchmark treatment and actual backend planning policy |
+| `preparation.sampledBackendStatements` | Retained statements visible on one backend after the matrix; includes driver-owned queries |
+| `preparation.sampledBackendCustomPlans` / `sampledBackendGenericPlans` | Execution counters on that sampled backend, rather than pool-wide totals |
 
 One driver submission can contain several SQL statements, so `sql.commands` is
 not a SQL parser's statement count. A transaction does not make its statements
@@ -123,10 +144,12 @@ Do not disable domain jobs to make a path pass its budget.
 `catalog.ts` holds reviewed maximum command counts. They represent current costs,
 including costly linear fanout; they are not desired targets. Lower them with a
 proven optimization. Do not automatically regenerate budgets or loosen one to
-hide a regression. The 1/10/100 cases expose scaling: current public sends cost
-`32 + recipients`, reply sends `66 + 7 × recipients`, and user update batches
-`2 + 4 × updates` for these fixtures. Chat-list queries remain constant at 11
-while result volume grows. These are local fixture observations, not production
+hide a regression. The 1/10/100 cases expose scaling: the reviewed public-send budget is now constant at
+`32` after batching sidebar owner locks, reply sends retain a conservative
+`66 + 7 × recipients` bound, and user update batches
+`2 + 4 × updates` for these fixtures. Unchanged dialog batches cost 5 commands at 1/10/100 owners. Chat-list
+command budgets include their snapshot and admission overhead while remaining
+constant as result volume grows. These are local fixture observations, not production
 traffic statistics.
 
 `test:backend` also selects the existing send, chat-list, history, message integrity,
@@ -191,3 +214,47 @@ matrix, runtime/driver/database/platform/CPU/pool settings, warmups and sample
 counts. They report saved commands, exchanges and milliseconds separately, along
 with before/after RTT calibration. A comparison is evidence to inspect, not an
 automatic correctness proof or a promise of production latency savings.
+
+## Prepared query policy
+
+The default `INLINE_DATABASE_PREPARED_QUERIES=access` prepares only the existing
+`sync.replay` and `sync.sidecars` access family. Its 100 batch lengths per source
+fit a 200-signature budget. Other Drizzle queries retain unnamed execution and
+custom planning. Rows and authorization decisions are never cached.
+
+`INLINE_DATABASE_PREPARED_QUERIES=all-custom` opts into broader scalar/ordinary-array
+preparation, including transactions, savepoints and reserved clients. Startup
+must first observe `plan_cache_mode=force_custom_plan` on the actual query path;
+readiness checks it again. This requires an operator-qualified application role
+default on every backend, with existing pooled sessions reestablished after the
+role change. PgBouncer does not accept this as an ordinary startup parameter.
+The application never changes roles or session settings. A sampled startup check
+cannot establish that every existing backend has inherited the role default.
+
+Broad admission requires a second observation, with at most 200 pending candidates.
+This avoids retaining one-off SQL shapes, such as transient `inArray` lengths.
+It admits at most 100 broad signatures and leaves 100 slots for access queries,
+within the same total cap of 200. Retained signatures live until pool shutdown;
+they are not an LRU. Once a partition or total budget fills, new shapes use the
+original unnamed path. Changing hot lists to a single array binding avoids a
+new signature for each list length. The default access mode keeps its full 200
+slots. Explicit per-query options and the pool-level preparation switch remain
+authoritative. Queries longer than 32 KiB, more than 301 parameters, and custom
+typed parameters retain their prior mode.
+
+Warm admitted calls skip Describe/Flush while PostgreSQL builds custom plans
+from current bindings. The wire fixture opts into `all-custom` using a dedicated
+direct connection with the setting established before startup. The benchmark
+launcher provides its own disposable-database qualification via `--preparation`. Database tests
+assert partial-index selection and zero generic plans after more than five
+executions, plus rollback, changed array bindings and admission pressure.
+
+Measurements from commit `5dc49a0f2` used unconditional broad preparation under
+the database's prior planning policy. They do not describe this safe default or
+qualify `all-custom` latency. New comparisons must record the preparation mode,
+role setting, admission limits and enough warmups (at least three) for repeated
+shapes; inspect `pg_prepared_statements` counters to verify prepared reuse and planning policy.
+
+Role-wide custom planning also applies to the previously tuned access queries.
+Qualify representative replay latency as well as simple point reads before
+enabling broad mode; faster protocol exchanges alone do not prove a net gain.

server/bench/backend.test.ts

@@ -4,34 +4,57 @@ import { createHash } from "node:crypto"
 import { cpus } from "node:os"
 import { resolve } from "node:path"
 import { sql } from "drizzle-orm"
-import { closeDb, db, initDb } from "@in/server/db"
+import { closeDb, db, initDb, validateDatabaseStartup } from "@in/server/db"
 import { setupTestDatabase, teardownTestDatabase } from "@in/server/__tests__/database"
 import { scenarios, scenarioVersion, selectScenarios } from "@in/server/__tests__/performance/catalog"
 import { prepareScenario } from "@in/server/__tests__/performance/scenarios"
 import { observeScenarioWork } from "@in/server/__tests__/performance/observe"
 import { assertQueryBudget, measureOperation, type Sample } from "@in/server/__tests__/performance/measure"
 import { createWireProxy } from "@in/server/__tests__/performance/wire"
-import { summarize, type BenchmarkReport } from "@in/server/__tests__/performance/report"
+import { summarize, type BenchmarkReport, type BenchmarkPreparation } from "@in/server/__tests__/performance/report"
 
 const encoded = process.env["INLINE_BACKEND_BENCH_CONFIG"]
 if (!encoded) throw new Error("Use bun run bench:backend; this worker needs the isolated benchmark launcher")
-const config = JSON.parse(encoded) as { ids: string[]; samples: number; warmup: number; rtts: number[]; output: string }
+const config = JSON.parse(encoded) as { ids: string[]; samples: number; warmup: number; rtts: number[]; preparation: BenchmarkPreparation["mode"]; output: string }
+if (config.preparation !== "access" && config.preparation !== "all-custom") throw new Error("Invalid benchmark preparation mode")
+const previousPreparation = process.env["INLINE_DATABASE_PREPARED_QUERIES"]
 const selected = selectScenarios(config.ids)
 const background = observeScenarioWork()
 let proxy: Awaited<ReturnType<typeof createWireProxy>> | undefined
 
 beforeAll(async () => {
   await setupTestDatabase()
   const url = new URL(process.env["DATABASE_URL"]!)
+  if (config.preparation === "all-custom") {
+    // Only the lifecycle's freshly cloned database may receive this default.
+    // No role change, provisioning database change, or session-only shortcut.
+    const template = process.env["INLINE_TEST_DATABASE_TEMPLATE"]
+    const name = url.pathname.slice(1)
+    if (!template || !/^inline_test_template_[0-9]+_[0-9a-f]{32}$/.test(template) ||
+        !new RegExp(`^test_db_${template.slice(-32)}_[0-9a-f]{12}$`).test(name) ||
+        !["localhost", "127.0.0.1"].includes(url.hostname)) {
+      throw new Error("Refusing to configure an unowned benchmark database")
+    }
+    const [current] = await db.execute<{ name: string }>(sql`select current_database() as name`)
+    if (current?.name !== name) throw new Error("Benchmark database identity changed")
+    await db.$client.unsafe(`ALTER DATABASE "${name}" SET plan_cache_mode = force_custom_plan`)
+  }
+  process.env["INLINE_DATABASE_PREPARED_QUERIES"] = config.preparation
   proxy = await createWireProxy({ host: url.hostname, port: Number(url.port || 5432) }, 0)
   url.hostname = "127.0.0.1"
   url.port = String(proxy.port)
   await closeDb()
   initDb(url.toString())
+  await validateDatabaseStartup()
 })
 afterAll(async () => {
   try { await background.close() } finally {
-    try { await teardownTestDatabase() } finally { await proxy?.close() }
+    try { await teardownTestDatabase() } finally {
+      try { await proxy?.close() } finally {
+        if (previousPreparation === undefined) Reflect.deleteProperty(process.env, "INLINE_DATABASE_PREPARED_QUERIES")
+        else process.env["INLINE_DATABASE_PREPARED_QUERIES"] = previousPreparation
+      }
+    }
   }
 })
 
@@ -107,6 +130,24 @@ test("verified backend benchmark matrix", async () => {
   if (report.results.length !== selected.length * config.rtts.length || selected.some((entry) => !scenarios.includes(entry))) {
     throw new Error("Incomplete benchmark matrix")
   }
+  const [settings] = await db.execute<{ mode: string; statements: number; custom: string; generic: string }>(sql`
+    select current_setting('plan_cache_mode') as mode, count(*)::integer as statements,
+      coalesce(sum(custom_plans), 0)::text as custom, coalesce(sum(generic_plans), 0)::text as generic
+    from pg_prepared_statements
+  `)
+  if (!settings || !["auto", "force_custom_plan", "force_generic_plan"].includes(settings.mode)) {
+    throw new Error("Cannot record benchmark planning policy")
+  }
+  if (config.preparation === "all-custom" && settings.mode !== "force_custom_plan") {
+    throw new Error("Benchmark lost custom-plan qualification")
+  }
+  report.preparation = {
+    mode: config.preparation, planCacheMode: settings.mode as BenchmarkPreparation["planCacheMode"],
+    sampledBackendStatements: settings.statements,
+    sampledBackendCustomPlans: Number(settings.custom), sampledBackendGenericPlans: Number(settings.generic),
+  }
+  console.info(`Preparation ${report.preparation.mode}; plan_cache_mode=${report.preparation.planCacheMode}; ` +
+    `sampled backend: ${settings.statements} retained statements, ${settings.custom} custom / ${settings.generic} generic executions`)
   if (await sourceDigest() !== digest) throw new Error("Backend sources changed during the benchmark; rerun on a stable checkout")
   // The launcher promotes this only after Bun hooks and DB disposal also pass.
   writeFileSync(config.output, JSON.stringify({ ...report, status: "candidate" }, null, 2) + "\n", { flag: "wx" })

server/docs/database-observability.md

@@ -84,15 +84,26 @@ transactions and deployment overlap.
 The shared query client now prepares only tagged `sync.replay` and
 `sync.sidecars` access queries with at most 301 parameters (100 chat IDs used
 three times, plus one user). Their two sources yield at most 200 statement
-shapes per connection. Larger batches and transaction-local queries retain the
-existing driver behavior. Explicit `prepare: false` is respected. This caches
+shapes per connection. Larger batches retain the existing driver behavior. The same bounded policy
+also applies to transactions, savepoints and reserved clients. Explicit `prepare: false` is respected. This caches
 plans, never permissions or results. Production PgBouncer was verified to use
 transaction mode with `max_prepared_statements=200`; keep this capability when
 changing providers or pool configuration. Local integration tests exercise
 actual Drizzle prepared reads through PgBouncer, and authorization tests cover
 revocation after warm repeated reads. Array-binding alternatives were rejected
-after live probes showed PostgreSQL kept replanning them instead of selecting
-the much faster generic plan for the existing query shape.
+after historical live probes showed PostgreSQL kept replanning them instead of
+selecting the faster generic plan for that query shape. This is evidence for the
+narrow family, not for enabling generic plans across all queries.
+
+Broader preparation requires explicit `INLINE_DATABASE_PREPARED_QUERIES=all-custom`
+and an operator-established `force_custom_plan` default on the query role. The
+startup and readiness probes inspect `plan_cache_mode` through the query pool;
+startup rejects unqualified opt-ins, and readiness disables broad admission if
+the sampled backend loses qualification. Establish the role default and recycle
+existing PgBouncer backend sessions before enabling it: sampling one backend
+does not prove the setting on every backend. No role or session settings are
+changed by this code. See [the prepared-query policy](../BENCHMARKING.md#prepared-query-policy)
+for the bounded second-observation admission policy and benchmark limits.
 
 Replay access checks execute the canonical SELECT directly on the shared
 client. The former single-query explicit transaction added BEGIN/SET/COMMIT

server/scripts/benchmark-backend.test.ts

@@ -25,6 +25,7 @@ test.each([
   ["--warmup", "NaN", "--warmup must"],
   ["--rtt", "0,,3", "Empty RTT"],
   ["--rtt", "-1", "RTT must"],
+  ["--preparation", "invalid", "--preparation must be access or all-custom"],
   ["--output", ".env.json", "never environment files"],
   ["--retry", "3", "Unknown option"],
 ])("invalid %s %s fails before provisioning: %s", async (flag, value, message) => {
@@ -37,6 +38,7 @@ function report(): BenchmarkReport {
   return {
     schemaVersion: 1, status: "passed", createdAt: "2026-01-01T00:00:00.000Z", revision: { commit: "fixture", dirty: false, sourceDigest: "a".repeat(64) },
     environment: { bun: "1.4.0", postgres: "15", postgresJs: "3.4.7", drizzle: "fixture", platform: "darwin", arch: "arm64", cpu: "fixture", poolMax: 10, clientPrepareDefault: true, concurrency: 1, cache: "cold-application-warm-pool" },
+    preparation: { mode: "access", planCacheMode: "auto", sampledBackendStatements: 2, sampledBackendCustomPlans: 5, sampledBackendGenericPlans: 0 },
     warmup: 1, sampleCount: 1,
     results: [{ id: "send.dm.open", version: 1, size: 1, rttMs: 3, maxCommands: 27, calibrationMs: [3], samples: [{
       operationMs: 10, settledMs: 12, commandsBeforeReturn: 2,
@@ -73,3 +75,36 @@ test("failed, empty, incomplete and incompatible runs cannot be compared as succ
     expect(() => compareReports(report(), changed)).toThrow()
   }
 })
+
+
+test.each(["access", "all-custom"])("explicit %s preparation can list without provisioning", async (mode) => {
+  const result = await run(`--preparation=${mode}`, "--list")
+  expect(result.code).toBe(0)
+  expect(result.output).toContain("send.public.100")
+})
+
+test("preparation comparisons identify the treatment and its actual planning policy", () => {
+  const before = report(), after = report()
+  after.preparation = { mode: "all-custom", planCacheMode: "force_custom_plan", sampledBackendStatements: 10, sampledBackendCustomPlans: 20, sampledBackendGenericPlans: 0 }
+  expect(compareReports(before, after)[0]).toMatchObject({
+    preparationBefore: "access", preparationAfter: "all-custom",
+    planCacheModeBefore: "auto", planCacheModeAfter: "force_custom_plan",
+  })
+})
+
+test("preparation metadata rejects unqualified modes and malformed backend counters", () => {
+  for (const metadata of [
+    { mode: "unknown", planCacheMode: "auto", sampledBackendStatements: 0, sampledBackendCustomPlans: 0, sampledBackendGenericPlans: 0 },
+    { mode: "all-custom", planCacheMode: "auto", sampledBackendStatements: 0, sampledBackendCustomPlans: 0, sampledBackendGenericPlans: 0 },
+    { mode: "access", planCacheMode: "auto", sampledBackendStatements: -1, sampledBackendCustomPlans: 0, sampledBackendGenericPlans: 0 },
+    { mode: "access", planCacheMode: "auto", sampledBackendStatements: 0 },
+  ]) expect(() => validateReport({ ...report(), preparation: metadata })).toThrow("preparation")
+})
+
+test("legacy reports remain readable but cannot silently compare to a qualified mode", () => {
+  const legacy = report()
+  delete legacy.preparation
+  expect(() => validateReport(legacy)).not.toThrow()
+  expect(() => compareReports(legacy, report())).toThrow("Preparation policy is unknown")
+  expect(() => compareReports(legacy, structuredClone(legacy))).not.toThrow()
+})