OncoPrint and plots at scale: positional assembly, concurrent and batched fetches

docs/PERFORMANCE.md

@@ -812,6 +812,66 @@ What is left, measured at 128-551 studies:
   definition, which every pooled query then evaluates: 0.9s → 3.0s at 32 studies with an OQL
   draft when forced. No study on the primary connection is that large; the answers are the same.
 
+## 3e. OncoPrint and plots at scale (October 2026)
+
+Measured by calling the engine directly (no result cache) against the public warehouse, two or
+three alternating runs each against the code before; the warehouse is shared, so single runs move
+by up to 30% and only alternating pairs are compared. Every case listed returns a contract
+identical to the code before's (sha256 of the contract without its timings, 13 specs: heatmaps,
+VAF, a generic assay, Ward clustering, overlays, derived rows, merged and sample-level tracks,
+cohort groups, five refusals), and both oncoprint views of the analysis route render identically.
+
+In the browser the figure is not the cost: at 409k patients the binary contract decodes in under
+1 ms and draws in 45-107 ms (`/oncoprint-lab`, headless Chromium).
+
+| oncoprint | before | after |
+|---|---|---|
+| msk_impact_50k_2026, 48k patients, 10 genes | 1.4-1.9s | 1.3-1.5s (assembly 0.5-0.8s → 0.3-0.4s) |
+| 5 largest studies, 166k patients | 3.8-5.0s | 2.5-2.7s (assembly 2.1-2.6s → 0.6-0.7s) |
+| GENIE public + 5 largest, 409k patients | 10.8-13.9s | 7.5-8.2s (assembly 5.4-7.0s → 1.7-2.2s) |
+| 50 studies, 5 genes | 5.6-6.0s | 3.4-3.8s |
+| 312 studies, 242k patients, 5 genes | 14.7-17.4s (1,560 queries) | 4.5-5.6s |
+
+- **Assembly addressed every row by its column id**, a string-keyed hash lookup per row per
+  phase, on ids rebuilt as `study + "_" + pid` for every row of a multi-study figure (1.0-1.4 µs
+  each measured, against 0.33 for a plain id). `buildContract` now meets each id once, in a
+  per-study map, and hands `assembleIndexed` rows by column position; `assembleContract` keeps the
+  id-keyed interface (CLI, local tables, the Python conformance tests) as a thin adapter onto it.
+  The three hottest loops are functions of their own: inside one 400-line function V8 optimised
+  each loop on its own and threw the code away on reaching the next ("insufficient type feedback").
+- **Every study's fetches start together**: clinical tracks, heatmaps, overlays, derived rows and
+  the driver annotation no longer wait for the alterations, and the annotation reuses the cached
+  OncoKB release instead of asking for it again (126-232 ms) and reads only driver rows.
+- **The whole-study fetches are one statement per batch of studies** (`batchStudies`): small
+  studies share a statement up to 25,000 samples, a larger study keeps its own. One statement per
+  connection was measured slower for six of the largest studies (query phase 6.2s against 5.1-5.7s)
+  — §3c's finding again: separate statements spread across the cluster. Each study's sample count,
+  for this and nothing else, is cached for the process's life.
+- **Panel coverage** computes each panel's gene mask before joining samples, and joins only these
+  studies' samples: 195 → 86 ms at 48k patients, identical rows over 42 study × gene list × unit cases.
+
+Plots changed less. One study was already 0.35-0.85s at 52k samples. Across studies:
+
+- The pooled box plot fetched every study's points and then dropped them all when the total
+  passed the 10,000-point cap (11 studies, 150k samples: 11 queries for nothing). It now skips
+  them when the box's own count is past the cap, which is exact: a study's points are at least
+  its box's cases. That fetch was also, by accident, the only place every study's metadata was
+  checked against the axes; the check is now explicit and first, so a plot one study cannot answer
+  is refused in 0.7-1.2s instead of 3.2-6.2s at 312 studies.
+- Coverage counts run beside the second round of queries instead of after it; a single-study
+  scatter asks for its coefficients beside its extent.
+- The analysis loaded every study's profiles three queries at a time per study, and listed each
+  study's import date one query after another once the views were done: 312 queries, 3.1s, after
+  a 312-study plot had finished. Both are now one statement per connection (`studyProfilesMany`,
+  `studiesImported`).
+- **What is left is the plot's own query count**: four per study, 1,248 at 312 studies, 11-25s as the shared database's load moves.
+  Sent as one `UNION ALL` request per kind (built, measured, not shipped) they were as slow as
+  sending them one after another — 11 statements 600-700 ms against 160-250 ms eight at a time.
+  Two levers remain, neither taken here: `CLICKHOUSE_MAX_INFLIGHT=16` took that view from 19.4s
+  to 12.2s and 32 gained nothing more (a deployment setting: it is everyone's share of the
+  database); or a plot statement that reads many studies in one scan, which means compiling an
+  axis — clinical, OQL, profile, generic assay — for several studies at once, against invariant 3.
+
 ## 4. Time to display vs cbioportal.org
 
 Oncoprint, 5 genes, measured from navigation to the figure's pixels going stable

lib/analysis.ts

@@ -386,6 +386,8 @@ export async function runAnalysis(template: AnalysisDoc | TemplateDoc, opts: Run
   const evalCohortDef = async (def: CohortDef): Promise<EvaledCohort> => {
     const studies = [];
     for (const s of cohortStudies(def, docStudy)) studies.push(await resolveStudy(s, docStudy));
+    // every study's profiles at once, rather than three queries for each
+    await studyProfilesMany(studies);
     const perStudy: Record<string, { sql: string; count: number; mode: string }> = {};
     // Every instance is asked for a COUNT, including the patient-level ones.
     // Cohort evaluation has no use for identities: the size is a count, and the

lib/annotation.ts

@@ -86,18 +86,20 @@ async function syncedStudies(): Promise<string[]> {
   return _syncedStudies;
 }
 
-export async function buildAnnotationIndex(eventRows: EvRow[], genes: string[], study?: string | string[]): Promise<AnnotationIndex> {
+/**
+ * `eventRows` may still be on its way: the store is read without them, and only the mock that
+ * stands in for a missing store waits for them.
+ */
+export async function buildAnnotationIndex(eventRows: EvRow[] | Promise<EvRow[]>, genes: string[], study?: string | string[]): Promise<AnnotationIndex> {
   try {
-    const pin = process.env.ONCOKB_PIN ||
-      (await chQuery(
-        `SELECT oncokb_version FROM ${ANNOT_DB}.variant_annotation
-         GROUP BY oncokb_version ORDER BY max(resolved_at) DESC LIMIT 1`)).data[0]?.oncokb_version;
+    const pin = await oncokbPin();
     if (!pin) throw new Error("annotation store is empty");
     const rows = await chQuery(
       `SELECT alteration_type AS t, hugo_gene_symbol AS gene, alteration, max(is_driver) AS is_driver
        FROM ${ANNOT_DB}.variant_annotation
        WHERE oncokb_version = ${sq(pin)} AND hugo_gene_symbol IN (${genes.map(sq).join(",")})
-       GROUP BY t, gene, alteration`);
+       GROUP BY t, gene, alteration
+       HAVING is_driver > 0`);
     // Nested by gene rather than one set of "type|gene|alteration" keys: isDriver runs once per
     // event — 800k times on a 400k-patient figure — and composing that key allocated a string
     // every time. Two lookups on short strings cost far less than one on a long fresh one.
@@ -137,6 +139,6 @@ export async function buildAnnotationIndex(eventRows: EvRow[], genes: string[],
     // or availability problem, and nobody is waiting for the answer anyway.
     if (isAbortError(e)) throw e;
     // not licensed / store empty / db absent -> degrade loudly to the mock
-    return mockIndex(eventRows);
+    return mockIndex(await eventRows);
   }
 }

lib/federation.ts

@@ -1750,11 +1750,14 @@ abstract class BaseResolver implements AggregateCapable {
     const done = async (d: PlotData): Promise<PlotData> => { const c = await coverage; return c ? { ...d, coverage: c } : d; };
 
     if (shape === "scatter") {
+      // the coefficients need no extent, so they are asked for beside it
+      const stP = this.runLogged<ChRow>("plot:stats", plotStatsSql(xSel, ySel, ctx.cohortSql), conn);
+      stP.catch(() => {});
       const c = await this.runLogged<ChRow>("plot:extent", plotCountSql(xSel, ySel, ctx.cohortSql, true), conn);
       const n = Number(c.data[0]?.n || 0);
       if (!n) return done({ shape, n: 0, ...label, note: "no " + unit + " has a value on both axes" });
       const [st, bins] = await Promise.all([
-        this.runLogged<ChRow>("plot:stats", plotStatsSql(xSel, ySel, ctx.cohortSql), conn),
+        stP,
         this.runLogged<ChRow>("plot:bins", plotBinsSql(xSel, ySel, ctx.cohortSql, spec.bins), conn)
       ]);
       const r = st.data[0] || {};