Data model, CodeMirror SQL editor and a result store (2.2.3)

.changeset/sql-editor-codemirror.md

@@ -0,0 +1,49 @@
+### New Features
+
+#### SQL editor
+- The console and notebook cells run on CodeMirror: faster to open and type in, with suggestions that stay at the caret
+- Schema-aware completion: tables after FROM, UPDATE and INTO, the named table's columns with their types first, `alias.` and `schema.` lookups, functions with fill-in arguments, enum values and your own functions
+- Columns are suggested for any table in the schema, opened or not, and typing `"` lists column names
+- 34 snippets named after their keyword (`sel`, `ins`, `upd`, `join`), with a preview beside the list
+- SQL editor settings in Settings → Database and behind the toolbar's sliders: text size, line wrap (Alt+Z), line numbers, fold arrows, suggestions while typing and problem markers
+- A spinner beside each running statement turns into a ✓ when it finishes
+- A failed statement shows the database's message and where it failed, with the failing text underlined in the editor and in the results
+- Run comes first in the console toolbar and Stop takes its place, so nothing moves when a run starts
+
+#### Data model (was ER diagram)
+- The ER diagram is now the Data model: one tab with five views of the same tables, Diagram, Mermaid, Tree, Dictionary and DDL
+- Each view keeps its own controls in one toolbar: search and settings on the Diagram, the column search on the Dictionary, copy and Export .sql on the DDL, and the table filter, hops, zoom and Flow / All / List on the Tree
+- The Diagram places cards and routes lines in one pass, so a line never crosses a card and lines sharing a channel get a track each. A 135-table schema lays out in under a second instead of 45
+- Hub tables (`tenants`, `users`) are named in a pill on each referencing row instead of drawing a line from every card; their lines show for the card under the pointer or the selected one
+- Mermaid shows the page as `erDiagram` source beside a live preview, to copy, edit, or save as a diagram with Edit in Diagrams
+- Tree shows what a table points at and what points at it: as a map of the whole page (the default), a flowchart around one table, or a list with row counts and 1:1 / N:1 / 1:N on each link
+- Dictionary lists every column on the page with its type, nullability, keys and references, searchable with ⌘F; a table name opens the table
+- DDL shows the CREATE statements of the tables on the page, with copy and export to a .sql file
+- The table filter shows what is on the page and changes it a tick at a time, with All, Linked and + Neighbours, Enter to keep only the search matches, and Focus for one table and its links
+- Full screen from the toolbar or ⌘⇧↵ (Ctrl+Shift+Enter); Esc leaves it
+- Mermaid diagrams follow the app theme, stay sharp at every zoom, and zoom out only until the whole picture fits
+
+#### Data grid
+- Staged edits are reviewed in a resizable dock under the grid instead of a dialog, so the rows stay in view; the SQL follows the grid and can be edited with completion
+- Following a foreign key to a single row opens that row as JSON (Settings → Data grid → Expand single related row)
+
+#### AI
+- A plain chat turn sends about 2k tokens instead of 8k: skills and tools come along only when the conversation needs them, and old results are trimmed from the history
+- The conversation title no longer costs a second model call, which kept tripping the free tier's rate limit
+- Chat and code text follow the app zoom
+
+### Bug Fixes
+- A very large result no longer closes the app: a 5-million-row `SELECT` used to come back as one message. Postgres results now spool into a store in the backend as they arrive, the grid loads the rows near the screen as you scroll, and sorting runs in the store by the column's type
+- A console query on Postgres waits on half the network round trips (4 instead of 8 on a fresh connection), so a small query to a far database comes back in about half the time
+- Primary keys come from the catalog on every engine, so a `uuid` or text key shows as a key on the cards, in keys-only mode and in the Mermaid source
+- Scrolling with an expanded JSON row open no longer drops to 11-20 fps, and the grid no longer shows torn rows after a zoom or opening a dock
+- Switching back to a tab with an expanded JSON row no longer jumps
+- The title bar no longer slides up under the window edge
+- The connecting spinner turns with Reduce Motion on, and the Stop button's spinner is centred
+- Format keeps short clauses on one line
+- A query without a trailing `;` is no longer flagged; a missing `;` between two queries is
+- Line numbers stay aligned after a zoom or font change
+
+### Changes
+- `id`, `name`, `type` and other everyday column names are no longer coloured as keywords
+- The diagram settings are one compact row each, with Copy as PNG and one Download row for PNG, SVG and Mermaid

.gitignore

@@ -51,3 +51,6 @@ file.txt
 **/.claude/phobos-state.md
 **/.claude/phobos-activity.log
 **/.claude/phobos-benchmark.jsonl
+
+# Local brag/demo media, not source
+brag-output/

package.json

@@ -46,9 +46,11 @@
     "vitest": "^4.1.11"
   },
   "dependencies": {
+    "@codemirror/autocomplete": "^6.20.3",
     "@codemirror/commands": "^6.11.1",
     "@codemirror/lang-html": "^6.4.12",
     "@codemirror/lang-json": "^6.0.2",
+    "@codemirror/lang-sql": "^6.10.0",
     "@codemirror/language": "^6.12.4",
     "@codemirror/search": "^6.7.2",
     "@codemirror/state": "^6.7.6",
@@ -67,6 +69,7 @@
     "@hugeicons/core-free-icons": "^4.2.2",
     "@hugeicons/svelte": "^1.1.4",
     "@lezer/highlight": "^1.2.4",
+    "@replit/codemirror-vim": "^6.4.0",
     "@tailwindcss/vite": "^4.3.0",
     "@tanstack/svelte-hotkeys": "^0.10.0",
     "@tauri-apps/plugin-autostart": "^2.5.1",
@@ -82,6 +85,7 @@
     "clsx": "^2.1.1",
     "echarts": "^5.6.0",
     "echarts-wordcloud": "^2.1.0",
+    "elkjs": "^0.11.1",
     "idb": "^8.0.3",
     "marked": "^18.0.4",
     "mermaid": "^11.16.1",

src-tauri/Cargo.toml

@@ -18,7 +18,7 @@ crate-type = ["staticlib", "cdylib", "rlib"]
 tauri-build = { version = "2.6.2", features = [] }
 
 [dependencies]
-serde_json = { version = "1.0", features = ["preserve_order"] }
+serde_json = { version = "1.0", features = ["preserve_order", "raw_value"] }
 serde = { version = "1.0", features = ["derive"] }
 log = "0.4"
 tauri = { version = "2.11.2", features = ["macos-private-api", "image-png"] }

src-tauri/src/commands.rs

@@ -1024,6 +1024,86 @@ pub async fn pg_execute_sql_multi(
     execute_sql_multi(state, sql, query_id).await
 }
 
+/// `pg_execute_sql_multi` for the SQL console: a single Postgres statement's
+/// rows arrive through `on_rows` in chunks while they stream in, and the reply
+/// carries none. The channel always ends with a `done` chunk, whatever the
+/// engine or outcome, so the window knows every row has landed.
+///
+/// `keep_in_store`: the rows go to the result store under `query_id` instead
+/// of through the channel, which then carries progress (Settings → Database →
+/// Stream results). The window reads them with `result_window`.
+#[tauri::command]
+pub async fn pg_execute_sql_stream(
+    state: State<'_, DbState>,
+    store: State<'_, crate::db::result_store::ResultStore>,
+    sql: String,
+    query_id: Option<String>,
+    on_rows: tauri::ipc::Channel<crate::db::RowChunk>,
+    keep_in_store: Option<bool>,
+) -> Result<Vec<crate::db::StatementOutcome>, String> {
+    let store_id = query_id.clone().filter(|_| keep_in_store == Some(true));
+    let spool = match store_id.as_deref() {
+        Some(id) => Some(store.create(id)?),
+        None => None,
+    };
+    let mut sink = crate::db::RowSink::with_spool(on_rows, spool);
+    let result = crate::db::execute_sql_script(state, sql, query_id, Some(&mut sink)).await;
+    // A streamed run already sent its end marker; anything else gets one now.
+    sink.end();
+    // Nothing streamed (a script, a write, an engine that answers in the
+    // reply) or the run failed: no stored result to keep.
+    if let Some(id) = store_id.as_deref() {
+        let failed = result.as_ref().map_or(true, |o| o.iter().any(|s| s.error.is_some()));
+        if !sink.started() || failed {
+            store.drop_result(id);
+        }
+    }
+    result
+}
+
+/// Rows `[start, start + count)` of a stored result, in its current order, as
+/// a JSON array of row arrays (sent as stored, never re-encoded).
+#[tauri::command]
+pub async fn result_window(
+    store: State<'_, crate::db::result_store::ResultStore>,
+    id: String,
+    start: usize,
+    count: usize,
+) -> Result<tauri::ipc::Response, String> {
+    let result = store.get(&id).ok_or("This result is no longer available. Run the query again.")?;
+    let text = tauri::async_runtime::spawn_blocking(move || result.read_window(start, count))
+        .await
+        .map_err(|e| e.to_string())??;
+    Ok(tauri::ipc::Response::new(text))
+}
+
+/// Sort a stored result by a column (typed by the column's database type), or
+/// back to the order it streamed in with `column: null`.
+#[tauri::command]
+pub async fn result_sort(
+    store: State<'_, crate::db::result_store::ResultStore>,
+    id: String,
+    column: Option<usize>,
+    desc: bool,
+) -> Result<(), String> {
+    let result = store.get(&id).ok_or("This result is no longer available. Run the query again.")?;
+    tauri::async_runtime::spawn_blocking(move || result.sort(column, desc))
+        .await
+        .map_err(|e| e.to_string())?
+}
+
+/// A line from the dev-only frame probe (src/lib/perf-probe.js) into the log.
+#[tauri::command]
+pub fn perf_log(line: String) {
+    log::info!("perf: {line}");
+}
+
+/// Forget a stored result (its tab ran again or closed); its file goes with it.
+#[tauri::command]
+pub fn result_drop(store: State<'_, crate::db::result_store::ResultStore>, id: String) {
+    store.drop_result(&id);
+}
+
 /// Execute SQL against an arbitrary saved connection without switching the
 /// global active connection. Used by Data Diff for cross-host comparisons.
 #[tauri::command]

src-tauri/src/db/clickhouse.rs

@@ -251,6 +251,7 @@ pub async fn get_column_structure(
                 column_default: default,
                 foreign_key: None,
                 fk_constraint_name: None,
+                is_primary_key: false,
                 comment,
             })
         })

src-tauri/src/db/duckdb.rs

@@ -338,6 +338,7 @@ pub async fn get_column_structure(handle: &DuckdbHandle, table: &str) -> Result<
                     column_default,
                     foreign_key: None,
                     fk_constraint_name: None,
+                    is_primary_key: false,
                     comment: None,
                 })
             })

src-tauri/src/db/mod.rs

@@ -15,6 +15,7 @@ pub mod redis;
 #[cfg(test)]
 mod dialect_matrix;
 mod query;
+pub mod result_store;
 mod schema;
 pub mod sql_util;
 pub mod wide_columns;
@@ -40,9 +41,9 @@ pub use insights::{
 pub use geo::{geo_features, geo_overview, GeoBbox, GeoFeatures, GeoOverview};
 pub use ssh_tunnel::TunnelState;
 pub use query::{
-    delete_table_row, delete_table_rows, execute_ddl, execute_sql, execute_sql_multi, execute_sql_on_conn,
+    delete_table_row, delete_table_rows, execute_ddl, execute_sql, execute_sql_multi, execute_sql_on_conn, execute_sql_script, StatementOutcome,
     fetch_cell_value, CellValueResult, get_column_stats, get_table_rows, count_table_rows, insert_table_row, update_table_cell, ping_connection,
-    ColumnStats, InsertRowResult, KeysetCursor, RowFilter, SortSpec, SqlResult, TableRows,
+    ColumnStats, InsertRowResult, KeysetCursor, RowChunk, RowFilter, RowSink, SortSpec, SqlResult, TableRows,
 };
 pub use schema::{
     list_schemas, list_tables, list_schemas_on_conn, list_tables_on_conn,

src-tauri/src/db/mssql.rs

@@ -351,6 +351,7 @@ pub async fn get_column_structure(handle: &MssqlHandle, schema: &str, table: &st
                 column_default,
                 foreign_key: None,
                 fk_constraint_name: None,
+                is_primary_key: false,
                 comment: None,
             })
         })

src-tauri/src/db/posthog.rs

@@ -289,6 +289,7 @@ pub async fn get_column_structure(config: &PosthogConfig, table: &str) -> Result
             column_default: None,
             foreign_key: None,
             fk_constraint_name: None,
+            is_primary_key: false,
             comment: None,
         })
         .collect())