Repository navigation
Split Nova into web, query and proxy tiers, and remove the per-statement bottlenecks - #183
Merged
Merged
Conversation
… off the event loop One backend process served the application API, SQL and ML execution, and the MySQL proxy on a single event loop, so a heavy statement or a BI client slowed the whole application. NOVA_PROCESS_ROLE (all, web, query) now decides which startup duties a process runs. `all` is unchanged. A split deployment runs one web process (schema bootstrap and singleton loops), any number of query processes for /api/v1/query and /api/v1/ml, and `python -m app.proxy` for MySQL, behind a path-routing gateway. docker-compose.split.yml and `./dev.sh --split` run that topology; the default stack is untouched. Inside a process, statements of 8 KB or more are parsed on a worker thread with its own ANTLR prediction caches, and ML_PREDICT rows are built beside inference instead of on the loop. QUERY_MAX_CONCURRENCY can cap statements per process, refusing the excess with 429 and an audited REFUSED entry. LLM function registration takes a Redis lock so two processes cannot drop each other's functions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…M gateway, session-pinned agents Follow-up to the process split, closing the gaps found in review. Readiness: /ready answers 200 only after startup with the engine and Redis reachable. The split stack waits on it, and the gateway retries a replica that refuses a connection, for requests that are safe to repeat. Load: QUERY_MAX_CONCURRENCY now defaults to 64 per process and covers query execute and explain, the ML execution routes and MySQL proxy statements outside a transaction. Refusals are audited and counted. SQL CPU: the whole prepare stage of a large statement (normalise, guard, build, pre-check, plan, fingerprint) runs on the worker pool, whose threads use their own parser caches for any nested parse. Redaction is remembered within a request, and fingerprinting no longer rescans the token stream per alias. Worst event-loop stall for a 114 KB statement went from about 1.4 s to about 0.1 s. AI functions: AI_* bodies call an internal Nova gateway with a per-alias token instead of embedding the provider key. The gateway adds the key, fixes the model, and records token usage in USAGE_LLM_FUNCTIONS, shown in AI monitoring. Matching functions are left alone at startup. Scale-out: the gateway uses upstream groups that follow Docker DNS, pins assistant and agent routes to one query replica per session (their consent and cancel state is process-local), balances MySQL across proxy replicas, and can serve HTTPS. Schema bootstrap is serialised across web replicas. Prometheus discovers replicas through DNS in a split stack, and check_stack gains --topology split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With ML_TRAINING_EXECUTOR=worker, CREATE ML_MODEL and POST /ml/train queue
a job in CONFIG_ML_JOBS and wait for nova-worker, so no API process holds a
training run. The response contract is unchanged.
A job stores no credential: it names the caller's session, and the worker
trains as that user only while the same session is live with the same
principal, active role and security-context version. Otherwise the job
fails as session_expired; there is no fallback identity. The training SQL
is stored encrypted. A job no worker claims is withdrawn, a caller that
goes away cancels its job, and GET /ml/jobs/{id} shows status to the
session that queued it. Callers without a session still train locally.
Also documents the hardened split topology, and renames the new AI
monitoring source label so it no longer matches the existing "AI
Functions" option.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t works Both found by running the split stack end to end against a real engine. Audit: every statement wrote its audit row with its own INSERT. One such INSERT takes StarRocks about 100 ms and concurrent ones do not overlap (about 58 a second measured), so throughput stayed near 60 statements a second however many query replicas ran. Rows that arrive while a write is in flight now share the next INSERT. Each caller still waits for the write holding its own row, a lone row is written exactly as before, a refused group is rewritten row by row, and an uncertain failure is not retried. One query replica went from about 60 to about 135 statements a second at 48 concurrent clients. Gateway: `hash $http_authorization consistent` did not pin anything. The consistent variant places peers by server name, and every replica shares one DNS name, so sessions were balanced round robin and a consent decision reached a replica that was not running the turn. The plain hash indexes the resolved addresses; ten of ten requests per session now reach one replica, and an assistant turn with a consent prompt completes through the gateway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
One backend process served the application API, SQL and ML execution, agent runs and the MySQL proxy on a single event loop. A heavy statement or a BI client slowed the whole application, and nothing could be scaled on its own.
What changes
Process tiers.
NOVA_PROCESS_ROLE(all,web,query) decides which startup duties a process runs.allis the default and behaves as before. A split deployment runs web, query and MySQL proxy tiers behind an nginx gateway (docker/docker-compose.split.yml,./dev.sh --split). The default stack is untouched.Routing. SQL and ML go to the query tier. Assistant and agent routes go to one query replica per session, because their consent and cancel state is process-local.
/api/v1/internal/is not routed publicly. The gateway balances MySQL across proxy replicas and can serve HTTPS.SQL CPU off the event loop. The prepare stage of a large statement (normalise, guard, parse, build, plan, fingerprint) runs on a worker pool whose threads use their own parser caches. Redaction is remembered within a request.
Audit throughput. Audit rows that arrive together share one INSERT. Each caller still waits for its own row.
AI functions.
AI_*bodies call an internal Nova gateway with a per-alias token instead of embedding the provider key. Token usage is recorded and shown in AI monitoring.ML training on the worker. With
ML_TRAINING_EXECUTOR=worker, training is queued fornova-worker, bound to the caller's session. No credential is stored and there is no fallback identity.Operations.
/ready, compose healthchecks, a default load cap of 64 statements per process with audited refusals, schema bootstrap serialised across web replicas, and Prometheus discovery of replicas.Measured
Local machine, real engine, split stack with two web, two query and two proxy replicas.
/healthp99 while the query tier is saturatedThe remaining ceiling for many small statements is the engine's small-INSERT latency (about 100 ms), not Nova. Per-statement latency stays near 0.2 s because every statement waits for its audit row; lowering that means asynchronous audit, which is deliberately not done here.
Verification
backend/.envcarries dev keys).AI_COMPLETEthrough the gateway with usage recorded, training through the worker andML_PREDICT, an assistant turn with a consent prompt, replica failover, HTTPS, and Prometheus discovery of every replica.Known limits
AI_*usage is recorded per function and alias, not per user: the engine does not say who ran the SQL.algorithm = autowith the balanced ML budget uses the whole 60 s budget and can expire during registration. This predates the change../dev.sh --splititself was not run, to avoid starting a second scheduler and worker against a live development stack; its processes and the Vite routing rule were exercised separately.🤖 Generated with Claude Code