Skip to content

feat(qwp): add QwpQueryClient for QWP egress over WebSocket#11

Open
bluestreak01 wants to merge 7 commits intomainfrom
vi_egress
Open

feat(qwp): add QwpQueryClient for QWP egress over WebSocket#11
bluestreak01 wants to merge 7 commits intomainfrom
vi_egress

Conversation

@bluestreak01
Copy link
Copy Markdown
Member

Summary

  • Add QwpQueryClient, the client-side counterpart of the QWP egress endpoint that QuestDB's HTTP server now exposes at /read/v1. Sends a SQL query, receives result batches as binary WebSocket frames, decodes them into a column-major QwpColumnBatch view.
  • Connect-string entry point QwpQueryClient.fromConfig(\"ws::addr=localhost:9000;...\") mirrors the Sender.fromConfig shape used by the existing ingestion API.
  • Dedicated I/O thread reads + decodes ahead of the user thread; batches arrive via a bounded SPSC queue so decoding of batch N+1 overlaps with the user's processing of batch N.
  • Zero-allocation hot path: per-cell accessors read directly from native memory; STRING / VARCHAR / SYMBOL access returns a reusable DirectUtf8Sequence view into the underlying WebSocket payload buffer.
  • Four user-facing examples added under examples/com/example/query/.

API shape

try (QwpQueryClient client = QwpQueryClient.fromConfig(\"ws::addr=localhost:9000;\")) {
    client.connect();
    client.execute(\"SELECT ts, sym, price FROM trades LIMIT 1000\", new QwpColumnBatchHandler() {
        @Override public void onBatch(QwpColumnBatch batch) {
            for (int r = 0; r < batch.getRowCount(); r++) {
                long ts = batch.getLong(0, r);
                DirectUtf8Sequence sym = batch.getStrA(1, r);
                double px = batch.getDouble(2, r);
                // ... process row
            }
        }
        @Override public void onEnd(long totalRows) { ... }
        @Override public void onError(byte status, String message) { ... }
    });
}

QwpColumnBatch exposes both schema-agnostic accessors (getLong, getDouble, getString, isNull) and per-type raw-address APIs (valuesAddr, nonNullIndex, getLongValue, getIntValue) for callers writing the tightest possible inner loops.

Threading model

  • One I/O thread per QwpQueryClient, spawned on connect(), marked daemon.
  • I/O thread owns the WebSocketClient, the QwpResultBatchDecoder, and a small pool of QwpBatchBuffer instances (default 4, configurable via withBufferPoolSize).
  • Per-frame: take a buffer from the free pool, memcpy the WS payload into it, decode in place, push onto the events queue.
  • User thread drains events inside execute(), invokes the handler, releases the buffer back to the pool.
  • Back-pressure: when the buffer pool is exhausted (slow consumer), the I/O thread blocks on freeBuffers.take(), the WS recv buffer fills, and TCP flow control closes the server's send window.
  • close() sends Thread.interrupt() to the I/O thread, joins for up to 5 s, then frees the pool. If join times out, the daemon thread is left running and the buffer pool + WebSocket socket are leaked for the JVM lifetime rather than freed under an active reader; wasLastCloseTimedOut() surfaces the condition for callers.

Performance characteristics

  • After warmup the decode path allocates nothing on the JVM heap. Per-column native scratches (null bitmaps, dense values, string heaps, symbol-id arrays) grow to the maximum observed size and are reused across batches. Pooled DirectUtf8String views replace per-cell String and byte[] allocation.
  • Per-row dispatch in getLong still has a wire-type switch; callers in tight scan loops can use the type-specialised getLongValue / getIntValue accessors to skip it.
  • The per-batch memcpy from WS recv buffer into the buffer's owned native scratch is the cost of decoupling the I/O thread from the user thread. ~10-100 µs for typical batches; the alternative (zero-copy ringing of the WS buffer) is left to a future revision.

Limitations

  • Phase 1 sends one query at a time per connection; no client-side multiplexing of concurrent queries on a shared socket.
  • No bind-parameter encoding yet on the client. The egress server's QwpEgressRequestDecoder accepts binds for every scalar wire type, but QwpQueryClient.execute only takes a SQL string today.
  • No CREDIT flow control sent from the client. The server streams at line rate; back-pressure relies on TCP windowing rather than an explicit credit window.
  • No CANCEL frame sent from the client. A query runs to completion (or server error) once submitted.
  • TLS (wss::) connect-string is reserved but not implemented.
  • A QwpResultCursor row-iterator wrapper around the column-batch consumer API is on the Phase 2 backlog.

Test plan

  • 11 client-side codec unit tests (decoder + frame parser) pass.
  • The end-to-end behaviour is exercised by 24 server-side bootstrap tests in the parent QuestDB repository — see the matching parent PR.
  • Examples build cleanly under Java 8 source / Java 11 target.

🤖 Generated with Claude Code

@RaphDal
Copy link
Copy Markdown
Collaborator

RaphDal commented Apr 18, 2026

Fails
🚫

Please update the PR title to match this format:
type(subType): description

Where type is one of:
feat, fix, chore, docs, style, refactor, perf, test, ci, revert

And: subType is one of:
build, log, core, ilp, http, conf, utils

For Example:

perf(sql): improve pattern matching performance for SELECT sub-queries

Generated by 🚫 dangerJS against 4136fe2

@mtopolnik
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😞 fail : 203 / 754 (26.92%)

file detail

path covered line new line coverage
🔵 io/questdb/client/cutlass/qwp/client/QwpQueryClient.java 0 140 00.00%
🔵 io/questdb/client/cutlass/qwp/client/QwpColumnBatch.java 8 175 04.57%
🔵 io/questdb/client/cutlass/qwp/client/QwpEgressIoThread.java 13 133 09.77%
🔵 io/questdb/client/cutlass/qwp/client/QueryEvent.java 6 13 46.15%
🔵 io/questdb/client/cutlass/qwp/client/QwpResultBatchDecoder.java 136 248 54.84%
🔵 io/questdb/client/cutlass/qwp/client/QwpBatchBuffer.java 18 23 78.26%
🔵 io/questdb/client/cutlass/qwp/client/QwpDecodeException.java 2 2 100.00%
🔵 io/questdb/client/cutlass/qwp/client/QwpEgressColumnInfo.java 6 6 100.00%
🔵 io/questdb/client/cutlass/qwp/protocol/QwpConstants.java 4 4 100.00%
🔵 io/questdb/client/cutlass/qwp/client/QwpColumnLayout.java 10 10 100.00%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants