Skip to content

Add RFC for Arrow Flight SQL endpoint - #63

Draft
nielspardon wants to merge 1 commit into
prestodb:mainfrom
nielspardon:RFC-0024-arrow-flight-sql-endpoint
Draft

Add RFC for Arrow Flight SQL endpoint#63
nielspardon wants to merge 1 commit into
prestodb:mainfrom
nielspardon:RFC-0024-arrow-flight-sql-endpoint

Conversation

@nielspardon

@nielspardon nielspardon commented Jul 2, 2026

Copy link
Copy Markdown

RFC-0024: Arrow Flight SQL endpoint for Presto

This PR proposes an optional Arrow Flight SQL endpoint for Presto. When enabled, the coordinator exposes an Apache Arrow Flight SQL service alongside the existing HTTP/REST client protocol, so clients can query Presto over Flight SQL — most notably via ADBC drivers — and receive results as native Apache Arrow record batches instead of row-oriented JSON.

The endpoint is off by default and purely additive: the REST client protocol, existing clients, the JDBC driver, and SQL grammar are all unchanged.

Why

  • Standards-based connectivity — any Flight SQL / ADBC client (Python, Go, Rust, C/C++, Java, R, JDBC-over-ADBC) can talk to Presto without a Presto-specific client library.
  • Columnar transport — Arrow avoids JSON (de)serialization and plugs directly into the Arrow ecosystem (pandas/Polars, DuckDB, Spark, ...).
  • Performance headroom — Flight (gRPC/HTTP2) supports partitioned, parallel data streams, giving a path to distribute result fetching across workers later.

This is the complement of RFC-0004: RFC-0004 makes Presto a Flight client of other systems; this makes Presto a Flight server for its own query results.

Key design decisions

  • Embedded in the coordinator as a separable optional module (gRPC/Arrow deps only loaded when enabled).
  • Phased data path: phase 1 translates result pages to Arrow at the coordinator (works for both Java and native); phase 2 streams Arrow directly from workers in parallel via the Velox Arrow bridge (deferred, but the design is forward-compatible).
  • Reuses Presto authentication & access control (password/LDAP, JWT/OAuth2, Kerberos, mTLS) — same identity, row filters, and column masks as REST.
  • Targets both Java Presto and Prestissimo (native).

Contents

Summary, background (goals/non-goals), proposed implementation (modules, Flight SQL action table, code flow + pseudocode, Presto↔Arrow type mapping, auth, phasing, config), metrics, alternatives considered, adoption plan, and test plan.

Feedback welcome — particularly on the deployment model (embedded vs gateway), the type-mapping table, and the phase-2 worker-streaming direction.

@nielspardon
nielspardon marked this pull request as draft July 2, 2026 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant