RFC: Apache Doris connector based on the Arrow Flight base module - #62
RFC: Apache Doris connector based on the Arrow Flight base module#62ningsh7 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 810c27275f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | --- | --- | --- | | ||
| | `BOOLEAN` | `BOOLEAN` | | | ||
| | `TINYINT` / `SMALLINT` / `INT` / `BIGINT` | `TINYINT` / `SMALLINT` / `INTEGER` / `BIGINT` | | | ||
| | `LARGEINT` | `DECIMAL(38,0)` | Open question: `VARCHAR` instead? | |
There was a problem hiding this comment.
Avoid mapping LARGEINT to DECIMAL(38,0)
For tables that contain LARGEINT values outside 38 decimal digits, this mapping cannot be sound: Doris LARGEINT is a 16-byte signed integer with range up to 2^127 - 1 (Doris docs), which has 39 digits, while Presto decimals max out at precision 38. Planning the connector around DECIMAL(38,0) means valid Doris rows can overflow or fail during Arrow-to-Presto conversion; the RFC should instead choose a lossless default such as VARCHAR/unsupported or explicitly define overflow behavior.
Useful? React with 👍 / 👎.
This RFC proposes a first-class Apache Doris connector for Presto, because Doris is currently reachable mainly through the MySQL connector path, which is row-serialized and single-stream.
It builds on RFC-0004 and extends
presto-base-arrow-flightso scans can use Flight SQL endpoints as parallel Presto splits instead of going through a JDBC-driver row path.The initial scope is read-only: metadata discovery, table scans, projection/filter/limit pushdown, authentication, TLS, and type mapping.
Prior art exists in trinodb/trino#29120 by the same proposer, demonstrating Flight-SQL-based reads from Doris end to end.
Reviewer input is especially requested on whether this should remain Doris-specific or evolve toward a generic Flight SQL connector layer later.
The RFC number was selected as RFC-0024 after checking the current upstream main sequence.