Skip to content

Support DuckDB syntax: FROM-first, SEMI/ANTI joins, COPY, ATTACH, MACRO, PRAGMA and DuckDB 2.0 additions - #2643

Open
hayssams wants to merge 19 commits into
JSQLParser:masterfrom
starlake-ai:feature/duckdb-syntax
Open

hayssams wants to merge 19 commits into
JSQLParser:masterfrom
starlake-ai:feature/duckdb-syntax

Conversation

@hayssams

Copy link
Copy Markdown
Contributor

Adds support for nineteen DuckDB constructs that JSqlParser rejected (or silently mis-parsed), including the new DuckDB 2.0 syntax. One commit per feature, each with tests that assert both the AST and a stable round-trip through toString() and the deparser.

Features, with the DuckDB documentation that specifies them

Feature Example DuckDB reference
GLOB pattern matching WHERE b GLOB 'y*' Pattern matching
Standalone SEMI / ANTI joins FROM t ANTI JOIN u ON t.id = u.id FROM clause
MAP(key, value) column type CREATE TABLE t (m MAP(VARCHAR, INTEGER)) MAP type
INSERT ... BY NAME / BY POSITION INSERT INTO t BY NAME SELECT 1 AS b, 2 AS a INSERT
USING SAMPLE sizes and methods USING SAMPLE 10 PERCENT (bernoulli) Samples
DESCRIBE <query> DESCRIBE SELECT * FROM t DESCRIBE
PRAGMA PRAGMA table_info('t'), PRAGMA memory_limit = '1GB' Pragmas
INSTALL / LOAD extensions FORCE INSTALL h3 FROM community, LOAD httpfs Extensions
ATTACH / DETACH ATTACH 'file.db' AS mydb (TYPE SQLITE, READ_ONLY) ATTACH/DETACH
PREPARE / DEALLOCATE PREPARE q AS SELECT * FROM t WHERE a = $1 Prepared statements
COPY (import and export) COPY (SELECT * FROM t) TO 'out.parquet' (FORMAT PARQUET) COPY
CREATE MACRO (scalar and table) CREATE MACRO add(a, b := 5) AS a + b CREATE MACRO
FROM-first queries FROM t SELECT a, FROM t WHERE a > 1 FROM clause
UNPIVOT statement form UNPIVOT monthly_sales ON jan, feb INTO NAME month VALUE sales UNPIVOT
2.0 CONNECT / DISCONNECT CONNECT 'postgres://localhost/mydb' A preview of DuckDB v2.0
2.0 CREATE TRIGGER ... REFERENCING OLD/NEW TABLE with a statement body CREATE TRIGGER t AFTER UPDATE ON x REFERENCING OLD TABLE AS o NEW TABLE AS n FOR EACH STATEMENT INSERT INTO ... A preview of DuckDB v2.0
2.0 CREATE EXTENSION REPOSITORY CREATE EXTENSION REPOSITORY my_repo WITH PREFIX 'https://…' A preview of DuckDB v2.0
2.0 APPROX NEAREST n BY SIMILARITY joins JOIN products t APPROX NEAREST 2 BY SIMILARITY array_cosine_similarity(q.vec, t.vec) A preview of DuckDB v2.0
2.0 recursive CTE USING KEY WITH RECURSIVE tbl (a, b) USING KEY (a, avg(b)) AS (...) A preview of DuckDB v2.0

Notes

  • ANTI JOIN previously parsed silently as a table alias, so valid SQL produced a wrong AST. Join.isAnti() now records it, and LEFT/RIGHT/FULL SEMI|ANTI are accepted too.
  • New Statement types (PRAGMA, INSTALL/LOAD, ATTACH/DETACH, CONNECT/DISCONNECT, PREPARE/DEALLOCATE, COPY, CREATE MACRO, CREATE EXTENSION REPOSITORY) are wired through StatementVisitor, StatementVisitorAdapter, StatementDeParser, TablesNamesFinder, StatementFeatureVisitor and StatementValidator. COPY, PREPARE and UNPIVOT expose their nested query to TablesNamesFinder.
  • FROM-first is represented as a PlainSelect with a fromFirst flag, so the clause order round-trips; SelectDeParser follows the same order. Bare FROM t queries and BigQuery pipe syntax are untouched.
  • UnPivotQuery mirrors the existing PivotQuery, added as default visitor methods so no third-party visitor breaks.
  • New keywords are non-reserved and stay usable as identifiers; regression tests cover glob, anti, load, install, attach, detach and pragma.

Test plan

  • New tests in DuckDBTest, each feature with a regression case for the syntax it could have shadowed (POSITION(...), MySQL ALGORITHM COPY, Oracle CONNECT BY, PostgreSQL CREATE EXTENSION, EXECUTE FUNCTION triggers, FROM-clause UNPIVOT (...), pipe queries).
  • Full suite: 7050 tests, 1 failure — SpecialOracleTest.testAllSqlsParseDeparse, which fails on an unmodified master in this environment too.
  • JavaCC still reports 0 errors and 1 warnings, the same warning as on master.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2

hayssams and others added 8 commits September 15, 2026 14:18
Adds GLOB as a LikeExpression keyword so `col GLOB 'pattern'` and
`col NOT GLOB 'pattern'` parse and deparse. GLOB stays a non-reserved
keyword and remains usable as an identifier.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
DuckDB allows SEMI/ANTI JOIN without a LEFT prefix, and ANTI as a join
type at all. ANTI previously parsed silently as a table alias, producing
a wrong AST from valid SQL. Adds Join.isAnti(), the LEFT/RIGHT/FULL
SEMI|ANTI prefixes and the standalone form, keeping ANTI usable as an
identifier outside a join prefix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
DuckDB parameterises a type with other types, e.g. MAP(VARCHAR, INTEGER).
Type arguments previously accepted identifiers and literals but not data
type tokens.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
DuckDB matches an INSERT's source columns either positionally or by
name. BY previously bound as a table alias, swallowing the clause.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
DuckDB writes the sample size first and the method behind it:
USING SAMPLE 10%, USING SAMPLE 10 PERCENT (bernoulli),
USING SAMPLE 10 ROWS (system, 377), USING SAMPLE reservoir (50 ROWS).
The method is now optional, RESERVOIR is a known method, and a bare
sample size accepts a unit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
DESCRIBE now accepts a query as well as a table, so DESCRIBE SELECT ...,
DESCRIBE FROM ... and DESCRIBE WITH ... parse. The described query is
reachable from TablesNamesFinder and rendered by the deparser.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds PRAGMA name, PRAGMA name(arguments) and PRAGMA name = value, as
used by DuckDB and SQLite to read and set database settings. PRAGMA
stays a non-reserved keyword.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds [FORCE] INSTALL extension [FROM repository] and LOAD extension,
where the extension and repository may be an identifier or a string
literal. INSTALL and LOAD stay non-reserved keywords.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
@manticore-projects

Copy link
Copy Markdown
Contributor

Run './gradlew spotlessApply' to fix all violations.

hayssams and others added 7 commits September 15, 2026 16:13
Adds ATTACH [DATABASE] [IF NOT EXISTS] path [AS alias] [(options)] and
DETACH [DATABASE] [IF EXISTS] name, with options such as READ_ONLY or
TYPE SQLITE kept as strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds CONNECT target [AS alias] and DISCONNECT [name], which route a
session's queries to a remote database. Oracle's CONNECT BY is
unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds PREPARE name AS statement and DEALLOCATE [PREPARE] name. EXECUTE of
a prepared statement already parsed. The prepared statement is a nested
Statement, so TablesNamesFinder reaches its tables.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds COPY table [(columns)] FROM path [(options)] and
COPY {table | (query)} TO path [(options)]. Options such as
FORMAT PARQUET or AUTO_DETECT true are free-form keywords, so they are
collected verbatim. COPY stays valid as MySQL's ALGORITHM value.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds CREATE [OR REPLACE] [TEMPORARY] MACRO name (parameters) AS body for
both scalar macros and table macros (AS TABLE SELECT ...). Parameters may
carry a default value, as in add(a, b := 5).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds CREATE EXTENSION REPOSITORY [IF NOT EXISTS] name [WITH PREFIX
prefix], which registers a custom repository to install extensions from.
PostgreSQL's CREATE EXTENSION is unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
DuckDB writes CREATE TRIGGER ... REFERENCING OLD TABLE AS o NEW TABLE AS
n FOR EACH STATEMENT <statement>, i.e. the standard header with a SQL
statement instead of EXECUTE FUNCTION. The standard-form parser now
accepts a statement body, and such triggers are recognised and rendered
as standard rather than MySQL-style triggers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
@hayssams
hayssams force-pushed the feature/duckdb-syntax branch from f74c137 to a765cf7 Compare September 15, 2026 14:14
@hayssams

Copy link
Copy Markdown
Contributor Author

@manticore-projects
Fixed
Thanks!

hayssams and others added 4 commits September 15, 2026 18:05
DuckDB lets the FROM clause lead, with the SELECT clause optional:
FROM tbl SELECT a and FROM tbl WHERE a > 1. PlainSelect records the
order in a fromFirst flag, and both the AST's own rendering and
SelectDeParser emit the clauses accordingly. Bare FROM queries and the
pipe syntax are unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds the simplified statement form UNPIVOT tbl ON columns
[INTO NAME name VALUE values] as UnPivotQuery, the counterpart of the
existing PivotQuery. The FROM-clause form UNPIVOT (...) is unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds JOIN t APPROX NEAREST n BY SIMILARITY expression, the vector
similarity join of DuckDB 2.0. APPROX no longer binds as a table alias
in that position. Also adds the lookahead the UNPIVOT statement's INTO
clause needed to keep the grammar free of choice conflicts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
Adds WITH RECURSIVE tbl (a, b) USING KEY (a, avg(b)) AS (...), which
aggregates a recursive CTE by key instead of accumulating every row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqinwHBKuAtPEmtXr3b5P2
@hayssams
hayssams force-pushed the feature/duckdb-syntax branch from a765cf7 to c04286d Compare September 15, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants