docs: update README and CLAUDE.md for full parser + query engine

Author: renecannao

CLAUDE.md

@@ -4,12 +4,12 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-High-performance hand-written recursive descent SQL parser for ProxySQL. Supports MySQL and PostgreSQL dialects via compile-time templating (`Parser<Dialect::MySQL>` / `Parser<Dialect::PostgreSQL>`). Designed for sub-microsecond latency on the proxy hot path.
+High-performance hand-written recursive descent SQL parser and query engine for ProxySQL. Supports MySQL and PostgreSQL dialects via compile-time templating (`Parser<Dialect::MySQL>` / `Parser<Dialect::PostgreSQL>`). Designed for sub-microsecond latency on the proxy hot path. The query engine takes parsed ASTs and executes them through a Volcano-model operator pipeline.
 
 ## Build Commands
 
 ```bash
-make -f Makefile.new all           # Build library + run all 430 tests
+make -f Makefile.new all           # Build library + run all 871 tests
 make -f Makefile.new lib           # Build only libsqlparser.a
 make -f Makefile.new test          # Build + run tests
 make -f Makefile.new bench         # Build + run benchmarks
@@ -22,7 +22,7 @@ For release benchmarks: `sed 's/-g -O2/-O3/' Makefile.new > /tmp/Makefile.releas
 
 **Note:** The old `Makefile` (no `.new`) is for the legacy Flex/Bison parser — do not use it for new code.
 
-## Architecture
+## Parser Architecture
 
 ### Three-layer pipeline
 
@@ -68,18 +68,99 @@ Everything is in `namespace sql_parser`. All templates are parameterized on `Dia
 - Digest mode (`EmitMode::DIGEST`): literals→`?`, IN collapsing, keyword uppercasing
 - Bindings mode: materializes `?` placeholders with bound parameter values
 
-### Tests
+## Query Engine
 
-Google Test. 430 tests across 16 test files. Validated against 86K+ external queries (PostgreSQL regression, MySQL MTR, CockroachDB, Vitess, TiDB, sqlparser-rs, SQLGlot).
+### Architecture
+
+The engine follows a five-component pipeline:
+
+1. **Type System** (`types.h`, `value.h`) — `SqlType` describes column types (30+ kinds). `Value` is a 14-tag discriminated union for runtime values (null, bool, int64, uint64, double, decimal, string, bytes, date, time, datetime, timestamp, interval, json).
+2. **Expression Evaluator** (`expression_eval.h`) — Recursively evaluates AST expression nodes against a row. Handles arithmetic, comparisons, boolean logic (three-valued), BETWEEN, IN, LIKE, CASE/WHEN, function calls. Uses `CoercionRules<D>` for type promotion and `null_semantics` for NULL propagation.
+3. **Catalog** (`catalog.h`, `in_memory_catalog.h`) — Abstract interface for table/column metadata. `InMemoryCatalog` is the hash-map implementation. `CatalogResolver` creates column-resolve callbacks from catalog + table + row.
+4. **Plan Builder** (`plan_builder.h`) — Translates a SELECT AST into a `PlanNode` tree. Translation order: FROM (Scan/Join) → WHERE (Filter) → GROUP BY (Aggregate) → HAVING (Filter) → SELECT list (Project) → DISTINCT → ORDER BY (Sort) → LIMIT.
+5. **Executor** (`plan_executor.h`) — Converts a `PlanNode` tree into an `Operator` tree (Volcano model: open/next/close). Pulls rows through the tree and collects them into a `ResultSet`.
+
+### Key types
+
+- `Value` — 14-tag discriminated union. Constructors: `value_null()`, `value_int(i)`, `value_string(s)`, etc.
+- `SqlType` — Column type descriptor with `Kind` enum, precision, scale, unsigned flag, timezone flag.
+- `Row` — `{Value* values, uint16_t column_count}`. Arena-allocated via `make_row()`.
+- `PlanNode` — Arena-allocated union node. Types: SCAN, FILTER, PROJECT, JOIN, AGGREGATE, SORT, LIMIT, DISTINCT, SET_OP.
+- `Operator` — Abstract base class with `open()`, `next(Row&)`, `close()`. Nine implementations in `operators/`.
+- `ResultSet` — `{vector<Row> rows, vector<string> column_names, uint16_t column_count}`.
+
+### How to add a new operator
+
+1. Create `include/sql_engine/operators/xxx_op.h` — implement `Operator` (open/next/close)
+2. Add a new `PlanNodeType` enum value in `plan_node.h` and a corresponding union member in `PlanNode`
+3. Add `build_xxx()` in `PlanExecutor` (in `plan_executor.h`) and wire it into `build_operator()` switch
+4. Add translation logic in `PlanBuilder::build_select()` (in `plan_builder.h`)
+5. Include the new header in `plan_executor.h`
+6. Write tests in `tests/test_operators.cpp` or a new test file
+
+### How to add a new SQL function
+
+1. Write the function in the appropriate file under `include/sql_engine/functions/` (arithmetic.h, string.h, comparison.h, or cast.h). Signature: `Value fn(const Value* args, uint16_t arg_count, Arena& arena)`.
+2. Register it in `src/sql_engine/function_registry.cpp` inside `register_builtins()`:
+   ```cpp
+   register_function({"MYFUNC", 6, my_func_impl, 1, 2});
+   //                  name, name_len, impl, min_args, max_args (255=variadic)
+   ```
+3. Write tests in `tests/test_string_funcs.cpp`, `tests/test_arithmetic.cpp`, or `tests/test_registry.cpp`
+
+### How to implement a custom DataSource
+
+Implement the `DataSource` interface in `data_source.h`:
+
+```cpp
+class MySource : public DataSource {
+    const TableInfo* table_info() const override;  // return table metadata
+    void open() override;                           // initialize cursor
+    bool next(Row& out) override;                   // fill row, return false when done
+    void close() override;                          // release resources
+};
+```
+
+Then register it with the executor: `executor.add_data_source("table_name", &my_source);`
+
+### How to implement a custom Catalog
+
+Implement the `Catalog` interface in `catalog.h`:
+
+```cpp
+class MyCatalog : public Catalog {
+    const TableInfo* get_table(StringRef name) const override;
+    const TableInfo* get_table(StringRef schema, StringRef table) const override;
+    const ColumnInfo* get_column(const TableInfo* table, StringRef column_name) const override;
+};
+```
+
+`TableInfo` must outlive any queries that reference it. `ColumnInfo::ordinal` must match the column position in rows returned by the corresponding `DataSource`.
+
+### Engine namespace
+
+Everything is in `namespace sql_engine`. Templates are parameterized on `Dialect D` where dialect-specific behavior applies (coercion rules, `||` semantics, LIKE matching).
+
+## Tests
+
+Google Test. 871 tests across 30 test files. Validated against 86K+ external queries (PostgreSQL regression, MySQL MTR, CockroachDB, Vitess, TiDB, sqlparser-rs, SQLGlot).
 
 Run a single test: `./run_tests --gtest_filter="*SetTest*"`
 
-### Benchmarks
+### Test files by component
+
+**Parser:**
+`test_tokenizer.cpp`, `test_classifier.cpp`, `test_expression.cpp`, `test_select.cpp`, `test_insert.cpp`, `test_update.cpp`, `test_delete.cpp`, `test_set.cpp`, `test_compound.cpp`, `test_emitter.cpp`, `test_digest.cpp`, `test_stmt_cache.cpp`, `test_arena.cpp`, `test_misc_stmts.cpp`
+
+**Engine:**
+`test_value.cpp`, `test_row.cpp`, `test_coercion.cpp`, `test_null_semantics.cpp`, `test_like.cpp`, `test_expression_eval.cpp`, `test_eval_integration.cpp`, `test_catalog.cpp`, `test_registry.cpp`, `test_arithmetic.cpp`, `test_comparison.cpp`, `test_cast.cpp`, `test_string_funcs.cpp`, `test_operators.cpp`, `test_plan_builder.cpp`, `test_plan_executor.cpp`
+
+## Benchmarks
 
 Google Benchmark. 18 single-thread + 16 multi-thread + 4 percentile benchmarks.
 Comparison benchmarks against libpg_query and sqlparser-rs in `bench/bench_comparison.cpp`.
 
-### Corpus testing
+## Corpus testing
 
 `corpus_test` binary reads SQL from stdin (one per line), parses each, reports OK/PARTIAL/ERROR counts.
 Usage: `./corpus_test mysql < queries.sql` or `./corpus_test pgsql < queries.sql`

README.md

@@ -1,10 +1,10 @@
-# ParserSQL
+# ParserSQL — High-Performance SQL Parser & Query Engine
 
-A high-performance, hand-written recursive descent SQL parser for [ProxySQL](https://github.com/sysown/proxysql). Supports both MySQL and PostgreSQL dialects with compile-time dispatch — zero runtime overhead for dialect selection.
+A high-performance, hand-written recursive descent SQL parser and composable query engine for [ProxySQL](https://github.com/sysown/proxysql). Supports both MySQL and PostgreSQL dialects with compile-time dispatch — zero runtime overhead for dialect selection. The parser produces an AST that feeds directly into the query engine's plan builder and executor pipeline.
 
 ## Performance
 
-All operations run in sub-microsecond latency on modern hardware:
+All parser operations run in sub-microsecond latency on modern hardware:
 
 | Operation | Latency | Notes |
 |---|---|---|
@@ -26,19 +26,10 @@ Compared to other parsers on the same queries:
 
 See [docs/benchmarks/](docs/benchmarks/) for full results and [REPRODUCING.md](docs/benchmarks/REPRODUCING.md) for reproduction instructions.
 
-## Features
-
-- **Deep parsing (Tier 1):** SELECT, INSERT, UPDATE, DELETE, SET, REPLACE, EXPLAIN, CALL, DO, LOAD DATA
-- **Compound queries:** UNION / INTERSECT / EXCEPT with SQL-standard precedence and parenthesized nesting
-- **Tier 2 classification:** All other statement types (DDL, transactions, SHOW, GRANT, etc.)
-- **Query reconstruction:** Parse → modify AST → emit valid SQL
-- **Query digest:** Normalize queries for fingerprinting (literals → `?`, IN list collapsing, keyword uppercasing) with 64-bit FNV-1a hash
-- **Prepared statement cache:** LRU cache with `parse_and_cache()` / `execute()` for binary protocol support
-- **Both dialects:** MySQL and PostgreSQL via `Parser<Dialect::MySQL>` / `Parser<Dialect::PostgreSQL>`
-- **Thread-safe:** One parser instance per thread, zero shared state, no locks
-
 ## Quick Start
 
+### Parse and emit SQL
+
 ```cpp
 #include "sql_parser/parser.h"
 #include "sql_parser/emitter.h"
@@ -66,30 +57,86 @@ DigestResult dr = digest.compute(result.ast);
 // dr.normalized = "SELECT * FROM users WHERE id = ?"
 // dr.hash = 0x... (64-bit FNV-1a)
 
-// Modify AST and re-emit
-// ... modify nodes ...
-// Emitter emitter2(parser.arena());
-// emitter2.emit(result.ast);  // emits modified SQL
-
 // Reset arena after each query (O(1), reuses memory)
 parser.reset();
 ```
 
-## Building
+### Full pipeline: parse, plan, execute
 
-```bash
-# Build library + run tests
-make -f Makefile.new all
+```cpp
+#include "sql_parser/parser.h"
+#include "sql_engine/plan_builder.h"
+#include "sql_engine/plan_executor.h"
+#include "sql_engine/in_memory_catalog.h"
+#include "sql_engine/data_source.h"
 
-# Build and run benchmarks
-make -f Makefile.new bench
+using namespace sql_parser;
+using namespace sql_engine;
+
+// 1. Set up catalog (table metadata)
+InMemoryCatalog catalog;
+catalog.add_table("", "users", {
+    {"id",   SqlType::make_int(),        false},
+    {"name", SqlType::make_varchar(255), true},
+    {"age",  SqlType::make_int(),        true},
+});
+
+// 2. Populate data source
+Arena data_arena{65536, 1048576};
+std::vector<Row> rows = {
+    // ... build rows with make_row() + value_int(), value_string(), etc.
+};
+const TableInfo* table = catalog.get_table(StringRef{"users", 5});
+InMemoryDataSource source(table, std::move(rows));
+
+// 3. Register built-in functions (UPPER, LOWER, COALESCE, etc.)
+FunctionRegistry<Dialect::MySQL> functions;
+functions.register_builtins();
+
+// 4. Parse SQL
+Parser<Dialect::MySQL> parser;
+auto result = parser.parse("SELECT name, age FROM users WHERE age > 21", 43);
 
-# Build comparison benchmarks (requires libpg_query)
-cd third_party/libpg_query && make && cd ../..
-make -f Makefile.new bench-compare
+// 5. Build logical plan (AST -> plan tree)
+PlanBuilder<Dialect::MySQL> builder(catalog, parser.arena());
+PlanNode* plan = builder.build(result.ast);
+
+// 6. Execute plan
+PlanExecutor<Dialect::MySQL> executor(functions, catalog, parser.arena());
+executor.add_data_source("users", &source);
+ResultSet rs = executor.execute(plan);
+
+// 7. Read results
+for (size_t i = 0; i < rs.row_count(); ++i) {
+    Row& row = rs.rows[i];
+    // row.get(0) = name (Value), row.get(1) = age (Value)
+}
+
+parser.reset();
 ```
 
-Requires: `g++` or `clang++` with C++17 support. No external dependencies for the parser itself. Google Test and Google Benchmark are vendored in `third_party/`.
+## Features
+
+### Parser
+
+- **Deep parsing (Tier 1):** SELECT, INSERT, UPDATE, DELETE, SET, REPLACE, EXPLAIN, CALL, DO, LOAD DATA
+- **Compound queries:** UNION / INTERSECT / EXCEPT with SQL-standard precedence and parenthesized nesting
+- **Tier 2 classification:** All other statement types (DDL, transactions, SHOW, GRANT, etc.)
+- **Query reconstruction:** Parse → modify AST → emit valid SQL
+- **Query digest:** Normalize queries for fingerprinting (literals → `?`, IN list collapsing, keyword uppercasing) with 64-bit FNV-1a hash
+- **Prepared statement cache:** LRU cache with `parse_and_cache()` / `execute()` for binary protocol support
+- **Both dialects:** MySQL and PostgreSQL via `Parser<Dialect::MySQL>` / `Parser<Dialect::PostgreSQL>`
+- **Thread-safe:** One parser instance per thread, zero shared state, no locks
+
+### Query Engine
+
+- **Type system:** 30+ SQL types (`SqlType::Kind`) with 14-tag runtime `Value` (null, bool, int64, uint64, double, decimal, string, bytes, date, time, datetime, timestamp, interval, json)
+- **Expression evaluator:** Recursive AST evaluator with three-valued logic (NULL propagation), type coercion, short-circuit AND/OR, BETWEEN, IN, LIKE, CASE/WHEN, IS [NOT] NULL
+- **Catalog:** Abstract `Catalog` interface + `InMemoryCatalog` implementation for table/column metadata resolution
+- **Plan builder:** Translates parsed SELECT AST into a logical plan tree (FROM → WHERE → GROUP BY → HAVING → SELECT → DISTINCT → ORDER BY → LIMIT)
+- **Executor with 9 operators:** Scan, Filter, Project, NestedLoopJoin (INNER/LEFT/RIGHT/FULL/CROSS), Aggregate (COUNT/SUM/AVG/MIN/MAX), Sort, Limit, Distinct, SetOp (UNION/INTERSECT/EXCEPT)
+- **38 built-in functions:** Arithmetic (ABS, CEIL, FLOOR, ROUND, MOD, POWER, SQRT, LOG, LN, EXP, SIGN, TRUNCATE, RAND, GREATEST, LEAST), string (UPPER, LOWER, LENGTH, CONCAT, SUBSTRING, TRIM, LTRIM, RTRIM, REPLACE, REVERSE, LEFT, RIGHT, LPAD, RPAD, REPEAT), comparison (COALESCE, NULLIF, IF, IFNULL), type (CAST)
+- **Composable data sources:** Implement `DataSource` interface for custom storage backends
 
 ## Architecture
 
@@ -110,24 +157,36 @@ Input SQL bytes
        ├──── Tier 1 ──► Deep parser (SELECT, INSERT, UPDATE, DELETE, SET, ...)
        │                  │
        │                  ▼
-       │                Full AST in arena
-       │
-       └──── Tier 2 ──► Lightweight extractor
-                          │
-                          ▼
-                        StmtType + table name + metadata
+       │                Full AST in arena ──┐
+       │                                    │
+       └──── Tier 2 ──► Lightweight         │
+                         extractor           │
+                                             ▼
+                                      ┌──────────────┐
+                                      │ Plan Builder  │  AST → logical plan tree
+                                      └──────┬───────┘
+                                             │
+                                             ▼
+                                      ┌──────────────┐
+                                      │ Plan Executor │  Volcano-model iterator
+                                      │  (Operators)  │  open() / next() / close()
+                                      └──────┬───────┘
+                                             │
+                                             ▼
+                                         ResultSet
 ```
 
 **Key design decisions:**
-- **Arena allocator** — 64KB bump allocator, O(1) reset. All AST nodes allocated from arena. No per-node new/delete.
+- **Arena allocator** — 64KB bump allocator, O(1) reset. All AST nodes and plan nodes allocated from arena. No per-node new/delete.
 - **Zero-copy StringRef** — Token values point into original input buffer. No string copies during parsing.
 - **32-byte AstNode** — Compact intrusive linked-list (first_child + next_sibling). Half a cache line.
 - **Compile-time dialect dispatch** — `if constexpr` for MySQL vs PostgreSQL differences. Zero runtime overhead.
 - **Header-only parsers** — Maximum inlining opportunity. Only `arena.cpp` and `parser.cpp` are compiled separately.
+- **Volcano execution model** — Operators implement open/next/close; rows are pulled through the tree one at a time.
 
 ## Testing
 
-430 unit tests + validated against 86K+ queries from 9 external corpora:
+871 unit tests + validated against 86K+ queries from 9 external corpora:
 
 | Corpus | Queries | OK Rate |
 |---|---|---|
@@ -165,14 +224,71 @@ include/sql_parser/
     keywords_mysql.h      MySQL keyword table
     keywords_pgsql.h      PostgreSQL keyword table
 
+include/sql_engine/
+    types.h               SqlType with 30+ SQL type kinds
+    value.h               Tagged-union Value (14 tags) + constructors
+    row.h                 Row: array of Value indexed by ordinal
+    catalog.h             Abstract Catalog interface (TableInfo, ColumnInfo)
+    in_memory_catalog.h   Hash-map Catalog implementation
+    catalog_resolver.h    Column resolver callback factory
+    data_source.h         DataSource interface + InMemoryDataSource
+    expression_eval.h     Recursive AST expression evaluator
+    function_registry.h   FunctionRegistry<D> with register_builtins()
+    plan_node.h           PlanNode union (9 node types)
+    plan_builder.h        AST → logical plan translation
+    plan_executor.h       Plan → operator tree → ResultSet
+    operator.h            Abstract Operator base (open/next/close)
+    result_set.h          ResultSet: rows + column names
+    coercion.h            Type coercion rules (dialect-specific)
+    null_semantics.h      Three-valued logic (AND/OR/NOT with NULL)
+    like.h                LIKE pattern matching
+    tag_kind_map.h        Value::Tag ↔ SqlType::Kind mapping
+
+    operators/
+        scan_op.h         Table scan from DataSource
+        filter_op.h       WHERE/HAVING predicate evaluation
+        project_op.h      SELECT expression list evaluation
+        join_op.h         Nested-loop join (5 join types)
+        aggregate_op.h    GROUP BY + COUNT/SUM/AVG/MIN/MAX
+        sort_op.h         ORDER BY (in-memory sort)
+        limit_op.h        LIMIT + OFFSET
+        distinct_op.h     Duplicate elimination
+        set_op_op.h       UNION/INTERSECT/EXCEPT
+
+    functions/
+        arithmetic.h      ABS, CEIL, FLOOR, ROUND, MOD, POWER, SQRT, ...
+        comparison.h      COALESCE, NULLIF, IF, IFNULL
+        string.h          UPPER, LOWER, LENGTH, CONCAT, SUBSTRING, TRIM, ...
+        cast.h            CAST type conversion
+
 src/sql_parser/
     arena.cpp             Arena implementation
     parser.cpp            Classifier + integration
 
-tests/                    430 unit tests (Google Test)
+src/sql_engine/
+    function_registry.cpp Built-in function registration
+    in_memory_catalog.cpp InMemoryCatalog implementation
+
+tests/                    871 unit tests (Google Test)
 bench/                    18 benchmarks + comparison suite
 ```
 
+## Building
+
+```bash
+# Build library + run tests
+make -f Makefile.new all
+
+# Build and run benchmarks
+make -f Makefile.new bench
+
+# Build comparison benchmarks (requires libpg_query)
+cd third_party/libpg_query && make && cd ../..
+make -f Makefile.new bench-compare
+```
+
+Requires: `g++` or `clang++` with C++17 support. No external dependencies for the parser itself. Google Test and Google Benchmark are vendored in `third_party/`.
+
 ## License
 
 See [LICENSE](LICENSE) file.