Milestones 1 and 2: scaffolding, codegen, oracle, and corpus goldens

.github/workflows/ci.yml

@@ -0,0 +1,43 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    env:
+      # The whole point: no cgo, anywhere, ever.
+      CGO_ENABLED: "0"
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+      - name: gofmt
+        run: test -z "$(gofmt -l . | grep -v '^internal/reference/' | grep -v '\.pb\.go$')"
+      - run: go vet ./...
+      - run: go build ./...
+      - name: go test -race
+        env:
+          CGO_ENABLED: "1" # -race requires cgo in the toolchain, not in our code
+        run: go test -race ./...
+      - name: go test (pure)
+        run: go test ./...
+
+  generate-clean:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+      - name: cmd/generate output is committed and current
+        run: |
+          go run ./cmd/generate -aliases -keywords
+          git diff --exit-code

.github/workflows/regenerate.yml

@@ -0,0 +1,31 @@
+name: regenerate
+
+# Scheduled proof that the committed goldens are reproducible byte-for-byte
+# from the pinned oracle (libpg_query 17-6.2.2 via pg_query_go v6.2.2, cgo).
+# This is the only place cgo ever runs; the main module never links it.
+
+on:
+  schedule:
+    - cron: "17 6 * * 1" # weekly, Monday 06:17 UTC
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  regenerate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+      - name: Fetch pinned corpora
+        run: |
+          git clone --depth 1 --branch 17-6.2.2 https://github.com/pganalyze/libpg_query /tmp/libpg_query
+          git clone --depth 1 --branch v6.2.2 https://github.com/pganalyze/pg_query_go /tmp/pg_query_go
+      - name: Regenerate goldens (builds the cgo oracle; several minutes)
+        run: |
+          go run ./cmd/regenerate -libpg-query /tmp/libpg_query -pg-query-go /tmp/pg_query_go
+      - name: Goldens are reproducible
+        run: git diff --exit-code -- parser/testdata

CLAUDE.md

@@ -0,0 +1,49 @@
+# oliphant development guide
+
+Pure Go port of pg_query_go. Read `PLAN.md` first — it is the contract.
+Pin: libpg_query **17-6.2.2** (PostgreSQL 17.7, patched). No cgo anywhere in
+the main module; the only module allowed to link pg_query_go (cgo) is
+`oracle/`, and only `cmd/regenerate`/`cmd/difftest` ever run it.
+
+## The development loop
+
+1. `go run ./cmd/next-test` — picks the next `todo` case from
+   `parser/testdata/*/`.
+2. Implement the production in `internal/parse/` (or lexer rule in
+   `internal/lexer/`), with an attribution comment naming the `gram.y` /
+   `scan.l` rule (`// gram.y: simple_select`).
+3. `go test ./parser -run TestCorpus -check-parse` — runs the `todo` cases;
+   newly passing cases are removed from the `metadata.json` sidecars
+   automatically. Commit the sidecar diff with the code.
+4. `go test ./...` must stay green; a case that regresses from passing to
+   failing is a hard failure, never a new todo.
+
+## Hard rules
+
+- **Expected outputs are never edited by hand.** Goldens come from
+  `cmd/regenerate` (the pinned oracle) or not at all.
+- Generated files (`ast/pg_query.pb.go`, `aliases.go`,
+  `internal/lexer/keywords.go`) are only ever written by `cmd/generate`.
+- The public API mirrors pg_query_go exactly; consumers must migrate by
+  changing one import path. Never rename, add, or remove exported symbols
+  without checking pg_query_go v6.2.2.
+- Location fields must byte-match the reference; when in doubt, check which
+  `@N` the `gram.y` action uses.
+
+## Layout crib
+
+- `internal/reference/` — pinned `gram.y`, `scan.l`, `parser.c`, `kwlist.h`
+  (docs only). The `base_yylex` token-merge filter is in `parser.c`.
+- `srcdata/` — vendored libpg_query metadata JSON + `pg_query.proto`
+  (generator inputs).
+- `parser/testdata/<suite>/*.test` — corpus: cases split by `==`, input and
+  expectation split by `--`. `metadata.json` sidecar per file tracks `todo`.
+- `oracle/` — separate go module, cgo, never imported by the main module.
+
+## Regenerating
+
+- `go run ./cmd/generate -aliases -keywords` — pure Go, always safe.
+- `go run ./cmd/generate -proto` — requires `protoc` + `protoc-gen-go`.
+- `go run ./cmd/regenerate -libpg-query <checkout> -pg-query-go <checkout>` —
+  builds the cgo oracle, rewrites `parser/testdata` from scratch. Output must
+  be byte-identical run-to-run.

LICENSE

@@ -0,0 +1,31 @@
+MIT License
+
+Copyright (c) 2026 The sqlc Authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+This repository additionally contains:
+
+- Logic ported from the PostgreSQL parser (grammar, scanner, keyword tables,
+  deparser): see LICENSE.POSTGRESQL.
+- Logic ported from pganalyze/libpg_query and pganalyze/pg_query_go
+  (API surface, fingerprinting, JSON output conventions, test corpora):
+  see LICENSE.LIBPG_QUERY.

LICENSE.LIBPG_QUERY

@@ -0,0 +1,29 @@
+Copyright (c) 2015, Lukas Fittl <lukas@fittl.com>
+Copyright (c) 2016-2023, Duboce Labs, Inc. (pganalyze) <team@pganalyze.com>
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+this list of conditions and the following disclaimer in the documentation
+and/or other materials provided with the distribution.
+
+* Neither the name of pg_query nor the names of its contributors may be used
+to endorse or promote products derived from this software without specific
+prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

LICENSE.POSTGRESQL

@@ -0,0 +1,28 @@
+This project includes code derived from the PostgreSQL project (http://www.postgresql.org/),
+see the original full copyright notice below:
+
+---
+
+PostgreSQL Database Management System
+(formerly known as Postgres, then as Postgres95)
+
+Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+
+Portions Copyright (c) 1994, The Regents of the University of California
+
+Permission to use, copy, modify, and distribute this software and its
+documentation for any purpose, without fee, and without a written agreement
+is hereby granted, provided that the above copyright notice and this
+paragraph and the following two paragraphs appear in all copies.
+
+IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR
+DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING
+LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS
+DOCUMENTATION, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES,
+INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
+AND FITNESS FOR A PARTICULAR PURPOSE.  THE SOFTWARE PROVIDED HEREUNDER IS
+ON AN "AS IS" BASIS, AND THE UNIVERSITY OF CALIFORNIA HAS NO OBLIGATIONS TO
+PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.

PLAN.md

@@ -359,17 +359,20 @@ oliphant/
 Ordered so that every milestone lands with its own oracle-backed acceptance
 gate; the corpus harness exists before the first line of the lexer.
 
-1. **Scaffolding + codegen.** Module, licenses, CI; vendor `srcdata/`,
-   `pg_query.proto`, reference grammar files at the pin; `cmd/generate`
-   producing `ast/pg_query.pb.go`, root aliases, keyword tables;
-   `oliphant.go` API stubs returning not-implemented; `makefuncs.go` and the
-   `parser.Error` type ported. **Gate:** pg_query_go's `parse_test.go`
-   expected-tree literals compile unchanged against oliphant's types.
-2. **Corpus + oracle.** `oracle/` module wrapping pg_query_go v6.2.2;
-   `cmd/regenerate` extracting all corpus tiers into `.test` goldens
-   (trees, errors, scan tokens, normalize, fingerprint, deparse);
-   `internal/testfile` reader; harness running everything as `todo`.
-   **Gate:** goldens reproducible byte-for-byte across two regeneration runs.
+1. **Scaffolding + codegen.** ✅ *Landed 2026-08-15.* Module, licenses, CI;
+   vendor `srcdata/`, `pg_query.proto`, reference grammar files at the pin;
+   `cmd/generate` producing `ast/pg_query.pb.go`, root aliases, keyword
+   tables; `oliphant.go` API stubs returning not-implemented; `makefuncs.go`
+   and the `parser.Error` type ported. **Gate:** pg_query_go's
+   `parse_test.go` expected-tree literals compile unchanged against
+   oliphant's types — met (`parse_test.go`, import swap only).
+2. **Corpus + oracle.** ✅ *Landed 2026-08-15.* `oracle/` module wrapping
+   pg_query_go v6.2.2; `cmd/regenerate` extracting the corpus tiers into
+   `.test` goldens (trees, errors, scan tokens, normalize, fingerprint,
+   deparse, splits); `internal/testfile` reader; harness running everything
+   as `todo`. **Gate:** goldens reproducible byte-for-byte across two
+   regeneration runs — met, and re-verified weekly by the `regenerate` CI
+   workflow. See "As-built notes" below for measured counts and deferrals.
 3. **Lexer.** `scan.l` port + `base_yylex` filter layer; `Scan`,
    `SplitWithScanner`, `HashXXH3_64` (+ `internal/xxh3`) ship. **Gate:**
    token streams byte-identical to the oracle across the entire corpus,
@@ -417,6 +420,34 @@ gate; the corpus harness exists before the first line of the lexer.
     `tetratelabs/wazero`, and the cgo requirement; sqlc's endtoend suite is
     the acceptance gate. Windows and `CGO_ENABLED=0` become first-class.
 
+### As-built notes (milestones 1–2)
+
+Measured at the pin, where the plan's estimates differ:
+
+- The corpus as regenerated: **1,525 `.test` files / 217,229 cases (~80 MB)**
+  across eight suites (`parse`, `scan`, `normalize`, `normalize_utility`,
+  `fingerprint`, `deparse`, `split_scanner`, `split_parser`).
+- `postgres_regress/` has **224** files at `17-6.2.2` (not 233); `kwlist.h`
+  yields **491** keywords (not 494); the proto has **273** messages and
+  **71** enums (not 276/73). The inline tables carry 36 parse, 12 scan,
+  25 normalize, 25 normalize_utility, 78 fingerprint, 418 deparse, and
+  8 split inputs; `deparse-depesz/` is 150 `.psql` files under `*.d/`
+  subdirectories.
+- The `.test` format needed one addition over the family convention: input
+  or expectation lines that collide with the `== ` / `--` markers are
+  escaped with a leading `|` (SQL comment banners of exactly `--` are
+  everywhere in the regress files). `internal/testfile` round-trips this
+  bijectively.
+- `cmd/regenerate` preserves passing status across runs: a case identical in
+  name, input, and expectation to one that had graduated out of `todo` stays
+  passing, so a pin advance puts exactly the diff back on the todo list.
+- `cmd/generate -proto` requires `protoc` + `protoc-gen-go` on PATH; the
+  pure-Go `-aliases`/`-keywords` modes are what CI verifies.
+- Deferred, deliberately: the **sqlc endtoend tier** (needs per-directory
+  engine classification in the sqlc repo; import it alongside milestone 4),
+  **`plpgsql_regress/`** (milestone 11), and **summary** golden extraction
+  (milestone 10). The stretch tarball tier remains stretch.
+
 ## Regeneration (the PostgreSQL-upgrade story)
 
 Everything derived is derived by committed tooling from the pin:

README.md

@@ -1 +1,34 @@
-# oliphant
+# oliphant
+
+A hand-written, pure Go (no cgo, no wasm) drop-in replacement for
+[pganalyze/pg_query_go](https://github.com/pganalyze/pg_query_go): same
+exported functions, same generated protobuf types, same JSON output, same
+error type, same fingerprints. Consumers change one import path:
+
+```diff
+-import pg_query "github.com/pganalyze/pg_query_go/v6"
++import pg_query "github.com/sqlc-dev/oliphant"
+```
+
+Pinned to **libpg_query `17-6.2.2`** (PostgreSQL 17.7, patched) — the exact
+build pg_query_go v6.2.2 ships. Every golden in `parser/testdata` derives
+from that oracle, byte for byte, and is never edited by hand.
+
+See [`PLAN.md`](PLAN.md) for the full design and [`CLAUDE.md`](CLAUDE.md)
+for the development loop.
+
+## Status
+
+| Milestone | State |
+|---|---|
+| 1. Scaffolding + codegen | ✅ module, licenses, CI, vendored pin, `ast/` protobuf types, aliases, keyword tables, API surface, `parser.Error`; pg_query_go's parse_test tree literals compile unchanged |
+| 2. Corpus + oracle | ✅ cgo oracle (`oracle/`), `cmd/regenerate`, 1,525 golden files / 217k cases across 8 suites, byte-reproducible; harness + `cmd/next-test` running everything as todo |
+| 3. Lexer | — |
+| 4. Expressions + SELECT | — |
+| 5–7. DML, DDL, utility | — |
+| 8. Normalize + fingerprint | — |
+| 9. Deparse | — |
+| 10–13. Summary, PL/pgSQL, hardening, sqlc integration | — |
+
+Until the parser milestones land, every public entry point returns (or
+panics with, for `HashXXH3_64`) a clear not-implemented error.