Go codegen for ClickHouse, DuckDB, Spanner and SQL Server under database/sql (#4621)

* plugin: carry the analysis core's type expression on Column

The core analyzer types every column and parameter as an expression: a
name, whether a value may be null, and nested arguments, so
decimal(10, 2), array(array(integer)) and struct(x: integer) survive
intact. The plugin request flattened that to a single type name plus
not_null, is_array, array_dims, length and unsigned, which is not enough
for a type mapper to tell nvarchar(max) from nvarchar(50) or to see the
element of an array or the fields of a struct.

Add a recursive TypeExpr message to the plugin proto and a type_expr
field on Column, populated from the core's tree in the shim. It is only
set when the analysis core typed the column; the legacy path leaves it
unset and the flat fields stay as they were, so existing plugins keep
working. When a database analyzer overrides a column's flat type in
combineAnalysis, the tree no longer describes the column and is dropped.

A DuckDB codegen_json case pins the shape with decimal arguments, nested
arrays, a labeled struct, a map and a sized array. The existing JSON
request goldens gain the new field, unset, on every column.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* codegen/golang: map ClickHouse, DuckDB, Spanner and SQL Server types

The Go generator ran for the four engines the analysis core handles but
knew none of their types, so every field came out as any. Add a mapper
per engine that reads the column's type expression rather than the flat
name, which is what lets it see the element of an array, the key and
value of a map, the fields of a struct, and the type LowCardinality or
SimpleAggregateFunction wraps. Each maps to the Go types the engine's
database/sql driver hands back: shopspring decimals, orb geometries and
pointer-typed nullable elements for clickhouse-go; duckdb.Composite,
duckdb.Decimal, duckdb.Interval and duckdb.Map for duckdb-go, with a
list parameter typed as the slice itself since a Composite is not
accepted as an argument; the spanner package's Null types for
go-sql-spanner, whose arrays decode into them; and
mssql.UniqueIdentifier for go-mssqldb.

The tree-reading mappers render the whole type, arrays included, so
goType leaves the dimensions to them. Model columns come from the
catalog, which the core path flattened, so catalog.Column now carries the
expression too and the shim forwards it. The queries for these engines
pass slices straight to the driver, so pq.Array wrapping is limited to
PostgreSQL under database/sql, where lib/pq needs it. Import detection
follows a package qualifier into map keys and values and through any
number of slice and pointer prefixes.

A core_types case per engine generates the analyze_types schema and
queries with database/sql and pins the output; the end-to-end module
gains the drivers so the generated code compiles.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* codegen/golang: bind named placeholders with sql.Named, and run the new engines' examples

SQL Server and Spanner name a parameter @name in the query text and
ClickHouse's server-side placeholder is {name:Type}; their drivers take
such an argument as sql.Named rather than by position. When every
parameter of a query is named that way, pass each as sql.Named with the
name the query uses. A query written with ? keeps positional arguments.

Add an authors example for ClickHouse, SQL Server and Spanner beside the
existing ones, and one for DuckDB in a module of its own, since its
driver is cgo and the main module's builds keep cgo off. Test helpers
create a database per test on the server CLICKHOUSE_SERVER_URI,
MSSQL_SERVER_URI or SPANNER_SERVER_URI names and skip when none is; the
DuckDB example opens a database in memory. CI runs ClickHouse and SQL
Server as services, builds the end-to-end module with cgo for the DuckDB
driver, and tests the DuckDB module; docker-compose gains a ClickHouse
service beside the SQL Server and Spanner Omni ones for local runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* ci: give the ClickHouse service a password so the example can reach it

The ClickHouse image restricts the default user to connections from
localhost unless a password is configured, and the test job reaches the
service through a port mapping, so the ClickHouse example failed to
authenticate. Set a password on the service and in the URI, in
docker-compose too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* examples: make the examples a module of their own, and run Spanner Omni in CI

The examples' drivers were becoming dependencies of sqlc itself, and the
DuckDB driver, being cgo, had to live in a module apart. Move every
example into one module under examples/, which requires sqlc through a
replace, so the drivers are its dependencies alone and cgo can stay on
for all of it; sqlc's own go.mod loses them, and lib/pq with them, since
only the examples imported it. The DuckDB example rejoins the authors
example beside the others, and the test helpers for ClickHouse, SQL
Server and Spanner move into the examples module, since the drivers they
need are what kept them out of sqlc. The end-to-end suite skips the
directory under examples that holds those helpers rather than treating
it as an example.

sqlc-test-setup now installs Spanner Omni from its standalone server
release, checked against a pinned SHA-256, and starts a single server in
the background on port 15000, so CI runs the Spanner example against it
alongside the ClickHouse and SQL Server services. The Makefile and CI
run the examples from their module.

Drop the clearing of a column's type expression in combineAnalysis: the
compiler builds a live-database analyzer only on the legacy path and
returns before doing so on the core path, so the two never meet and the
expression is never set where the analyzer runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* docs: say N/A rather than Not implemented in the language support table

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* ci: run the examples in a job of their own

The examples need every database sqlc supports, while sqlc's own tests
need only PostgreSQL and MySQL. Split the test job in two, and let
sqlc-test-setup take the databases to install and start as arguments so
each job sets up what it uses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* ci: build the end-to-end module with cgo off again

The DuckDB golden imports the DuckDB driver, which is cgo, so it is left
out of that build and checked by the replay suite alone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* codegen/json: leave out an unset type_expr; test-setup: install ClickHouse and SQL Server

A column's type_expr is only set when the analysis core typed it, and the
JSON plugin wrote every other column with an explicit null, which put a
line on every column of every request the legacy path produced. Drop the
member when it is unset instead, so those requests read as they did
before the member existed; the goldens on the legacy path are back to
what they were.

sqlc-test-setup now installs and starts ClickHouse and SQL Server as
well, so the examples need no service containers from CI's host.
ClickHouse comes from the release tarball goldeneye pins, checked
against its SHA-512 and cached where goldeneye caches it, and runs from
a configuration the tool writes that gives the default user the password
the tests use. SQL Server comes from Microsoft's apt repository for
Ubuntu 22.04 and 24.04, set up non-interactively with the EULA accepted,
and starts through systemd where there is one and in the background
otherwise. The examples job in CI installs and starts every database
through the tool.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* codegen/golang: name the engines with constants; ci: build the end-to-end module with cgo

The end-to-end module is one of its own, so cgo there is no concern of
sqlc's builds, and the DuckDB golden compiles with the rest again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* test-setup: keep the SQL Server it starts off the tool's own output

A detached server holding the tool's stderr open kept anything reading
the tool's output waiting after the tool had finished. Its output goes to
a file of its own, the way the ClickHouse and Spanner Omni servers' does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* sqlc-test-setup: accept a successful mssql-conf setup without stat-ing its conf

On a GitHub runner the tool runs as an unprivileged user, and
/var/opt/mssql is readable by the mssql user only, so stat-ing
/var/opt/mssql/mssql.conf failed with permission denied even though
setup had completed. Treat a successful setup as installed, and when
setup fails check for the configuration through sudo before deciding
whether the failure was only the service start.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* codegen/json: write an unset type_expr as null again

The regex that stripped the member from protojson's output was a hack
around EmitUnpopulated, which has no per-field form. A column the
legacy path typed now carries "type_expr": null like every other
unset message field, and the goldens are regenerated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* Bind named placeholders from the compiler, scan DuckDB lists and ClickHouse nullables correctly

A parameter whose placeholder names it, SQL Server's and Spanner's @name
and ClickHouse's {name:Type}, is now marked by the compiler: the mssql
and googlesql converters carry the name on the ParamRef as the ClickHouse
one already did, the analyzer keeps it whichever use of the parameter it
sees first, and the plugin Parameter gains a named field. The Go codegen
reads that instead of searching the query text for "@name", which bound
a CAST-wrapped @A positionally, because the analyzer had named it after
the column it was compared with, and matched "@id" inside "@identity_no".

A row struct is scanned into and a params struct is passed as arguments,
and DuckDB wants different types for the two: a list is duckdb.Composite
when scanned and a plain slice when passed. Row structs used the params
form. The imports for the element of a generic type such as
duckdb.Composite[[]time.Time] were also missing, and sql.Named's
database/sql import reached the querier interface, which does not use it.

clickhouse-go dereferences a Nullable value only when Nullable is the
column's outermost type, so LowCardinality(Nullable(T)) and
SimpleAggregateFunction(f, Nullable(T)) come through as pointers that a
sql.Null wrapper cannot scan; they take the pointer form now.

sqlc-test-setup start skips SQL Server wherever install did, rather than
failing on an Ubuntu release the packages are not published for.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* plugin: pass the placeholder's name on Parameter rather than a flag

The name a placeholder carries, @name or {name:Type}, is what the
driver binds by, so the plugin Parameter carries it as name, collected
from the query's ParamRef nodes by the compiler rather than taken from
the analyzer, which also names a parameter after the function it is
compared with. The Go codegen passes it to sql.Named instead of the
column's name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

* codegen/golang: bind parameters the new engines' drivers accept, and keep driver imports to those engines

- The driver packages the ClickHouse, DuckDB, Spanner and SQL Server
  mappers draw on are imported only for those engines, and not when an
  override's type carries the same qualifier. A PostgreSQL numeric
  overridden to another package's decimal.Decimal imported shopspring's
  as well and did not compile.
- ClickHouse parameters are typed for what clickhouse-go sends: the std
  driver never calls Value, and a {name:Type} placeholder is sent as text
  formatted from the value, so a sql.Null wrapper went out as its struct
  fields and a decimal, UUID, address or wide integer quoted. A nullable
  parameter is a pointer, and those types are strings.
- DuckDB BIT scans into duckdb.Bit. A DECIMAL parameter is passed as its
  text and a STRUCT parameter as a map, or any when nullable, since the
  driver binds neither a Decimal nor a pointer to a map, and binds a nil
  map as a struct of nulls.
- Repeated {name:Type} placeholders share one parameter, and SQL Server
  compares @name without regard to case, so @name and @name are one.
- A query that mixes named placeholders with positional ones, such as the
  ? sqlc.arg becomes, is an error: it cannot be bound either way.
- The four engines reject a sql_package other than database/sql.
- A catalog column's type_expr says whether it is nullable, as a result
  column's does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRHPAjUHm5bE8noT5ut2C5

* sqlc-test-setup: wait for SQL Server and Spanner Omni to serve, and keep interrupted installs from looking done

- start waits for SQL Server's log to say it is ready even when the
  service is already listening, as it is once mssql-conf setup starts it
  through systemd, rather than returning while it recovers its databases.
  A server this machine does not run, such as docker-compose's, still
  counts once it listens.
- A failed mssql-conf setup is tolerated only where there is no systemd,
  the one failure that leaves a usable configuration behind, and an
  install that stopped before setup finished reruns it rather than
  reporting SQL Server installed. The setup command line, which carries
  the sa password, is no longer logged.
- Spanner Omni counts as ready once ListDatabases on its default instance
  succeeds, not once its port accepts a connection, which it does before
  the instance exists. The request is encoded by hand, so the Spanner
  client libraries stay out of the module.
- The Omni release is unpacked in process into a directory beside where
  it goes and moved into place whole, so an interrupted unpack no longer
  leaves a launcher that looks installed.
- Downloads time out after 15 minutes, and the examples job after 45; its
  Go cache covers the examples module's go.sum as well.
- docs: say which of the new engines' types still come out as any, and
  how DuckDB's JSON columns are read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRHPAjUHm5bE8noT5ut2C5

* examples: regenerate the ClickHouse authors example for pointer parameters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRHPAjUHm5bE8noT5ut2C5

* duckdb: make JSON a type of its own

DuckDB's catalog lists JSON as a spelling of VARCHAR, so the seed made it
an alias and a JSON column came out as string. But DuckDB keeps JSON
apart: typeof() reports it as JSON, and duckdb-go decodes a JSON value
into whatever the document holds, which only any can receive.

goldeneye now asks typeof() for each alias, and seeds a spelling that
DuckDB reports by its own name as a type that stands on the logical type.
For JSON that is the base varchar. The DuckDB Go mapper scans JSON into
any and binds a JSON parameter from its text as a string. A new
duckdb_json case covers columns, parameters and json_extract, and its
generated code was run against DuckDB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRHPAjUHm5bE8noT5ut2C5

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: kyleconroy

.github/workflows/ci.yml

@@ -22,6 +22,9 @@ jobs:
         CGO_ENABLED: "0"
         GOOS: ${{ matrix.goos }}
         GOARCH: ${{ matrix.goarch }}
+
+  # sqlc's own tests: the end-to-end suite needs PostgreSQL and MySQL, which
+  # sqlc-test-setup installs on the runner.
   test:
     runs-on: ubuntu-24.04
     steps:
@@ -44,17 +47,17 @@ jobs:
       env:
         CGO_ENABLED: "0"
 
+    # A module of its own, so cgo stays on for the DuckDB driver the
+    # DuckDB golden imports.
     - name: build internal/endtoend
       run: go build ./...
       working-directory: internal/endtoend/testdata
-      env:
-        CGO_ENABLED: "0"
 
     - name: install databases
-      run: go run ./cmd/sqlc-test-setup install
+      run: go run ./cmd/sqlc-test-setup install postgresql mysql
 
     - name: start databases
-      run: go run ./cmd/sqlc-test-setup start
+      run: go run ./cmd/sqlc-test-setup start postgresql mysql
 
     - name: test ./...
       run: gotestsum --junitfile junit.xml -- --tags=examples -timeout 20m -failfast ./...
@@ -66,6 +69,38 @@ jobs:
         MYSQL_SERVER_URI: "root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatements=true&parseTime=true"
         CGO_ENABLED: "0"
 
+  # The examples run generated code against every database sqlc supports.
+  # They are a module of their own, so their drivers are not dependencies
+  # of sqlc, and cgo stays on for the DuckDB driver. sqlc-test-setup
+  # installs and starts every database on the runner.
+  examples:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 45
+    steps:
+    - uses: actions/checkout@v7
+    - uses: actions/setup-go@v7
+      with:
+        go-version: '1.26.5'
+        cache-dependency-path: |
+          go.sum
+          examples/go.sum
+
+    - name: install databases
+      run: go run ./cmd/sqlc-test-setup install
+
+    - name: start databases
+      run: go run ./cmd/sqlc-test-setup start
+
+    - name: test examples
+      run: go test --tags=examples -timeout 20m ./...
+      working-directory: examples
+      env:
+        POSTGRESQL_SERVER_URI: "postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable"
+        MYSQL_SERVER_URI: "root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatements=true&parseTime=true"
+        CLICKHOUSE_SERVER_URI: "clickhouse://default:mysecretpassword@127.0.0.1:9000"
+        MSSQL_SERVER_URI: "sqlserver://sa:Mysecretpassword1!@127.0.0.1:1433?encrypt=disable"
+        SPANNER_SERVER_URI: "localhost:15000"
+
   vuln_check:
     runs-on: ubuntu-24.04
     timeout-minutes: 5

CLAUDE.md

@@ -12,7 +12,7 @@ This document provides essential information for working with the sqlc codebase,
 
 ## Database Setup with sqlc-test-setup
 
-The `sqlc-test-setup` tool (`cmd/sqlc-test-setup/`) automates installing and starting PostgreSQL and MySQL for tests. Both commands are idempotent and safe to re-run.
+The `sqlc-test-setup` tool (`cmd/sqlc-test-setup/`) automates installing and starting PostgreSQL, MySQL, ClickHouse, SQL Server and Spanner Omni for tests. Both commands are idempotent and safe to re-run, and take the databases to act on as arguments (`postgresql`, `mysql`, `clickhouse`, `mssql`, `spanner`); with none named they act on all five.
 
 ### Install databases
 
@@ -24,6 +24,9 @@ This will:
 - Configure the apt proxy (if `http_proxy` is set, e.g. in Claude Code remote environments)
 - Install PostgreSQL via apt
 - Download and install MySQL 26.7 from Oracle's deb bundle
+- Download the ClickHouse 25.8 release into the user cache, where goldeneye caches it too
+- Install SQL Server 2025 from Microsoft's apt repository and run its setup, accepting the EULA on your behalf (Ubuntu 22.04 and 24.04 only)
+- Download the Spanner Omni standalone server release into the user cache (linux/amd64 only)
 - Resolve all dependencies automatically
 - Skip anything already installed
 
@@ -36,12 +39,20 @@ go run ./cmd/sqlc-test-setup start
 This will:
 - Start PostgreSQL and configure password auth (`postgres`/`postgres`)
 - Start MySQL via `mysqld_safe` and set root password (`mysecretpassword`)
-- Verify both connections
+- Start a ClickHouse server in the background on localhost, with the default user's password `mysecretpassword`
+- Start SQL Server through systemd, or in the background where there is no systemd, with the sa password `Mysecretpassword1!`
+- Start a Spanner Omni single server in the background, serving plaintext gRPC on port 15000
+- Verify the connections
 - Skip steps that are already done (running services, existing config)
 
 Connection URIs after start:
 - PostgreSQL: `postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable`
 - MySQL: `root:mysecretpassword@tcp(127.0.0.1:3306)/mysql`
+- ClickHouse: `clickhouse://default:mysecretpassword@127.0.0.1:9000` (`CLICKHOUSE_SERVER_URI`)
+- SQL Server: `sqlserver://sa:Mysecretpassword1!@127.0.0.1:1433?encrypt=disable` (`MSSQL_SERVER_URI`)
+- Spanner Omni: `localhost:15000` (`SPANNER_SERVER_URI`)
+
+The examples' tests for ClickHouse, SQL Server and Spanner read those variables and skip when unset. `docker-compose.yml` runs the same databases for anyone who prefers containers.
 
 ### Run tests
 
@@ -172,9 +183,13 @@ them as an artifact.
 
 ### Example Tests
 
-- **Location:** `/examples/` directory
-- **Requirements:** Tagged with "examples", requires live databases
-- **Databases:** PostgreSQL, MySQL, SQLite examples
+- **Location:** `/examples/` directory, a Go module of its own so the
+  drivers the examples run against are not dependencies of sqlc, and so
+  cgo stays on for the DuckDB driver
+- **Requirements:** Tagged with "examples", requires live databases; run
+  with `cd examples && go test --tags=examples ./...`
+- **Databases:** PostgreSQL, MySQL, SQLite, ClickHouse, DuckDB, Spanner and
+  SQL Server examples; each skips when its server is not named
 
 ## Database Services
 
@@ -207,8 +222,9 @@ make start             # Start database containers
 
 - **File:** `.github/workflows/ci.yml`
 - **Go Version:** 1.26.4
-- **Database Setup:** Uses `sqlc-test-setup` (not Docker) to install and start PostgreSQL and MySQL directly on the runner
-- **Test Command:** `gotestsum --junitfile junit.xml -- --tags=examples -timeout 20m ./...`
+- **Jobs:** `build` cross-compiles sqlc with cgo off; `test` runs sqlc's own tests; `examples` runs the examples module against every database; `vuln_check` runs `govulncheck`
+- **Database Setup:** `sqlc-test-setup` (not Docker) installs and starts the databases directly on the runner: PostgreSQL and MySQL for `test`, and all five for `examples`
+- **Test Command:** `gotestsum --junitfile junit.xml -- --tags=examples -timeout 20m ./...` in `test`, and `go test --tags=examples ./...` under `examples/` in `examples`
 - **Additional Checks:** `govulncheck` for vulnerability scanning
 
 ## Development Workflow

Makefile

@@ -14,9 +14,11 @@ test-managed:
 
 vet:
 	go vet ./...
+	cd ./examples && go vet --tags=examples ./...
 
 test-examples:
 	go test --tags=examples ./...
+	cd ./examples && go test --tags=examples ./...
 
 build-endtoend:
 	cd ./internal/endtoend/testdata && go build ./...