Skip to content

Conversation

@ignat980
Copy link

@ignat980 ignat980 commented Nov 10, 2025

fix(compiler): robustly strip psql meta commands without breaking SQL

Replace naive line-based removal with a single-pass state machine that correctly distinguishes psql meta-commands from backslashes in SQL code, literals, and comments.

The previous implementation would incorrectly strip any line starting with a backslash, breaking valid SQL containing:

  • Backslashes in string literals (e.g. E'\\n', escape sequences)
  • Meta-command text in comments or documentation
  • Dollar-quoted function bodies with backslash content

Changes:

  • Track parsing state for single quotes, dollar quotes, and block comments
  • Only remove backslash commands at true line starts outside any literal context
  • Properly handle escaped quotes (''), nested block comments (/* /* */ */)
  • Support dollar-quoted tags with identifiers (e.g. $tag$...$tag$)
  • Add comprehensive test suite covering:
    • All documented psql meta-commands (\connect, \set, \d*, etc.) See PostgreSQL psql docs
    • String literals with backslashes and nested quotes
    • Dollar-quoted blocks with various tag formats
    • Nested block comments containing meta-command text
    • Edge cases: empty input, whitespace-only, missing newlines

Performance improvements:

  • Pre-allocate output buffer with strings.Builder.Grow()
  • Single pass eliminates redundant string operations
  • Reduces allocations by avoiding intermediate line slices

Testing

  • go test ./internal/compiler
  • 100% test coverage of new function removePsqlMetaCommands()

Addresses gbarr's comment in #4082 which closes #4065

kyleconroy and others added 15 commits September 1, 2025 16:00
* Update examples for 1.30.0

* Update tests for v1.30.0

* Update version to v1.30.0
…qlc-dev#4091)

Bumps the production-dependencies group with 2 updates: [github.com/spf13/cobra](https://github.com/spf13/cobra) and [github.com/spf13/pflag](https://github.com/spf13/pflag).


Updates `github.com/spf13/cobra` from 1.9.1 to 1.10.1
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](spf13/cobra@v1.9.1...v1.10.1)

Updates `github.com/spf13/pflag` from 1.0.7 to 1.0.9
- [Release notes](https://github.com/spf13/pflag/releases)
- [Commits](spf13/pflag@v1.0.7...v1.0.9)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-version: 1.10.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: github.com/spf13/pflag
  dependency-version: 1.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the production-dependencies group with 1 update: [github.com/spf13/pflag](https://github.com/spf13/pflag).


Updates `github.com/spf13/pflag` from 1.0.9 to 1.0.10
- [Release notes](https://github.com/spf13/pflag/releases)
- [Commits](spf13/pflag@v1.0.9...v1.0.10)

---
updated-dependencies:
- dependency-name: github.com/spf13/pflag
  dependency-version: 1.0.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 6.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](actions/setup-go@v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps golang from 1.25.0 to 1.25.1.

---
updated-dependencies:
- dependency-name: golang
  dependency-version: 1.25.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…y with 4 updates (sqlc-dev#4100)

Bumps the production-dependencies group with 4 updates in the / directory: [github.com/jackc/pgx/v5](https://github.com/jackc/pgx), [golang.org/x/sync](https://github.com/golang/sync), [google.golang.org/grpc](https://github.com/grpc/grpc-go) and google.golang.org/protobuf.


Updates `github.com/jackc/pgx/v5` from 5.7.5 to 5.7.6
- [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md)
- [Commits](jackc/pgx@v5.7.5...v5.7.6)

Updates `golang.org/x/sync` from 0.16.0 to 0.17.0
- [Commits](golang/sync@v0.16.0...v0.17.0)

Updates `google.golang.org/grpc` from 1.75.0 to 1.75.1
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.75.0...v1.75.1)

Updates `google.golang.org/protobuf` from 1.36.8 to 1.36.9

---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v5
  dependency-version: 5.7.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: golang.org/x/sync
  dependency-version: 0.17.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: google.golang.org/grpc
  dependency-version: 1.75.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: google.golang.org/protobuf
  dependency-version: 1.36.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…y with 3 updates (sqlc-dev#4130)

Bumps the production-dependencies group with 3 updates in the / directory: [google.golang.org/grpc](https://github.com/grpc/grpc-go), google.golang.org/protobuf and [modernc.org/sqlite](https://gitlab.com/cznic/sqlite).


Updates `google.golang.org/grpc` from 1.75.1 to 1.76.0
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.75.1...v1.76.0)

Updates `google.golang.org/protobuf` from 1.36.9 to 1.36.10

Updates `modernc.org/sqlite` from 1.38.2 to 1.39.0
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.38.2...v1.39.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.76.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: google.golang.org/protobuf
  dependency-version: 1.36.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: modernc.org/sqlite
  dependency-version: 1.39.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…y with 4 updates (sqlc-dev#4127)

Bumps the production-dependencies group with 4 updates in the /docs directory: [markupsafe](https://github.com/pallets/markupsafe), [certifi](https://github.com/certifi/python-certifi), [pyparsing](https://github.com/pyparsing/pyparsing) and [sphinxext-rediraffe](https://github.com/sphinx-doc/sphinxext-rediraffe).


Updates `markupsafe` from 3.0.2 to 3.0.3
- [Release notes](https://github.com/pallets/markupsafe/releases)
- [Changelog](https://github.com/pallets/markupsafe/blob/main/CHANGES.rst)
- [Commits](pallets/markupsafe@3.0.2...3.0.3)

Updates `certifi` from 2025.8.3 to 2025.10.5
- [Commits](certifi/python-certifi@2025.08.03...2025.10.05)

Updates `pyparsing` from 3.2.3 to 3.2.5
- [Release notes](https://github.com/pyparsing/pyparsing/releases)
- [Changelog](https://github.com/pyparsing/pyparsing/blob/master/CHANGES)
- [Commits](pyparsing/pyparsing@3.2.3...3.2.5)

Updates `sphinxext-rediraffe` from 0.2.7 to 0.3.0
- [Release notes](https://github.com/sphinx-doc/sphinxext-rediraffe/releases)
- [Commits](sphinx-doc/sphinxext-rediraffe@v0.2.7...v0.3.0)

---
updated-dependencies:
- dependency-name: markupsafe
  dependency-version: 3.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: certifi
  dependency-version: 2025.10.5
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: pyparsing
  dependency-version: 3.2.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: sphinxext-rediraffe
  dependency-version: 0.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the production-dependencies group with 1 update: [modernc.org/sqlite](https://gitlab.com/cznic/sqlite).


Updates `modernc.org/sqlite` from 1.39.0 to 1.39.1
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.39.0...v1.39.1)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-version: 1.39.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps golang from 1.25.1 to 1.25.3.

---
updated-dependencies:
- dependency-name: golang
  dependency-version: 1.25.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Create the sqlfile.Split method that splits a .sql file into multiple
statements.
…qlc-dev#4137)

Bumps the production-dependencies group in /docs with 1 update: [idna](https://github.com/kjd/idna).


Updates `idna` from 3.10 to 3.11
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v3.10...v3.11)

---
updated-dependencies:
- dependency-name: idna
  dependency-version: '3.11'
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. 🔧 golang labels Nov 10, 2025
@ignat980
Copy link
Author

@andrewmbenton please review

@gbarr
Copy link

gbarr commented Nov 10, 2025

Thanks @ignat980 this looks a much more complete solution than I was expecting

@kyleconroy
Copy link
Collaborator

@ignat980 if you open this against main I can get this merged. Just make sure to includes Andrew's commits.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 10, 2025
andrewmbenton and others added 2 commits December 11, 2025 09:39
Replace naive line-based removal with a single-pass state machine that correctly distinguishes psql meta-commands from backslashes in SQL code, literals, and comments.

The previous implementation would incorrectly strip any line starting with a backslash, breaking valid SQL containing:
- Backslashes in string literals (E'\\n', escape sequences)
- Meta-command text in comments or documentation
- Dollar-quoted function bodies with backslash content

Changes:
- Track parsing state for single quotes, dollar quotes, and block comments
- Only remove backslash commands at true line starts outside any literal context
- Properly handle escaped quotes (''), nested block comments (/* /* */ */)
- Support dollar-quoted tags with identifiers ($tag$...$tag$)
- Add comprehensive test suite covering:
  * All documented psql meta-commands (\connect, \set, \d*, etc.)
  * String literals with backslashes and nested quotes
  * Dollar-quoted blocks with various tag formats
  * Nested block comments containing meta-command text
  * Edge cases: empty input, whitespace-only, missing newlines

Performance improvements:
- Pre-allocate output buffer with strings.Builder.Grow()
- Single pass eliminates redundant string operations
- Reduces allocations by avoiding intermediate line slice
@ignat980 ignat980 force-pushed the fix/strip-psql-meta branch from 3436dde to 60585cd Compare December 11, 2025 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files. 🔧 golang

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants