Agent/SCIM postgresql query analyzer#1
Merged
Conversation
added 6 commits
July 10, 2026 16:28
Extend scim-sql with ai.singlr.postgresql, a generic PostgreSQL query-analysis API that parses complete SQL and reports structural facts: statement kind and count, relations (physical/CTE/function, with aliases), columns, functions, named parameters, policy-relevant features, and a deterministic normalized form for hashing and audit. It parses and describes SQL only - no execution, catalog resolution, authorization, or policy. The grammar is the ANTLR grammars-v4 PostgreSQL grammar vendored at commit 76093c04af6a51f38a67d14f7e71ff0a9b4400da with one deliberate extension: named parameters (:name) are first-class expression values and can never be identifiers, aliases, or relations. Provenance, verbatim upstream licenses, local modifications, and the upgrade path are recorded in NOTICE.md and shipped in the jars under META-INF. Analysis walks all nested scopes with progressive CTE scoping (non-recursive CTE bodies do not see their own name; siblings see earlier names; inner scopes shadow outer). Input is bounded by length, token count, and nesting depth; exceptions carry only a stable reason plus line/column and never echo SQL content. Grammar warnings are pinned (two benign upstream warning-146 lexer rules) and any new warning or error fails the build. An upstream regression corpus, adversarial/hostile input tests, and module-export tests protect behavior. Existing SCIM behavior and API are unchanged; the artifact gains no runtime dependency beyond the existing ANTLR runtime. Version bumped to 1.1.0-SNAPSHOT.
- Resolve CTE names only where PostgreSQL permits a CTE source, so INSERT, UPDATE, DELETE, and SELECT INTO targets stay PHYSICAL - Fold identifiers and normalized SQL with ASCII-only lowercasing to match PostgreSQL's UTF-8 identifier folding - Put all sibling CTE names in scope under WITH RECURSIVE - Rework MERGE grammar as ordered WHEN clauses, accepting DO NOTHING, repeated WHEN clauses, and INSERT DEFAULT VALUES
- Reject psql meta-commands: the META lexer mode silently turned
backslash commands into statement separators, so an allowed SELECT
could smuggle \! or \copy past analysis; a backslash now lexes as
ErrorCharacter and fails closed (corpus files updated accordingly)
- Reject Unicode-escaped identifiers (U&"...") at lex time: their
effective name differs from their spelling, so name-based policies
could be bypassed with an equivalent spelling
- Collect JOIN USING names as column references
- Report DROP TABLE / VIEW / MATERIALIZED VIEW / FOREIGN TABLE targets
as physical relations
- Lex nested block comments iteratively with a depth counter: the
recursive rule needed ~1s for 5 KB of 1250-deep nesting and overflowed
the stack near 1500; now 6 ms at depth 20000
- Accept PostgreSQL newline string continuation ('a'\n'b'), matching
scan.l semantics verified against PostgreSQL 16: line comments count
as separating whitespace, block comments and dollar-quoted strings
do not, same-line adjacency stays an error
- Reject identifiers longer than 63 UTF-8 bytes (post-unescape) so the analyzer never reports a name PostgreSQL would silently truncate to a different effective identifier - Collect bare ON CONFLICT arbiter and CREATE INDEX columns via the colid alternative of index_elem - Accept WITH ... MERGE: grammar allows a leading with_clause_, MERGE is a with-clause owner for CTE scoping, and the USING relation (but never the MERGE target) can resolve as a CTE source
- Split greedy :name lexing into COLON + name when the colon continues an expression, so compact JSON key:value and array slices (arr[lo:hi]) parse; arr[:name] deliberately binds to the named-parameter extension (documented) - Parse PostgreSQL 14+ unquoted CREATE FUNCTION bodies (RETURN expr and BEGIN ATOMIC stmt; ... END); nested body statements do not affect the root statement count - Report relations targeted through any_name: COMMENT ON TABLE/COLUMN/ CONSTRAINT..ON/TRIGGER..ON, SECURITY LABEL ON TABLE/COLUMN, and DROP TRIGGER/RULE/POLICY .. ON - Classify REASSIGN OWNED as DDL - Keep normalizedSql single-line: string continuation joins with a space (collision-safe: same-line string adjacency is rejected by the parser) - Refresh NOTICE.md local-modification provenance (also records the earlier iterative block-comment lexing change)
The vendored grammars-v4 grammar (verified at parity with upstream HEAD) ships several defects and unported productions; each fix below follows PostgreSQL's authoritative gram.y and is recorded in NOTICE.md: - json_aggregate_func existed but was never referenced: wired into func_expr/func_expr_windowless with FILTER/OVER, returning clause made optional; aggregates now reported as functions - MERGE: PG17 RETURNING clause and WHEN NOT MATCHED BY SOURCE/BY TARGET - GROUP BY [ALL|DISTINCT] (PG14) - CTE SEARCH and CYCLE clauses (PG14); fixed upstream BREADTH: 'BREATH' lexer typo that made SEARCH BREADTH FIRST unparseable - IS [NOT] JSON [VALUE|ARRAY|OBJECT|SCALAR] predicate (PG16) - JSON_TABLE table function (PG17), reported as a FUNCTION relation with alias like xmltable, which now also reports one - XMLTABLE column PATH option (PATH is a lexer keyword here, so the generic identifier option could not match it) - json_format_clause now uses the real FORMAT token; upstream's FORMAT_LA literal is Bison lookahead residue that never occurs in SQL text - UNIQUE/PRIMARY KEY ... WITHOUT OVERLAPS (PG18) Conformance battery over PG14-18 syntax went 18/30 to 30/30; all forms pinned as tests. Grammar warnings remain pinned with no new entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.