fix: keep a keyword reserved when directly followed by a -- line comment - #978
Open
Jeremy-xuan wants to merge 1 commit into
Open
Jeremy-xuan wants to merge 1 commit into
Jeremy-xuan wants to merge 1 commit into
Conversation
BigQuery is the only dialect with identChars.dashes enabled, which makes the keyword guard in regexFactory reject any keyword followed by a dash. That guard exists to avoid splitting dashed identifiers like SELECT-a, but a dash only extends an identifier when it is followed by a character that can start one. A dash followed by another dash starts a line comment instead, so SELECT-- c and FROM-- c had their keywords demoted to plain identifiers, losing the clause break and making formatting non-idempotent.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: keep a keyword reserved when directly followed by a
--line commentWhat's wrong
In BigQuery, a keyword that is immediately followed by a
--line comment is nottokenized as a keyword, so the clause break is lost and formatting becomes
non-idempotent.
The same happens to any clause keyword, and the first pass output is the wrong one:
Tokenizing shows the cause directly — in BigQuery,
SELECTandFROMcome out asIDENTIFIERinstead ofRESERVED_SELECT/RESERVED_CLAUSE:Root cause
src/lexer/regexFactory.tsbuilds the reserved-word regex with a negative lookaheadthat prevents a keyword from matching when it is really the beginning of a longer
identifier:
With
identChars.dashesenabled, that emits(?![-]), which rejects every keywordfollowed by a dash. The guard is right about
SELECT-a(a single dashed identifier —BigQuery is the only dialect with
dashes: true) but wrong aboutSELECT--, where thedash does not extend the identifier at all: with
withDashesthe segment after a dashmust start with an identifier character, and
--is the start of a line comment.Because two-
-never extends an identifier, the keyword rule is skipped, the genericidentifier rule matches
SELECTinstead, and the formatter treats it as a plain word.After the first pass the comment sits behind a space, the guard no longer applies, and
the second pass produces different (correct) output — hence the non-idempotency.
The fix
A dash only extends an identifier when it is followed by a character that can start an
identifier segment, so only that case should be rejected:
identFirstCharsPattern()is extracted fromidentifierPattern()so the lookahead andthe identifier pattern take their "first character" class from the same place (the
segment after a dash repeats that same pattern).
identifierPattern()produces abyte-identical regex for every existing configuration — I checked the old and new
implementations against 8
IdentCharsconfigurations ({},{first, rest},{dashes},{dashes, first},{allowFirstCharNumber},{first, rest, allowFirstCharNumber},{rest},{dashes, rest}): 8/8 identical. Only thedashes: truelookahead changes, and BigQuery is the only dialect that sets it.Tests
Added two regression tests to
test/bigquery.test.ts(language-specific tests belongin their dialect test file), covering both a leading clause keyword and a trailing one.
Both fail before the change and pass after it.
SELECT-a FROM t;still tokenizes as a singleIDENTIFIER("SELECT-a")— the guardstill does its original job.