Skip to content

fix: keep a keyword reserved when directly followed by a -- line comment - #978

Open
Jeremy-xuan wants to merge 1 commit into
sql-formatter-org:masterfrom
Jeremy-xuan:fix/dashed-ident-line-comment
Open

Jeremy-xuan wants to merge 1 commit into
sql-formatter-org:masterfrom
Jeremy-xuan:fix/dashed-ident-line-comment

Conversation

@Jeremy-xuan

Copy link
Copy Markdown

fix: keep a keyword reserved when directly followed by a -- line comment

What's wrong

In BigQuery, a keyword that is immediately followed by a -- line comment is not
tokenized as a keyword, so the clause break is lost and formatting becomes
non-idempotent.

-- input
SELECT-- c
a FROM t;
-- current output      (SELECT demoted to an identifier: no indent for the select list)
SELECT -- c
a
FROM
  t;
-- re-formatting that output changes it again
SELECT -- c
  a
FROM
  t;

The same happens to any clause keyword, and the first pass output is the wrong one:

-- input
SELECT a FROM-- c
t;
-- current output      (FROM glued onto the select list)
SELECT
  a FROM -- c
  t;

Tokenizing shows the cause directly — in BigQuery, SELECT and FROM come out as
IDENTIFIER instead of RESERVED_SELECT / RESERVED_CLAUSE:

"SELECT-- c\na FROM t;"  → IDENTIFIER("SELECT")  ...
"SELECT a FROM-- c\nt;"  → RESERVED_SELECT("SELECT") IDENTIFIER("a") IDENTIFIER("FROM") ...

Root cause

src/lexer/regexFactory.ts builds the reserved-word regex with a negative lookahead
that prevents a keyword from matching when it is really the beginning of a longer
identifier:

const rejectIdentCharsPattern = ({ rest, dashes }: IdentChars): string =>
  rest || dashes ? `(?![${rest || ''}${dashes ? '-' : ''}])` : '';

With identChars.dashes enabled, that emits (?![-]), which rejects every keyword
followed by a dash. The guard is right about SELECT-a (a single dashed identifier —
BigQuery is the only dialect with dashes: true) but wrong about SELECT--, where the
dash does not extend the identifier at all: with withDashes the segment after a dash
must start with an identifier character, and -- is the start of a line comment.

Because two-- never extends an identifier, the keyword rule is skipped, the generic
identifier rule matches SELECT instead, and the formatter treats it as a plain word.
After the first pass the comment sits behind a space, the guard no longer applies, and
the second pass produces different (correct) output — hence the non-idempotency.

The fix

A dash only extends an identifier when it is followed by a character that can start an
identifier segment, so only that case should be rejected:

const rejectIdentCharsPattern = (identChars: IdentChars): string => {
  const { rest, dashes } = identChars;
  const alternatives = [
    ...(rest ? [`[${rest}]`] : []),
    ...(dashes ? [`-(?=${identFirstCharsPattern(identChars)})`] : []),
  ];
  return alternatives.length ? `(?!${alternatives.join('|')})` : '';
};

identFirstCharsPattern() is extracted from identifierPattern() so the lookahead and
the identifier pattern take their "first character" class from the same place (the
segment after a dash repeats that same pattern). identifierPattern() produces a
byte-identical regex for every existing configuration — I checked the old and new
implementations against 8 IdentChars configurations ({}, {first, rest},
{dashes}, {dashes, first}, {allowFirstCharNumber}, {first, rest, allowFirstCharNumber}, {rest}, {dashes, rest}): 8/8 identical. Only the
dashes: true lookahead changes, and BigQuery is the only dialect that sets it.

Tests

Added two regression tests to test/bigquery.test.ts (language-specific tests belong
in their dialect test file), covering both a leading clause keyword and a trailing one.
Both fail before the change and pass after it.

pnpm test        →  Test Suites: 27 passed, 27 total
                    Tests:       5848 passed, 1 skipped, 5849 total
pnpm run lint / pretty:check / ts:check / build   →  all clean

SELECT-a FROM t; still tokenizes as a single IDENTIFIER("SELECT-a") — the guard
still does its original job.

BigQuery is the only dialect with identChars.dashes enabled, which makes the
keyword guard in regexFactory reject any keyword followed by a dash. That guard
exists to avoid splitting dashed identifiers like SELECT-a, but a dash only
extends an identifier when it is followed by a character that can start one.
A dash followed by another dash starts a line comment instead, so SELECT-- c
and FROM-- c had their keywords demoted to plain identifiers, losing the
clause break and making formatting non-idempotent.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant