fix: ignore comments when post-processing tokens

[Jeremy-xuan]

fix: ignore comments when post-processing tokens

What's wrong

The dialect postProcess() steps compare a token with its immediate neighbours, and
comments are tokens too — so a comment between two syntax elements defeats these
checks. Three symptoms, one cause.

1. BigQuery throws a Parse error (comments before OFFSET inside an array subscript):

SELECT arr[/* c */ OFFSET(0)] FROM t;

Parse error at token: OFFSET at line 1 column 20

A line comment does the same (arr[-- c\nOFFSET(0)]). arr[OFFSET(/* c */ 0)] is
unaffected — only a comment between [ and OFFSET breaks.

2. ClickHouse stops demoting SELECT in privilege statements:

CHECK GRANT SELECT ON db.table          -- SELECT is a privilege, not a clause
CHECK GRANT /* c */ SELECT ON db.table  -- comment defeats the demotion

Before: CHECK GRANT\n /* c */\nSELECT\n ON db.table
After: CHECK GRANT\n /* c */\n SELECT ON db.table — matches the comment-free output.

3. Spark formats window(...) as a WINDOW clause instead of a function call:

SELECT window (time) FROM tbl;        -- window(time)          (a call)
SELECT window /* c */ (time) FROM tbl; -- window\n  /* c */\n  (time)   (a clause)

4. MySQL/MariaDB/TiDB/SingleStoreDB and ClickHouse stop treating SET( and VALUES() as functions
(the tokenizer keeps them as clauses when a comment sits between the name and the parens):

mysql, without comment:  ... IDENTIFIER(a) RESERVED_FUNCTION_NAME(SET) OPEN_PAREN( ...
mysql, with comment:     ... IDENTIFIER(a) RESERVED_FUNCTION_NAME(SET) BLOCK_COMMENT OPEN_PAREN( ...

Both are RESERVED_FUNCTION_NAME after this change; before it, the commented variant
stayed RESERVED_CLAUSE/RESERVED_KEYWORD and was formatted as a statement.

Root cause

// bigquery.formatter.ts
function detectArraySubscripts(tokens: Token[]) {
  let prevToken = EOF_TOKEN;
  return tokens.map(token => {
    if (token.text === 'OFFSET' && prevToken.text === '[') { ... }

// clickhouse.formatter.ts, mariadb/likeMariaDb.ts and spark.formatter.ts
const nextToken = tokens[i + 1] || EOF_TOKEN;
const prevToken = tokens[i - 1] || EOF_TOKEN;

These look at the literal neighbours, so any comment in between hides the token that
the check is actually about. plsql.formatter.ts already does this correctly — it
tracks previousReservedToken instead of the previous token — so the fix follows an
existing in-repo pattern.

The fix

Two small helpers in src/lexer/token.ts, used by all four post-processing steps:

export const prevNonCommentToken = (tokens: Token[], index: number): Token => { ... };
export const nextNonCommentToken = (tokens: Token[], index: number): Token => { ... };

Comment token types skipped: LINE_COMMENT, BLOCK_COMMENT, DISABLE_COMMENT
(the sql-formatter-disable region is also not syntax).

plsql.formatter.ts was left alone: it already skips everything that is not a reserved
token, so it was never affected.

Scope

This covers the four postProcess implementations. The same hazard exists in
bigquery.formatter.ts's combineParameterizedTypes(), which also compares itself
against the next token (tokens[i + 1]?.text === '<') — but that one cannot be fixed
the same way, because it slices the token range it decided on and joins it with no
separator. Skipping comments there changes which tokens fall inside that range, so a
one-line change to the comparison makes things worse rather than better. It needs its
own change and is left out of this PR on purpose.

Tests

Regression tests in the test files that own each behaviour:

• test/bigquery.test.ts — OFFSET after a block comment and after a line comment (2)
• test/clickhouse.test.ts — EXPLAIN AST, CHECK GRANT and SET( with a comment (5)
• test/spark.test.ts — WINDOW() after a block comment and after a line comment (2)
• test/behavesLikeMariaDbFormatter.ts — VALUES() after a block and a line comment,
  which runs against MySQL, MariaDB, TiDB and SingleStoreDB through the shared suite (2 × 4 dialects)
• test/mysql.test.ts — SET( with a comment (1)

Every assertion fails on master and passes with this change (checked by reverting
src/ to master and re-running: 18 failures, all of them these tests).

pnpm test   →  Test Suites: 27 passed, 27 total
               Tests:       5864 passed, 1 skipped, 5865 total
pnpm run lint / pretty:check / ts:check / build   →  all clean