[SPARK-52709][SQL] Fix parsing of STRUCT<>

[yadavay-amzn]

What changes were proposed in this pull request?

Fix the STRUCT<> parsing bug by adding a parser-level action to decrement complex_type_level_counter when STRUCT NEQ is matched in the dataType rule.

Root cause: When the lexer sees STRUCT<>, it tokenizes it as STRUCT + NEQ (because NEQ is defined before LT). This increments the counter for STRUCT but never decrements it (no GT token), corrupting the counter. Subsequent >> then fails to be recognized as shift-right.

Previous attempt: PR #51480 was merged then reverted because it modified the NEQ lexer rule, which broke ARRAY(col1 <> col2) where <> is the not-equal operator.

This fix: Follows @cloud-fan's suggestion to handle it at the parser level. When the parser matches STRUCT NEQ in the dataType rule, we know NEQ is being used as empty angle brackets (not as a comparison), so we decrement the counter via an inline action.

Why are the changes needed?

SELECT CAST(null AS STRUCT<>), 2 >> 1 fails with a syntax error because the corrupted counter prevents >> from being recognized as shift-right.

Does this PR introduce any user-facing change?

No. This is a correctness fix for SQL parsing.

How was this patch tested?

Added test in SparkSqlParserSuite covering 6 cases:

1. STRUCT<> followed by >> (the original bug)
2. Multiple STRUCT<> followed by >>
3. STRUCT<> followed by >>> (unsigned shift right)
4. ARRAY(1 <> 2) still works (the regression case from the reverted PR)
5. Nested complex types MAP<STRING, ARRAY<INT>> still work
6. Mix of empty struct + nested types + shift right

Test results:

• SparkSqlParserSuite: 46/46 passed
• DataTypeParserSuite: 59/59 passed
• PlanParserSuite: 80/80 passed

Was this patch authored or co-authored using generative AI tooling?

Yes

[sarutak]

Hi @yadavay-amzn, thank you for your contribution!
GA workflow seems failed. Could you enable it?
https://github.com/apache/spark/pull/55285/checks?check_run_id=70712131544

[sarutak]

The GA failure seems not related to this change. Could you rebase to master ?

[cloud-fan]

The fix is correct and well-scoped. The parser-level approach is the right call — it avoids the lexer-level ambiguity that caused the previous revert (PR #51480), and the dataType rule guarantees STRUCT NEQ always means empty angle brackets (never a comparison operator).

Tests cover the key scenarios thoroughly, including the regression guard for ARRAY(1 <> 2).

[cloud-fan]

@yadavay-amzn please reply in the JIRA ticket, so that I can assign it to you, thanks!