fix: prevent stack overflow in symbolToTypeNode with recursive return type references

[wdskuki]

Fixes #63273

Problem

When processing a type like:

function clone(): <T>(obj: T) => T extends any ? ReturnType<typeof clone>[0] : 0;

The compiler enters infinite recursion:

1. typeToTypeNodeHelper processes the conditional type
2. Calls symbolToTypeNode(clone, Value) to get typeof clone
3. Which calls createAccessFromSymbolChain → getNameOfSymbolAsWritten(clone)
4. Which tries to get the declaration name, triggering step 1 again
5. Stack overflow

Solution

Add recursive reference detection in symbolToTypeNode using a visitedSymbols set in the NodeBuilderContext. When a symbol is detected as already being processed, return never type to break the cycle.

Changes

• Added visitedSymbols: Set<string> | undefined to NodeBuilderContext interface
• Added cycle detection at the start of symbolToTypeNode
• Clean up visitedSymbols on all return paths

Testing

Test case from #63273:

function clone(): <T>(obj: T) => T extends any ? ReturnType<typeof clone>[0] : 0;

Before: RangeError: Maximum call stack size exceeded
After: Compiles successfully (returns never for the recursive reference)

Related Issues

• Crash: RangeError: Invalid string length in addSpans during instantiation of recursive template literal types #63271 - Similar crash with recursive template literal types
• Crash: RangeError: Maximum call stack size exceeded in isRelatedTo with recursive tuple types and spread elements #63270 - Another stack overflow with recursive tuple types
• Regression: RangeError: Maximum call stack size exceeded in instantiateTypeWorker with recursive intersections and invalid type queries #63269 - Regression with recursive intersections

These may benefit from similar protection or may need separate fixes.

[wdskuki]

@microsoft-github-policy-service agree

[RyanCavanaugh]

See https://github.com/microsoft/TypeScript?tab=readme-ov-file#contribute - bugfixes should be in typescript-go repo