[WIP][SPARK-55206][PYTHON][SQL] Transpilation minimal functional implementation with python

[holdenk]

What changes were proposed in this pull request?

This is the first PR adding initial transpilation from Python to Catalyst and the framework for others to plug in different transpilers which can target other targets. This replaces #53547 and is the first PR as part of the transpilation SPIP https://docs.google.com/document/d/1cHc6tiR4yO3hppTzrK1F1w9RwyEPMvaeEuL2ub2LURg/edit?tab=t.0#heading=h.iz92aps6qbo5

Why are the changes needed?

Python UDF performance has been a perinial challenge in Spark, and while PyArrow makes the data copy slightly less expensive it's still much slower than native code. Additionally, Python UDF usage has increased substantailly with Pandas on Spark and it is especially hard for folks to write catalyst expressions here.

Does this PR introduce any user-facing change?

New configuration flags are user visible (default to off).

How was this patch tested?

New unit tests that trigger always and hypothesis tests that trigger only selectively.

Was this patch authored or co-authored using generative AI tooling?

Hypothesis teests were generated by Claude 4.7, eq semantics null and truthiness w/claude 4.6, GH code review (unknown backing model) on itiial draft). snake camel swap with sonnet.

[holdenk]

CC @devin-petersohn would love your input and review, I know it's still in draft phase though. I'm going to try and finish it up over the coming week

[gaogaotiantian]

I have a question - how correct and maintainable we want this to be? Python ast modules changes from version to version (not like huge changes but they do change. New nodes will be created and old nodes can be deprecated). We need to support a wide range of Python versions, do we build multiple versions of ast transpilers for different python versions?

Also, do we expect the users to "try it out" and see if it kind of works, or we aim to be "always correct or raising an error"? There's a huge difference.

For example:

        case ast.Compare():
            # All comparison operators produce bool.
            return True

Nope, pd.Series([1]) < pd.Series([0]) returns a pd.Series.

        case ast.BoolOp():
            # and / or of booleans produces bool.
            return True

Nope, 0 or 2 == 2

The non-boolean check for operations are not fully valid either - any user defined type can return a True for + - which is not that uncommon.

Like I mentioned when this SPIP was firstly discussed - Python is a super dynamic language and having a transpiler that is always correct on where it claims to be correct is not easy. So I want to confirm the purpose of the SPIP - do we want to be always correct, or correct most of the time to cover more transpilable UDFs, with the cost that we could produce wrong result?

[holdenk]

For ast module changes I think we're ok with just not transpiling new / different items. For the comparison operators right now we're limiting the UDF transpilation logic to scalar inputs we can scope it down more to also exclude non scalar captures (and expand back out later if we want for arrow / pandas types but handle them explicitly). To be clear this is. WIP PR but was talking with Scott about this in the context of some of his work so I wanted to raise the current draft for folks while
I'm finishing it up / getting it ready for a full review over the coming week.