bug: graphframes on Databricks Free Edition throws No handler found for extension type

[olbapjose]

Describe the bug

I have installed graphframes-py in Databricks Free Edition with environment version 5 by !pip install graphframes-py. I can import it and create a graph, but algorithms like pageRank or connectedComponents throw No handler found for extension type: type.googleapis.com/org.graphframes.connect.proto.GraphFrames.

• The workaround suggested in another thread of setting spark.databricks.service.server.enabled to false is not possible because Free Edition does not allow spark.conf.set.
• Downloading the JAR file manually is possible but I cannot make Spark read it, as it is not possible to either write on current classpath folders (I have tried but they are all protected) nor modify the CLASSPATH variable - it won't take the new value.

To Reproduce

Steps to reproduce the behavior:

1. Create a notebook in Databricks Free Edition
2. !pip install graphframes-py
3. Create a graph, and try g.pageRank(...) or g.connectedComponents.
4. However, g.degrees.show() or g.inDegrees.show() or g.outDegrees.show() are working normally

Expected behavior

The algorithms should run normally.

System [please complete the following information]:

• OS: e.g. [Ubuntu 18.04]
• Python Version (if applied): Python 3.12.3 that comes with Databricks Free environment 5
• Spark / PySpark version: 4.1.0
• GraphFrames version: graphframes-py==0.9.2 and graphframes-py==0.11.0

Component

• Scala Core Internal
• Scala API
• Spark Connect Plugin
• PySpark Classic
• PySpark Connect

Additional context

Are you planning on creating a PR?

• I'm willing to make a pull-request

[SemyonSinchenko]

GraphFrames uses the runtime check of the current environment under the hood using is_remote from the pyspark.sql.utils. Based on the result of the check it choose one of:

1. graphframes.classic.graphframe.GraphFrame
2. graphframes.connect.graphframes_client.GraphFrameConnect

As a workaround you can try to explicitly import classic-backed implementation:

from graphframes.classic.graphframe import GraphFrame as GraphFrameClassic
from graphframes import GraphFrame

graph = GraphFrame._from_impl(GraphFrameClassic(vertices, edges))

This should work if you can anyhow install on the claster the dependency. I guess this guide can help. You need to add only one dependency: io.graphframes::graphframes-spark4:0.11.0

P.S. I'm still thinking that it is not a bug but expected behavior. If your environment is SparkConnect you need to use the implementation of the GraphFrames Client for SparkConnect. And using it require to add a server-side SparkConnect plugin. GraphFrames porject follows the only official way of extending the SparkConnect protocol via RelationPlugin as it is described in Apache Spark documentation: https://spark.apache.org/docs/4.0.0/app-dev-spark-connect.html#1-spark-connect-protocol-extension

So, it is not a bug but mostly a Databricks Rubntime specific story imo.

P.P.S. I will add an option to override the behavior and enforce classic (py4-based) as an advanced-user flag (with default behavior as it is now).

P.P.P.S. cc: @alexott @dmatrix @WeichenXu123 @joelrobin18 because it is related to Databricks Runtime and I'm not aware of some internal implementation details of it.

[SemyonSinchenko]

g.degrees.show() or g.inDegrees.show() or g.outDegrees.show() are working normally

Simple things like degrees are implented in pure PySpark instead of wrapping them to the protocol extension plugin. But all the algorithms are implemented in the core and PySpark API has only bindings.

[olbapjose]

Thanks a lot, but I am not sure how I can add io.graphframes::graphframes-spark4:0.11.0. As far as I know, Databricks Free Edition offers no way to add JAR dependencies. I can install Python packages via !pip install from a notebook cell, but not sure how I can do the same with any non-python dependency.

[SemyonSinchenko]

Does this guide works https://docs.databricks.com/aws/en/libraries/package-repositories#maven-or-spark-package on a free edition?

[olbapjose]

"In the Library Source button list, select Maven." -> the Library source button only exists in Databricks on Azure/AWS/GCP but not in Free Edition.

[SemyonSinchenko]

"In the Library Source button list, select Maven." -> the Library source button only exists in Databricks on Azure/AWS/GCP but not in Free Edition.

I'm sorry to say this but it looks like there is no way to install the latest GraphFrames on a DBX free edition... At least I do not see such an option from the DBX documentation.

[SemyonSinchenko]

I'm not sure but but maybe init_scripts are available in Free Edition? If so, it should be straight-forward to add a SparkConnect plugin and the JAR