Decide how the Delta Lake contrib jar is built, versioned, and published in releases

[andygrove]

Context

#5365 adds the Delta Lake scan contrib as a separate Maven module (contrib/delta-spark) that only enters the reactor under the delta profile and produces its own jar, comet-contrib-delta-spark<version>_<scala>. The user guide and the module README tell users to put that jar on the classpath alongside delta-spark. This issue is for deciding how that jar reaches users, and more generally how table-format integrations are packaged and released, since Delta and Iceberg currently take different shapes.

How the release tooling behaves today

• dev/release/build-release-comet.sh runs ./mvnw install into a temporary local repository for six profile combinations (Spark 3.4 and 3.5 with Scala 2.12 and 2.13, then Spark 4.0 and 4.1). None of those invocations pass -Pdelta, so the contrib module is never built during a release.
• dev/release/publish-to-maven.sh does not run mvn deploy. It signs and uploads every .jar and .pom found under org/apache/datafusion in that local repository. Two things follow:
  • maven.deploy.skip has no effect on what gets published. Whether the contrib jar ships depends only on whether -Pdelta is on the install lines.
  • Anything install produces is published. feat: add native Delta Lake scan contrib module (page/row-group pruning) #5365 adds an unconditional test-jar execution to spark/pom.xml (about 6.7 MB per profile) so that the contrib suites can reuse CometTestBase. As written it would ship six -tests.jar artifacts per release.
• The native library needs no release change. The release containers build with default Cargo features, and the delta feature (deletion-vector decoding plus the planner arm) is in the default set, so the released libcomet already carries everything the contrib jar needs.

Questions to settle

1. Should the contrib jar be published at all? If yes, -Pdelta goes on the Spark 3.5, 4.0 and 4.1 install lines (the 3.4 pairing is delta-core 2.4 and does not compile). If no, the docs should say build-from-source is the only route and the maven.deploy.skip=false override in the contrib pom should go. An experimental feature that only exists if users build from source is unlikely to get real-workload testing.
2. What Delta versions does each published jar support? Each jar compiles against one Delta release per Spark line (3.3.2, 4.0.1, 4.3.1). Nothing verifies binary compatibility across Delta minors, and pom.xml already records that Delta 4.1.0 and 4.2.0 are incompatible with Spark 4.1.3. Publishing implies a support statement in the user guide.
3. Should the spark test-jar be published? Options: bind the test-jar execution inside the delta profile so it only exists when the contrib is built, or accept publishing it (there is partial precedent in the -test-sources.jar).
4. Why is Delta packaged differently from Iceberg, and should it stay that way? Iceberg support lives inside comet-spark: iceberg-spark-runtime is a test-scope dependency only and IcebergReflection addresses every Iceberg class by name, so there is no profile, no separate jar, and no dependency on a specific Iceberg release. The Delta contrib compiles against delta-spark internals (DeltaParquetFileFormat, DeletionVectorDescriptor, DeltaColumnMapping), which forces a provided dependency, one artifact per Spark-and-Delta pairing, and rules out Spark 3.4. That structural difference explains the module, the profile, the separate CI workflow and the opt-in config. The two consistent end states are (a) fold Delta into comet-spark behind reflection like Iceberg, or (b) keep the contrib shape and finish it (publish the jar, document the Delta pairing, gate the test-jar). Deciding this on purpose also settles part of the core-versus-contrib boundary question raised in the feat: add native Delta Lake scan contrib module (page/row-group pruning) #5365 review and in Converge the two Delta read paths into one plugin (clean architecture + performance) #5411.

Related

• feat: add native Delta Lake scan contrib module (page/row-group pruning) #5365 (the contrib PR)
• Explore integration with Delta Lake #174 (Delta Lake support)
• Converge the two Delta read paths into one plugin (clean architecture + performance) #5411 (convergence of the two Delta read paths)