Add optional native Lance scan support

[wirybeaver]

Note

Depends on #5728; please review and merge the build gate PR #5728 first.

Which issue does this PR close?

Part of #4632.

Rationale for this change

Lance tables are planned by Lance Spark. Comet needs an optional integration that can recognize those V2 scans, consume a stable native-read descriptor, and serialize the assigned work without adding Lance-specific code or dependencies to default builds.

The Lance Spark descriptor contract is proposed in lance-format/lance-spark#624. The architecture diagram in #4632 shows how Lance Spark planning, Comet's contrib SPIs, plan-data injection, protobuf, and the future Rust Lance reader fit together.

What changes are included in this PR?

This stacked prototype adds the Spark-side Lance integration on top of #5728:

• Adds the opt-in contrib-lance Maven profile. Lance Scala sources and ServiceLoader resources are absent from default builds.
• Uses the existing CometScanContrib and PlanDataInjector SPIs from feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2] #4952; this PR does not add another core SPI.
• Reflectively detects Lance Spark LanceScan and calls the proposed nativeScanPlan() descriptor, so default Comet builds have no compile-time Lance Spark dependency.
• Serializes dataset URI, resolved version, projection, filters, limits, storage options, and fragment splits into the typed Lance protobuf payload.
• Injects per-partition Lance split data through CometScanWithPlanData.
• Adds spark.comet.scan.lanceNative.enabled=false and preserves Spark fallback when the profile, config, descriptor, or scan shape is unsupported.

This PR intentionally stops at planning and serialization. The native entry point from #5728 remains an inert NotImplemented stub and neither PR adds the Lance Rust dependency. The actual Rust reader, dependency-version alignment, Arrow C Data packaging, and end-to-end native execution will be follow-up work after the integration boundary is agreed on.

How are these changes tested?

Build-gate checks from #5728:

• cargo check -p datafusion-comet --locked
• cargo check -p datafusion-comet --features contrib-lance --locked
• cargo check -p datafusion-comet --no-default-features --locked
• cargo check -p datafusion-comet --no-default-features --features contrib-lance --locked
• cargo clippy -p datafusion-comet --no-default-features --features contrib-lance --locked -- -D warnings
• cargo fmt --all -- --check
• git diff --check

Spark 4.1 contrib checks:

./mvnw test -Dtest=none \
  -Dsuites=org.apache.comet.rules.CometScanRuleSuite \
  -Pspark-4.1,contrib-lance \
  -Dscalastyle.skip=true

The six selected tests pass, including the Lance config gate and descriptor-to-protobuf serialization with fragment splits. cargo check --workspace and cargo check -p datafusion-comet --features contrib-lance --locked also pass on the stacked branch.

Native Lance execution is not tested in this PR because the build-gated native planner deliberately returns NotImplemented.

[andygrove]

Thanks @wirybeaver. I plan on reviewing this next week.

[parthchandra]

@wirybeaver thank you for this contribution and sorry for not getting to this sooner. The general direction we are now recommending for Comet data sources is to add them in a contrib directory until they are mature and/or have regular maintainers who can maintain them.
To that end, @schenksj 's PR #4700 merged an SPI (CometScanWithPlanData trait + ServiceLoader-based PlanDataInjector discovery + generalized foreachUntilCometInput / findAllPlanData) specifically so that contribs like this one don't need to touch core. This PR predates that merge so it still wires Lance in the old way

— below are the concrete changes to adopt the SPI, significantly shrinking the core footprint.

---

1. Delete CometLanceNativeScanLike — use CometScanWithPlanData instead

CometLanceNativeScanLike duplicates what the now-merged CometScanWithPlanData trait already provides (sourceKey, commonData, perPartitionData, plus optional dynamicPruningFilters / withDynamicPruningFilters).

In CometLanceNativeScanExec (contrib), change:

// Before
extends CometLeafExec with CometLanceNativeScanLike

// After
extends CometLeafExec with CometScanWithPlanData

Delete CometLanceNativeScanLike.scala from core entirely.

This single change makes the operators.scala modifications unnecessary:

• foreachUntilCometInput now matches case _: CometLeafExec as its first arm — any CometLeafExec is recognized as an input boundary so no enumeration entry is needed.
• findAllPlanData has a generic arm case s: CometLeafExec with CometScanWithPlanData => ... that calls ensureSubqueriesResolved() and collects data keyed by sourceKey and no Lance-specific case is needed.

Delete both the _: CometLanceNativeScanLike and case lance: CometLanceNativeScanLike additions from operators.scala.

---

2. Move LancePlanDataInjector to contrib via ServiceLoader

The merged SPI discovers contrib PlanDataInjectors via java.util.ServiceLoader. Instead of adding LancePlanDataInjector to the hardcoded injectors list in core:

1. Move LancePlanDataInjector into spark/src/contrib-lance/scala/... (make it a class, not an object, so ServiceLoader can instantiate it via no-arg ctor).
2. Add a service file at spark/src/contrib-lance/resources/META-INF/services/org.apache.spark.sql.comet.PlanDataInjector containing:
   org.apache.comet.lance.LancePlanDataInjector
3. Remove the LancePlanDataInjector definition and registry entry from operators.scala.

---

3. Move LanceIntegration + CometScanRule hook to contrib

spark/src/main/scala/org/apache/comet/lance/LanceIntegration.scala is a reflection bridge in core. The pattern established by the Delta SPI is: core does not reference contrib, not even reflectively.

Recommended approach:

• Keep COMET_LANCE_NATIVE_ENABLED in CometConf.scala (config entries in core are fine — Iceberg does this too).
• Move scan detection (isLanceScan, nativeScanPlan, tryCreateNativeScan) into contrib, e.g. org.apache.comet.lance.LanceScanRuleExtension.
• For the CometScanRule hook: define a tiny trait CometScanContrib { def tryTransform(scanExec: BatchScanExec): Option[SparkPlan] } in core, discover implementations via ServiceLoader in CometScanRule at the case scanExec: BatchScanExec => match before the Iceberg arm, and let contrib
  provide the implementation. This way core gets a ~3-line generic dispatch, not a Lance-specific case.

Delete LanceIntegration.scala from core and the LanceIntegration.isLanceScan / tryCreateNativeScan case from CometScanRule.scala.

---

4. What remains in core after these changes

File	What stays
CometConf.scala	COMET_LANCE_NATIVE_ENABLED config entry
operator.proto	LanceScan, LanceScanCommon, LanceScanPartition messages + lance_scan = 118
planner.rs	OpStruct::LanceScan arm with #[cfg(feature = "contrib-lance")] gate
operators/lance_scan.rs	Rust LanceScanExec behind #[cfg(feature = "contrib-lance")]
operators/mod.rs	Feature-gated mod lance_scan + pub use
operator_registry.rs	LanceScan variant in the enum
jni_api.rs	OpStruct::LanceScan(_) => "LanceScan" name
pom.xml / spark/pom.xml	contrib-lance profile + source dir wiring

Everything else (LanceIntegration.scala, CometLanceNativeScanLike.scala, LancePlanDataInjector, the operators.scala modifications to foreachUntilCometInput / findAllPlanData, the Lance case in CometScanRule) moves to spark/src/contrib-lance/.

---

5. MSRV bump

The PR bumps rust-version from 1.88 to 1.92 for the entire workspace. If the Lance crate requires 1.92, gate it behind the feature flag or pin a lance rev that compiles on 1.88.

---

The net/desired result: a default build (no -Pcontrib-lance) that sees only the config entry and proto message on the Scala side, plus feature-gated-dead Rust code — identical to the Delta SPI pattern.

[wirybeaver]

@parthchandra Thanks for your point out. I will move the Lance comet connector to the contrib directory

[parthchandra]

@wirybeaver Please keep an eye on - #4952. We are trying to converge to a common framework that will work for both Delta and Lance and work in that PR will affect your work.

[wirybeaver]

@wirybeaver Please keep an eye on - #4952. We are trying to converge to a common framework that will work for both Delta and Lance and work in that PR will affect your work.

Thanks for the remind

[wirybeaver]

@schenksj Thanks for merging the Delta Lake. I will refactor this Lance PR this weekend

[wirybeaver]

geodatafusion / lance hasn't upgraded to datafusion-comet's datafusion version 55.0 and Arrow version 59.2 version.