Skip to content

feat: add multi-version Flink module structure (1.18/1.19/1.20)#48

Open
fightBoxing wants to merge 1 commit into
lance-format:mainfrom
fightBoxing:chore/multi-version-skeleton
Open

feat: add multi-version Flink module structure (1.18/1.19/1.20)#48
fightBoxing wants to merge 1 commit into
lance-format:mainfrom
fightBoxing:chore/multi-version-skeleton

Conversation

@fightBoxing
Copy link
Copy Markdown
Collaborator

Summary

Convert the project to a multi-module Maven build supporting 3 actively-supported Flink versions: 1.18.1 / 1.19.1 / 1.20.0. All modules share one source tree under root src/, so per-version forks of the source code are not needed.

This is a clean re-slice of #17 — it drops the 4 PR #14 catalog commits (which will land via their own PR) and removes the originally-included lance-flink-1.17 module (rationale below).

Closes #17.

Module Layout

.
├── pom.xml                  ← parent (packaging=pom, dependencyManagement)
├── src/                     ← single shared source tree
└── lance-flink-{1.18,1.19,1.20}/
    ├── pom.xml              ← references root src/ via build-helper-maven-plugin
    └── src/test/resources/log4j2-test.xml

Each submodule:

  • Pulls source from root src/ via build-helper-maven-plugin (add-source / add-test-source / add-resource / add-test-resource)
  • Overrides flink.version to its target (${flink118.version} / ${flink119.version} / ${flink120.version})
  • Inherits all plugin & dependency versions from parent dependencyManagement
  • Compiles to Java 8 bytecode

Plugin versions kept aligned with main (#39):

  • maven-compiler-plugin 3.15.0
  • maven-shade-plugin 3.6.2
  • maven-jar-plugin 3.5.0
  • maven-source-plugin 3.4.0
  • jacoco-maven-plugin 0.8.14

Why no lance-flink-1.17?

Originally this PR included a lance-flink-1.17 module. Local verification on JDK 8 surfaced two unrelated bugs inside Flink 1.17.x's bundled (shaded) Calcite, both of which prevent TableEnvironment creation:

  1. MetadataDef.<init>:48 — an assert on a generic-type bound that is always false under -ea (default for surefire), throwing AssertionError.
  2. CacheGeneratorUtil$CacheKeyStrategy$1.safeArgList:213IllegalArgumentException: fromIndex(2) > toIndex(0), surfaces even with -ea disabled.

Same source tree + same JDK on Flink 1.18 / 1.19 / 1.20 runs cleanly, confirming this is a Flink-side issue, not a connector issue.

Combined with Flink 1.17 reaching end-of-life on 2024-10, supporting it would require carrying workarounds for an upstream-broken planner on a no-longer-maintained branch. We chose to drop 1.17 to keep the matrix lean.

Verification (local, JDK 8 / Tencent Kona 8.0.14)

mvn validate            BUILD SUCCESS  (4 modules)
mvn compile             BUILD SUCCESS
mvn test-compile        BUILD SUCCESS
mvn package -DskipTests BUILD SUCCESS  (3 × 163 MB shaded jar)
mvn test                                                                 (per module)
  lance-flink-1.18:  Tests run: 178, Failures: 1, Errors: 0, Skipped: 17
  lance-flink-1.19:  Tests run: 178, Failures: 1, Errors: 0, Skipped: 17
  lance-flink-1.20:  Tests run: 178, Failures: 1, Errors: 0, Skipped: 17

The single failure is LanceReadOptimizationsTest$FilterPushDownTests.testInPredicatePushDown — a pre-existing main bug that reproduces identically across every Flink version, unrelated to this PR.

Diff Summary

8 files changed, 1148 insertions(+), 312 deletions(-)
  • New: lance-flink-{1.18,1.19,1.20}/{pom.xml, src/test/resources/log4j2-test.xml}
  • Modified: pom.xml (now parent pom)
  • Modified: LanceDynamicTableSource.java (remove unused TableSchema import)

Out of scope (intentionally excluded from #17)

Convert single-module project to multi-module Maven project, supporting
3 actively-supported Flink versions (1.18.1 / 1.19.1 / 1.20.0) sharing
one source tree under root src/.

Module structure:
  - Parent pom uses <dependencyManagement> for version harmonization
  - Each lance-flink-1.x submodule references root src/ via
    build-helper-maven-plugin (add-source / add-test-source /
    add-resource / add-test-resource)
  - All modules compile with Java 8 target

Plugin versions kept aligned with main (PR lance-format#39):
  - maven-compiler-plugin   3.15.0
  - maven-shade-plugin      3.6.2
  - maven-jar-plugin        3.5.0
  - maven-source-plugin     3.4.0
  - jacoco-maven-plugin     0.8.14

Why 1.17 is excluded:
  - Flink 1.17 reached end-of-life on 2024-10
  - Flink 1.17.x ships an internal shaded Calcite that has known
    bugs on JDK 8 (MetadataDef <init> assertion + safeArgList
    IndexOutOfBounds in CacheGeneratorUtil), causing TableEnvironment
    creation to fail. These are Flink-side issues, unrelated to this
    connector
  - Supporting an EOL version with upstream-broken planner is not
    worth the maintenance cost

Notes:
  - Sliced from feature/flink-multi-version-modules but excludes the 4
    PR lance-format#14 namespace catalog commits (those will land via their own PR)
  - Removes unused TableSchema import in LanceDynamicTableSource
  - Verified locally on JDK 8 (Tencent Kona):
      mvn validate / compile / test-compile / package -> BUILD SUCCESS
      mvn test -> 178 run, 1 failure, 0 errors, 17 skipped (per module)
      The single failure (LanceReadOptimizationsTest$FilterPushDown
      Tests.testInPredicatePushDown) is a pre-existing main bug that
      reproduces identically across 1.18 / 1.19 / 1.20

Closes lance-format#17 in favor of this clean slice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant