feat(flink): enable LSM end-to-end for Flink by default

[cshuo]

Describe the issue this Pull Request addresses

Flink did not enable the LSM tree storage layout end to end by default, leaving write-handle selection, table initialization, and read paths dependent on legacy behavior. This change makes LSM the default for applicable Flink operations while retaining the default layout for inserts and explicit legacy-layout compatibility cases.

This PR applies LSM only to data tables. Enabling LSM for the metadata table (MDT) is deferred to a follow-up PR.

Summary and Changelog

• Select LSM-aware Flink merge handles whenever the table uses the LSM tree storage layout.
• Resolve LSM as the default table storage layout except for insert, and persist the storage layout and log file format during table initialization.
• Route Flink reads through FormatUtils.createRecordReader, selecting the LSM reader for supported native log-file cases.
• Extend native LSM log iteration and schema-evolution handling for writer-schema projection and instant-specific schema resolution.
• Add and update unit/integration coverage for layout resolution, table initialization, reader selection, write handles, upgrade compatibility, and expected merge semantics.

Impact

• Functional impact: New applicable Flink tables default to LSM storage and use LSM-aware read/write paths; insert operations retain the default storage layout.
• Maintainability: Consolidates reader creation behind HoodieRecordReader and shares schema-evolution transformation logic between inline and native log blocks.
• Extensibility: Persists table storage-layout and log-file-format settings through HoodieTableMetaClient.TableBuilder, enabling consistent LSM table creation.

Risk Level

medium. The default storage layout changes across Flink write and read paths, with compatibility-sensitive tests explicitly retaining the default layout for older table versions. The commit adds focused unit and integration test coverage for the new selection and initialization behavior.

Documentation Update

none.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-bot]

CI report:

• 73357a7 UNKNOWN

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build