[HUDI-9666] Auto-deduce ComplexKeyGenerator single-field record-key encoding on branch-0.x

[ad1happy2go]

Change Logs

For a ComplexKeyGenerator with a single record-key field, the _hoodie_record_key encoding changed across releases (HUDI-7001): 0.14.0 and older wrote field:value, while 0.14.1/0.15.0/1.0.0–1.0.2 wrote the bare value. A table written by one release and then written by another silently mixes both formats, so upserts stop matching existing records and create duplicates.

The current guard (#17834) fails all single-field complex-keygen writes by default until the user manually picks an encoding via hoodie.write.complex.keygen.new.encoding. This PR makes the writer self-healing instead:

• Auto-deduction (new config hoodie.write.complex.keygen.auto.deduce.encoding, default true): on write initialization, read one stored _hoodie_record_key from an existing base file, detect the table's actual encoding, and pin it on the write config so upserts keep matching.
• Caching: the deduced value is stored in .hoodie/.aux/complex_key_encoding; subsequent writes read the cache instead of the parquet footer.
• New/empty tables: default to the new encoding (bare value) instead of failing the first write.
• The validation guard still applies when auto-deduction is explicitly disabled; readers are never affected.

Impact

Single-field ComplexKeyGenerator writes work out of the box across upgrades (no manual encoding config, no failed pipelines), while preserving the existing table's key format and preventing duplicate records.

Risk level

medium

Verified by three new functional test suites (auto-deduction of both encodings, existing-table upgrade safe/danger paths, new-table defaults). Also validated end-to-end with bundle jars against tables created by genuine 0.14.0 (field:value) and 0.14.1 (bare value) releases: encoding detected and preserved, 100/100 records upserted in place, zero duplicates.

Documentation Update

Config docs for hoodie.write.complex.keygen.new.encoding, hoodie.write.complex.keygen.auto.deduce.encoding, and hoodie.write.complex.keygen.validation.enable are updated in HoodieWriteConfig. The deployment-guide section on the complex key generator should be updated to mention auto-deduction as the default behavior.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

🤖 Generated with Claude Code

[hudi-bot]

CI report:

• a83d686 UNKNOWN
• aeb6e8c UNKNOWN

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build