Repository navigation
[flink][cdc] PostgreSQL CDC corrupts binary field values - #10973
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #10972.
PostgreSQL CDC currently converts plain binary values from Base64 through a Java string, silently replacing bytes that are invalid in the default charset. For example,
00 FF 80 41becomes00 EF BF BD EF BF BD 41on a UTF-8 JVM.Preserve Base64 values in snapshot and WAL records and attach the existing binary-field markers so the sink restores the original bytes. Preserve computed-column text semantics and existing BIT/DECIMAL handling. Share binary-field selection between the PostgreSQL and MySQL parsers.
Tests
Add eight PostgreSQL regression tests covering arbitrary bytes, null and empty values, snapshot/update/delete records, marker isolation, logical types, computed columns, existing text columns, and table write/read round trips. Six of these tests fail with the previous PostgreSQL parser