Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
3 Skipped Deployments
|
Blargian
approved these changes
Feb 3, 2026
Member
Blargian
left a comment
Blargian
left a comment
There was a problem hiding this comment.
LGTM, left a few suggestions based on Google style + formatting rules flagged by Vale
| - If there's no schema id embedded in the message, it will use the schema id or subject name specified in the ClickPipe configuration to retrieve the schema. | ||
| - If the message is written without an embedded schema id, and no schema id or subject name is specified in the ClickPipe configuration, then the schema will not be retrieved and the message will be skipped with a `SOURCE_SCHEMA_ERROR` logged in the ClickPipes errors table. | ||
| - If the message does not conform to the schema, then the message will be skipped with a `DATA_PARSING_ERROR` logged in the ClickPipes errors table. | ||
| - For Protobuf schemas only, ClickPipes will load any imported schemas defined as dependencies. Avro schemas with external references are not yet supported. |
Member
There was a problem hiding this comment.
Suggested change
| - For Protobuf schemas only, ClickPipes will load any imported schemas defined as dependencies. Avro schemas with external references are not yet supported. | |
| - For Protobuf schemas only, ClickPipes will load any imported schemas defined as dependencies. Avro schemas with external references aren't yet supported. |
| - If the Avro schema contains a field that is not included in the ClickHouse destination mapping, that field is ignored. | ||
| - If the Avro schema is missing a field defined in the ClickHouse destination mapping, the ClickHouse column will be populated with a "zero" value, such as 0 or an empty string. Note that DEFAULT expressions are not currently evaluated for ClickPipes inserts (this is temporary limitation pending updates to the ClickHouse server default processing). | ||
| - If the Avro schema field and the ClickHouse column are incompatible, inserts of that row/message will fail, and the failure will be recorded in the ClickPipes errors table. Note that several implicit conversions are supported (like between numeric types), but not all (for example, an Avro record field can not be inserted into an Int32 ClickHouse column). | ||
| - If the schema contains a field that is not included in the ClickHouse destination mapping, that field is ignored. |
Member
There was a problem hiding this comment.
Suggested change
| - If the schema contains a field that is not included in the ClickHouse destination mapping, that field is ignored. | |
| - If the schema contains a field that isn't included in the ClickHouse destination mapping, that field is ignored. |
|
|
||
| #### Supported Avro Data Types {#supported-avro-data-types} | ||
| ClickPipes supports all Avro Primitive and Complex types, and all Avro Logical types except `time-millis`, `time-micros`, `local-timestamp-millis`, `local_timestamp-micros`, and `duration`. Avro `record` types are converted to Tuple, `array` types to Array, and `map` to Map (string keys only). In general the conversions listed [here](/interfaces/formats/Avro#data-type-mapping) are available. We recommend using exact type matching for Avro numeric types, as ClickPipes does not check for overflow or precision loss on type conversion. | ||
| ClickPipes supports all Avro Primitive and Complex types, and all Avro Logical types except `local-timestamp-millis` and `local_timestamp-micros`. Avro `record` types are converted to Tuple, `array` types to Array, and `map` to Map (string keys only). In general the conversions listed [here](/interfaces/schema-inference#avro) are available. We recommend using exact type matching for Avro numeric types, as ClickPipes does not check for overflow or precision loss on type conversion. |
Member
There was a problem hiding this comment.
Suggested change
| ClickPipes supports all Avro Primitive and Complex types, and all Avro Logical types except `local-timestamp-millis` and `local_timestamp-micros`. Avro `record` types are converted to Tuple, `array` types to Array, and `map` to Map (string keys only). In general the conversions listed [here](/interfaces/schema-inference#avro) are available. We recommend using exact type matching for Avro numeric types, as ClickPipes does not check for overflow or precision loss on type conversion. | |
| ClickPipes supports all Avro Primitive and Complex types, and all Avro Logical types except `local-timestamp-millis` and `local_timestamp-micros`. Avro `record` types are converted to Tuple, `array` types to Array, and `map` to Map (string keys only). In general the conversions listed [here](/interfaces/schema-inference#avro) are available. We recommend using exact type matching for Avro numeric types, as ClickPipes doesn't check for overflow or precision loss on type conversion. |
|
|
||
| - [`DEFAULT`](/sql-reference/statements/create/table#default) is not supported. | ||
| - Individual messages are limited to 8MB (uncompressed) by default when running with the smallest (XS) replica size, and 16MB (uncompressed) with larger replicas. Messages that exceed this limit will be rejected with an error. If you have a need for larger messages, please contact support. | ||
| - Individual messages are limited to 2MB (uncompressed) by default when running with the smallest (XS) replica size, and 8MB (uncompressed) with larger replicas. Messages that exceed this limit will be rejected with an error. If you have a need for larger messages, please contact support. |
Member
There was a problem hiding this comment.
Suggested change
| - Individual messages are limited to 2MB (uncompressed) by default when running with the smallest (XS) replica size, and 8MB (uncompressed) with larger replicas. Messages that exceed this limit will be rejected with an error. If you have a need for larger messages, please contact support. | |
| - Individual messages are limited to 2 MB (uncompressed) by default when running with the smallest (XS) replica size, and 8MB (uncompressed) with larger replicas. Messages that exceed this limit will be rejected with an error. If you have a need for larger messages, [contact support](https://clickhouse.com/support/program). |
|
|
||
| Batches are inserted when one of the following criteria has been met: | ||
| - The batch size has reached the maximum size (100,000 rows or 32MB per 1GB of pod memory) | ||
| - The batch size has reached the maximum size (100,000 rows or 28MB per 1GB of pod memory) |
Member
There was a problem hiding this comment.
Suggested change
| - The batch size has reached the maximum size (100,000 rows or 28MB per 1GB of pod memory) | |
| - The batch size has reached the maximum size (100,000 rows or 28 MB per 1 GB of pod memory) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds specifics for the soon to be supported Protobuf format for Kafka Clickpipes. There are also some minor updates to reflect recent ClickPipes changes.