Skip to content

Update 26.2 (client compatibility, Iceberg translation) single sourcing for Cloud #1857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kbatuigas wants to merge 3 commits into
base: main
Choose a base branch
from
+12 −19
Open

Update 26.2 (client compatibility, Iceberg translation) single sourcing for Cloud #1857

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3f9929e
26.2 Cloud: decoded keys and headers (Iceberg)
kbatuigas Jul 30, 2026
e9eedff
DOC-2311
kbatuigas Jul 30, 2026
e0a2c28
DOC-2290
kbatuigas Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 0 additions & 15 deletions modules/develop/pages/kafka-clients.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,7 @@ Use this reference to:

== Kafka client compatibility

ifndef::env-cloud[]
Redpanda validates the Apache Kafka Java client and a set of widely used non-Java clients, at their current versions that support Kafka 4.x, using the ducktape and chaos test suites. Validation confirms connectivity and correctness across core Kafka APIs, such as produce, consume, and transaction operations, at current client versions. Modern clients auto-negotiate protocol versions or use an earlier protocol version accepted by Redpanda brokers.
endif::[]
ifdef::env-cloud[]
Clients developed for Kafka versions 0.11 or later are compatible with Redpanda. The Apache Kafka Java client at version 4.x is validated against Redpanda using the ducktape and chaos test suites; other clients in the table here remain compatible with the Kafka protocol versions they have always supported. Modern clients auto-negotiate protocol versions or use an earlier protocol version accepted by Redpanda brokers.
endif::[]

TIP: Always use the latest supported version of a Kafka client.

Expand All @@ -39,25 +34,15 @@ The following clients have been validated with Redpanda.
| https://github.com/confluentinc/librdkafka[librdkafka^]

| Go
ifndef::env-cloud[]
a|
* https://github.com/twmb/franz-go[franz-go^]
* https://github.com/confluentinc/confluent-kafka-go[confluent-kafka-go^]
* https://github.com/IBM/sarama[Sarama^]
endif::[]
ifdef::env-cloud[]
| https://github.com/twmb/franz-go[franz-go^]
endif::[]

| Python
ifndef::env-cloud[]
a|
* https://github.com/dpkp/kafka-python[kafka-python^]
* https://github.com/confluentinc/confluent-kafka-python[confluent-kafka-python^]
endif::[]
ifdef::env-cloud[]
| https://github.com/dpkp/kafka-python[kafka-python^]
endif::[]

| Rust
| https://github.com/kafka-rust/kafka-rust[kafka-rust^]
Expand Down
16 changes: 12 additions & 4 deletions modules/manage/pages/iceberg/specify-iceberg-schema.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,10 +120,14 @@ value_schema_latest:subject=<subject-name>,protobuf_name=<protobuf-message-full-
+
NOTE: If you don't specify the fully qualified Protobuf message name, Redpanda pauses the data translation to the Iceberg table until you fix the topic misconfiguration.

ifndef::env-cloud[]
== Configure key, value, and header translation

ifndef::env-cloud[]
For Redpanda clusters version 26.2 and later, in addition to the <<supported-iceberg-modes,supported modes>>, `redpanda.iceberg.mode` also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes are shorthand for common combinations of these sections (see <<iceberg-mode-shorthands>>).
endif::[]
ifdef::env-cloud[]
In addition to the <<supported-iceberg-modes,supported modes>>, `redpanda.iceberg.mode` also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes are shorthand for common combinations of these sections (see <<iceberg-mode-shorthands>>).
endif::[]

The `key` and `headers` sections change fields inside the `redpanda` system struct column (`redpanda.key` and the `value` field of each entry in `redpanda.headers`), while the `value` section changes the columns outside that struct. See <<how-iceberg-modes-translate-to-table-format,How Iceberg modes translate to table format>> for the base row structure that every generated table includes.

Expand Down Expand Up @@ -369,6 +373,7 @@ Redpanda rejects the following configurations when you create or alter a topic:
NOTE: Option values cannot contain `,` or `;`, and whitespace is not trimmed. Avoid extra spaces around subject names or Protobuf message names.

// Broker-upgrade gating not applicable in RP Cloud
ifndef::env-cloud[]
[IMPORTANT]
====
Configuring anything beyond `key:mode=binary` and `headers:value_type=binary` requires every broker in the cluster to be running Redpanda version 26.2 or later. Until all brokers are upgraded, Redpanda rejects these configurations with an error.
Expand All @@ -380,6 +385,7 @@ The following configurations require all brokers to be running version 26.2 or l
* `value:layout=nested`
* `value:mode=string`
====
endif::[]

=== Example configurations

Expand Down Expand Up @@ -423,17 +429,20 @@ rpk topic describe orders -c | grep redpanda.iceberg.mode
----

NOTE: If a section-based configuration is equivalent to one of the modes described in <<supported-iceberg-modes,Supported Iceberg modes>>, `rpk topic describe` displays it using that mode's name instead of the section syntax.
endif::[]

ifndef::env-cloud[]
[[resolve-schemas-within-a-context]]
== Resolve schemas within a Schema Registry context

If you use xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry contexts] to isolate schemas (for example, by environment or tenant), set the `redpanda.schema.registry.context` topic property to bind the topic to that context. Redpanda then resolves the schemas referenced by records in the topic against the configured context instead of the default context (`.`). Depending on the topic's Iceberg mode, Redpanda looks up either the schema ID embedded in each record (`value_schema_id_prefix` mode) or the latest schema for a subject (`value_schema_latest` mode) within that context.

@kbatuigas kbatuigas Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you use xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry contexts] to isolate schemas (for example, by environment or tenant), set the `redpanda.schema.registry.context` topic property to bind the topic to that context. Redpanda then resolves the schemas referenced by records in the topic against the configured context instead of the default context (`.`). Depending on the topic's Iceberg mode, Redpanda looks up either the schema ID embedded in each record (`value_schema_id_prefix` mode) or the latest schema for a subject (`value_schema_latest` mode) within that context.
If you use xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry contexts] to isolate schemas (for example, by environment or tenant), set the `redpanda.schema.registry.context` topic property to bind the topic to that context. Redpanda then resolves the schemas referenced by records in the topic against the configured context instead of the default context (`.`). This applies to all schema-decoding modes, for both keys and values: the `value_schema_id_prefix` and `value_schema_latest` shorthand modes, and any `key` or `value` section that uses `mode=schema_id_prefix` or `mode=schema_latest`.

@wdberkeley does this change look good per this bot feedback?


Both modes rely on a schema registered in Schema Registry to determine the Iceberg table structure.

ifndef::env-cloud[]
Starting in Redpanda 26.2, Schema Registry contexts are enabled by default. See xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry contexts] to learn about contexts and qualified subject naming before you configure this property.
endif::[]
ifdef::env-cloud[]
Schema Registry contexts are enabled by default. See xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry contexts] to learn about contexts and qualified subject naming before you configure this property.
endif::[]
Comment on lines +440 to +445

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Document Schema Registry context behavior for key decoding.

This section only names the legacy value modes, while this page now also supports key:mode=schema_id_prefix and key:mode=schema_latest. Clarify whether key schema lookups use the configured context, and document the behavior if they do.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@modules/manage/pages/iceberg/specify-iceberg-schema.adoc` around lines 440 -
445, Update the Schema Registry context guidance in the env-cloud and non-cloud
documentation blocks to explicitly describe whether key decoding modes
schema_id_prefix and schema_latest resolve schemas using the configured Schema
Registry context; if they do, document that lookup behavior alongside the
existing context and qualified-subject explanation.


.Set the Schema Registry context on a new topic
[,bash]
Expand Down Expand Up @@ -462,7 +471,6 @@ To change the Schema Registry context on a topic that is actively translating to
. Change `redpanda.schema.registry.context`.
. Re-enable Iceberg translation.
====
endif::[]

== How Iceberg modes translate to table format

Expand Down
Loading