Skip to content

Commit 99f77aa

Browse files
J2TeamNNLdatlechin
andauthored
docs(viewer): Protobuf schemas and the Kafka Schema Registry (#155)
Co-authored-by: Ngo Quoc Dat <datlechin@gmail.com>
1 parent c6ece26 commit 99f77aa

2 files changed

Lines changed: 22 additions & 6 deletions

File tree

‎databases/kafka.mdx‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ record format every message has used since.
2828
| **SASL Mechanism** | `PLAIN`, `SCRAM-SHA-256`, or `SCRAM-SHA-512`, for the two SASL protocols |
2929
| **Username** / **Password** | The SASL credentials |
3030
| **Broker Addresses** | Whether to dial the addresses the cluster advertises, or stay on the bootstrap. Leave this on **Use the addresses the cluster advertises** unless those addresses are unreachable from this Mac |
31+
| **Schema Registry URL** | Optional. The Confluent Schema Registry that **Display As > Schema Registry** fetches schemas from, such as `http://localhost:8081` |
32+
| **Schema Registry User** / **Schema Registry Password** | Optional. Basic authentication for the registry, such as a Confluent Cloud API key and secret |
3133

3234
There is no Database field. A connection is one cluster, and the sidebar shows its topics directly.
3335
Set the whole attempt's **Connect timeout** under **Options**; an empty field uses 30 seconds.
@@ -150,10 +152,18 @@ through the log, one page older each time.
150152
Row counts are approximate. The count is the end offset minus the start offset, which overcounts
151153
where retention has removed messages from the middle of a compacted topic.
152154

153-
Schema Registry is not read. A binary `value` column of Protobuf messages decodes field by field
154-
with **Display As > Protobuf**, which reads the five-byte Confluent wire-format prefix and shows the
155-
schema ID; see [Binary value viewer](/features/json-viewer#binary-value-viewer). Avro values decode
156-
with **Display As > Avro Schema…** and the topic's `.avsc` file, past the same prefix.
155+
With a **Schema Registry URL** on the connection, set **Display As > Schema Registry** on the
156+
`value` or `key` column. Each value names its schema by the ID in its five-byte Confluent wire-format
157+
prefix. TablePro fetches that Avro, Protobuf or JSON schema, with any files a Protobuf schema imports,
158+
and keeps it until you quit. A cell shows the stored bytes until its schema arrives, then redraws. A
159+
schema that cannot be fetched leaves the cell as hex, and the viewer shows the registry's reason;
160+
**Refresh** or opening the value asks again. A change to the registry settings applies from the next
161+
connect. The registry is reached directly from this Mac, not through the connection's SSH tunnel.
162+
163+
Without a registry, a `value` column of Protobuf messages decodes field by field with **Display As >
164+
Protobuf**, or with field names through **Display As > Protobuf Schema…** and the topic's `.proto`
165+
file. Avro values decode with **Display As > Avro Schema…** and the topic's `.avsc` file. Each reads
166+
past the same prefix; see [Binary value viewer](/features/json-viewer#binary-value-viewer).
157167

158168
Creating a topic, editing topic configuration, and resetting consumer group offsets are not
159169
available. Deleting a topic is, as the paragraph above describes.

‎features/json-viewer.mdx‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Cell and Row Viewers
3-
description: View and edit JSON, PHP serialized, image and binary cell values, decode MessagePack, Protobuf, igbinary and Avro, set per-column display formats, and inspect whole rows in the sidebar
3+
description: View and edit JSON, PHP serialized, image and binary cell values, decode MessagePack, Protobuf, igbinary and Avro, with schemas from a file or a Schema Registry, set per-column display formats, and inspect whole rows in the sidebar
44
---
55

66
Click the chevron on a JSON or blob cell to open its value in a popover anchored to that cell. Which viewer opens follows the column: its declared type, or the **Display As** choice on its header, and that choice changes rendering only, never what is stored.
@@ -23,6 +23,8 @@ Right-click a column header and pick **Display As**. The submenu lists the forma
2323
| PHP Serialized | text | Opens the cell in the PHP serialized viewer |
2424
| MessagePack, Protobuf, igbinary | binary | Decodes the bytes and opens the cell in the [binary value viewer](#binary-value-viewer) |
2525
| Avro Schema… | binary | Asks for the `.avsc` file the values were written with, then decodes like the formats above |
26+
| Protobuf Schema… | binary | Asks for the `.proto` file or descriptor set the values were written with, and the message type, then shows field names |
27+
| Schema Registry | binary, text | Fetches each value's schema from the connection's Schema Registry. Offered when the connection names one |
2628

2729
An override persists per column, scoped to the connection and table. A column with a format set copies in that format when you copy the single cell; a block of cells or a whole row always copies the stored value.
2830

@@ -102,7 +104,7 @@ An array or object row copies as its summary, such as `[2 items]`, not as its co
102104

103105
## Binary value viewer
104106

105-
Set **Display As > MessagePack**, **Protobuf** or **igbinary** on a binary column, or pick **Avro Schema…** and choose the column's schema file. The grid shows each value decoded on one line, and double-click or `Enter` opens the viewer. It is read-only. On an editable table the chevron still opens the hex editor, and so does double-clicking a NULL cell.
107+
Set **Display As > MessagePack**, **Protobuf** or **igbinary** on a binary column, pick **Avro Schema…** or **Protobuf Schema…** and choose the column's schema file, or pick **Schema Registry** on a connection that names one. The grid shows each value decoded on one line, and double-click or `Enter` opens the viewer. It is read-only. On an editable table the chevron still opens the hex editor, and so does double-clicking a NULL cell.
106108

107109
| Mode | MessagePack, Avro | Protobuf | igbinary |
108110
| --- | --- | --- | --- |
@@ -112,8 +114,12 @@ Set **Display As > MessagePack**, **Protobuf** or **igbinary** on a binary colum
112114

113115
Protobuf without its schema has no field names or types. A length-delimited field reads as a nested message when its bytes parse as one, else as text when they are UTF-8, else as escaped bytes, so a short string can read as a message. A value that starts with a Schema Registry frame, as Kafka producers write it, shows its schema ID and message index above the tree.
114116

117+
**Protobuf Schema…** takes a `.proto` file or a descriptor set from `protoc --descriptor_set_out`. Use a descriptor set built with `--include_imports` when the message uses types from other files: a `.proto` file is read alone, so a field of an imported type shows by number. When the file defines more than one message, TablePro asks which one the column holds, and the menu names the format after it, as in **Protobuf (Order)**. The tree shows each field by name, badged with its declared type, enums by value name, and maps as `key` and `value` entries. **Text** mode prints the value as `protoc --decode` does, except that UTF-8 text keeps its characters. A field the schema does not declare, or declares with another wire type, shows by number as it would without a schema. TablePro keeps a copy of the file, as it does for Avro.
118+
115119
Avro reads each value as one datum written with the chosen schema. A record shows its fields in schema order, a union shows the branch it holds, and logical types read as what they stand for: `decimal` as a number, `uuid`, `date`, `time` and `duration` as text, `timestamp` in UTC, `local-timestamp` without a zone. The menu names the column's format after the schema's root type, as in **Avro (Order)**. TablePro keeps a copy of the schema, so moving or deleting the file later changes nothing; pick **Avro Schema…** again to replace it. A value with the Kafka wire-format prefix decodes past it, and the viewer shows its schema ID. A value in Avro's single-object encoding, as Java's `toByteBuffer()` writes it, decodes when its schema fingerprint matches the chosen schema, and the viewer shows the fingerprint.
116120

121+
**Schema Registry** reads the schema ID from each value's wire-format prefix and fetches the schema from the registry named in the connection's settings, see [Kafka](/databases/kafka). Avro and Protobuf values decode as above, a Protobuf message picked by the message index in the prefix, and JSON Schema values open in the JSON tree. A cell shows the stored bytes while its schema is being fetched. A value with no prefix, or whose schema the registry does not return, stays hex, and the viewer says why.
122+
117123
A value that does not decode stays hex in the grid, and the viewer says where it broke above its hex dump. Copying a single cell copies the decoded text, cut at 40,000 characters. **Find** searches the decoded text.
118124

119125
## Blob and hex

0 commit comments

Comments
 (0)