Agent/databricks zerobus variant

[yorickvanzweeden]

Summary

Hi Vector team,

Databricks released another fix for their Zerobus Ingestion SDK. This PR:

• Adds support for Unity Catalog VARIANT columns in the databricks_zerobus sink.
• The implementation converts JSON values into Databricks’ Parquet VARIANT binary representation and recursively supports VARIANT fields nested in structs, lists, fixed-size lists, large lists, and maps.
• This also upgrades the Zerobus SDK to 2.4.0, which includes the Arrow schema support required for VARIANT columns.

Please let me know if you require any changes, or have any questions.

Vector configuration

The sink can be configured as follows:

 [sinks.databricks]
 type = "databricks_zerobus"
 inputs = ["events"]
 ingestion_endpoint = "https://<workspace-id>.zerobus.<region>.cloud.databricks.com"
 unity_catalog_endpoint = "https://<workspace>.cloud.databricks.com"
 table_name = "<catalog>.<schema>.<table>"
 
 [sinks.databricks.auth]
 strategy = "oauth"
 client_id = "${DATABRICKS_CLIENT_ID}"
 client_secret = "${DATABRICKS_CLIENT_SECRET}"

When using ${...} environment variables, Vector environment interpolation must be explicitly enabled or replaced with an appropriate secret-injection mechanism.

How did you test this PR?

• Ran cargo fmt --all -- --check.
• Ran focused VARIANT codec tests: 3 passed.
• Ran the full Arrow codec test suite: 16 passed.
• Ran Databricks Zerobus sink tests: 39 passed.
• Ran Clippy with -D warnings for the codec and Zerobus sink.
• Built Vector with databricks-zerobus-ingest-sdk 2.4.0.
• Tested top-level VARIANT object encoding.
• Tested nested VARIANT values in structs, lists, and maps.
• Tested invalid VARIANT JSON handling.
• Tested preservation of VARIANT child nullability.
• Regenerated third-party license metadata after the dependency updates.

Additionally, I ran an internal test against our Databricks instance.

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Related: [Rust] Fix variant Arrow schema databricks/zerobus-sdk#525