From 51db65f3e15f644c0449baede0dd7922b5de2fff Mon Sep 17 00:00:00 2001 From: Nico Berlee Date: Fri, 19 Dec 2025 14:26:02 +0100 Subject: [PATCH] output: azure_blob: compression with zstd update Signed-off-by: Nico Berlee --- pipeline/outputs/azure_blob.md | 78 +++++++++++++++++----------------- 1 file changed, 39 insertions(+), 39 deletions(-) diff --git a/pipeline/outputs/azure_blob.md b/pipeline/outputs/azure_blob.md index dc1ca69d2..42c6bc436 100644 --- a/pipeline/outputs/azure_blob.md +++ b/pipeline/outputs/azure_blob.md @@ -16,45 +16,45 @@ Ensure you have an Azure Storage account. [Azure Blob Storage Tutorial (video)]( Fluent Bit exposes the following configuration properties. -| Key | Description | Default | -| :------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------- | -| `account_name` | Azure Storage account name. | _none_ | -| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` | -| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `true` | -| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` | -| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` | -| `blob_uri_length` | Set the length of the generated blob URI used when creating and uploading objects to Azure Blob Storage. | `64` | -| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` | -| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` | -| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` | -| `compress` | Sets payload compression in network transfer. Supported value: `gzip`. | _none_ | -| `compress_blob` | Enables GZIP compression in the final `blockblob` file. This option isn't compatible when `blob_type` = `appendblob`. | `false` | -| `configuration_endpoint_bearer_token` | Bearer token for the configuration endpoint. | _none_ | -| `configuration_endpoint_password` | Basic authentication password for the configuration endpoint. | _none_ | -| `configuration_endpoint_url` | Configuration endpoint URL. | _none_ | -| `configuration_endpoint_username` | Basic authentication username for the configuration endpoint. | _none_ | -| `container_name` | Name of the container that will contain the blobs. | _none_ | -| `database_file` | Absolute path to a database file used to store blob file contexts. | _none_ | -| `date_key` | Key name used to store the record timestamp. | `@timestamp` | -| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` | -| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `false` | -| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ | -| `file_delivery_attempt_limit` | Maximum number of delivery attempts for a file. | `1` | -| `io_timeout` | HTTP IO timeout. | `60s` | -| `part_delivery_attempt_limit` | Maximum number of delivery attempts for a file part. | `1` | -| `part_size` | Size of each part when uploading blob files. | `25M` | -| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ | -| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ | -| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` | -| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ | -| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` | -| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` | -| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` | -| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` | -| `upload_part_freshness_limit` | Maximum lifespan of an uncommitted file part. | `6D` | -| `upload_parts_timeout` | Timeout for uploading parts of a blob file. | `10M` | -| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and haven't reached `upload_file_size` limit yet. | `30m` | -| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | +| Key | Description | Default | +| :------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------- | +| `account_name` | Azure Storage account name. | _none_ | +| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` | +| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `true` | +| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` | +| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` | +| `blob_uri_length` | Set the length of the generated blob URI used when creating and uploading objects to Azure Blob Storage. | `64` | +| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` | +| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` | +| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` | +| `compress` | Sets payload compression in network transfer. Supported values: `gzip`, `zstd`. | _none_ | +| `compress_blob` | Enables compression in the final `blockblob` file. When enabled without `compress`, it uses GZIP; if `compress` is also set, it inherits that codec. This option isn't compatible when `blob_type` = `appendblob`. Fluent Bit returns a configuration error and fails to start. | `false` | +| `configuration_endpoint_bearer_token` | Bearer token for the configuration endpoint. | _none_ | +| `configuration_endpoint_password` | Basic authentication password for the configuration endpoint. | _none_ | +| `configuration_endpoint_url` | Configuration endpoint URL. | _none_ | +| `configuration_endpoint_username` | Basic authentication username for the configuration endpoint. | _none_ | +| `container_name` | Name of the container that will contain the blobs. | _none_ | +| `database_file` | Absolute path to a database file used to store blob file contexts. | _none_ | +| `date_key` | Key name used to store the record timestamp. | `@timestamp` | +| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` | +| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `false` | +| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ | +| `file_delivery_attempt_limit` | Maximum number of delivery attempts for a file. | `1` | +| `io_timeout` | HTTP IO timeout. | `60s` | +| `part_delivery_attempt_limit` | Maximum number of delivery attempts for a file part. | `1` | +| `part_size` | Size of each part when uploading blob files. | `25M` | +| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ | +| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ | +| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` | +| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ | +| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` | +| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` | +| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` | +| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` | +| `upload_part_freshness_limit` | Maximum lifespan of an uncommitted file part. | `6D` | +| `upload_parts_timeout` | Timeout for uploading parts of a blob file. | `10M` | +| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and haven't reached `upload_file_size` limit yet. | `30m` | +| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | ## Get started