Skip to content

Use CRC32C for checksums instead of MD5 #6442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nadav-govari wants to merge 9 commits into
base: main
Choose a base branch
from
+132 −63
Open

Use CRC32C for checksums instead of MD5 #6442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
207cf76
Use CRC32C for checksums instead of MD5
nadav-govari May 18, 2026
600281b
Only apply crc32c on non-S3 flavors
nadav-govari May 19, 2026
78d6275
lints
nadav-govari May 19, 2026
f7f6a45
Make crc32 default and a turn-off button for it
nadav-govari May 19, 2026
92d1fbf
lints
nadav-govari May 19, 2026
9f2cfdf
Make it an enum
nadav-govari May 19, 2026
ed54ce6
restore the md5 computation since its a noop on the aws sdk
nadav-govari May 19, 2026
4169a3a
rename to checksum algorithm, update docs, backwards compat
nadav-govari May 22, 2026
44198f8
lints
nadav-govari May 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 15 additions & 14 deletions docs/configuration/storage-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,18 @@ This section contains one configuration subsection per storage provider. If a st

### S3 storage configuration

| Property | Description | Default value |
| --- | --- | --- |
| `flavor` | The optional storage flavor to use. Available flavors are `digital_ocean`, `garage`, `gcs`, and `minio`. | |
| `access_key_id` | The AWS access key ID. | |
| `secret_access_key` | The AWS secret access key. | |
| `region` | The AWS region to send requests to. | `us-east-1` (SDK default) |
| `endpoint` | Custom endpoint for use with S3-compatible providers. | SDK default |
| `force_path_style_access` | Disables [virtual-hosted–style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) requests. Required by some S3-compatible providers (Ceph, MinIO). | `false` |
| `disable_multi_object_delete` | Disables [Multi-Object Delete](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) requests. Required by some S3-compatible providers (GCS). | `false` |
| `disable_multipart_upload` | Disables [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) of objects. Required by some S3-compatible providers (GCS). | `false` |
| `disable_checksums` | Disables [checksums](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html) on requests and responses. Required by S3-compatible providers that do not support the additional checksum algorithms enabled by default in recent versions of the AWS SDK (Digital Ocean, Garage, GCS, MinIO). | `false` |
| Property | Description | Default value |
| --- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------|
Comment on lines +49 to +50
Copy link
Copy Markdown
Member

@guilload guilload May 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new formatting of the table is gross and makes the diff harder to review.

| `flavor` | The optional storage flavor to use. Available flavors are `digital_ocean`, `garage`, `gcs`, and `minio`. | |
| `access_key_id` | The AWS access key ID. | |
| `secret_access_key` | The AWS secret access key. | |
| `region` | The AWS region to send requests to. | `us-east-1` (SDK default) |
| `endpoint` | Custom endpoint for use with S3-compatible providers. | SDK default |
| `force_path_style_access` | Disables [virtual-hosted–style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) requests. Required by some S3-compatible providers (Ceph, MinIO). | `false` |
| `disable_multi_object_delete` | Disables [Multi-Object Delete](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) requests. Required by some S3-compatible providers (GCS). | `false` |
| `disable_multipart_upload` | Disables [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) of objects. Required by some S3-compatible providers (GCS). | `false` |
| `checksum_algorithm` | Upload [checksum](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html) algorithm. Allowed values: `crc32c` (computed and validated by the AWS SDK), `md5` (sent client-side via `Content-MD5`; useful for S3-compatible providers that predate `x-amz-checksum-*`), or `disabled`. | `crc32c` |
| `disable_checksums` | **Deprecated.** Previously a boolean that disabled all request/response checksums. Equivalent to setting `checksum_algorithm: disabled`. | `false` |

:::warning
Hardcoding credentials into configuration files is not secure and strongly discouraged. Prefer the alternative authentication methods that your storage backend may provide.
Expand All @@ -79,19 +80,19 @@ Storage flavors ensure that Quickwit works correctly with storage providers that

*Digital Ocean*

The Digital Ocean flavor (`digital_ocean`) forces path-style access, turns off multi-object delete requests, and disables checksums.
The Digital Ocean flavor (`digital_ocean`) forces path-style access and turns off multi-object delete requests.

*Garage flavor*

The Garage flavor (`garage`) overrides the `region` parameter to `garage`, forces path-style access, and disables checksums.
The Garage flavor (`garage`) overrides the `region` parameter to `garage` and forces path-style access.

*Google Cloud Storage*

The Google Cloud Storage flavor (`gcs`) turns off multi-object delete requests, multipart uploads, and disables checksums.

*MinIO flavor*

The MinIO flavor (`minio`) overrides the `region` parameter to `minio`, forces path-style access, and disables checksums.
The MinIO flavor (`minio`) overrides the `region` parameter to `minio` and forces path-style access.

Example of a storage configuration for Google Cloud Storage in YAML format:

Expand Down
5 changes: 3 additions & 2 deletions quickwit/quickwit-config/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,9 @@ pub use crate::node_config::{
};
use crate::source_config::serialize::{SourceConfigV0_7, SourceConfigV0_8, VersionedSourceConfig};
pub use crate::storage_config::{
AzureStorageConfig, FileStorageConfig, GoogleCloudStorageConfig, RamStorageConfig,
S3StorageConfig, StorageBackend, StorageBackendFlavor, StorageConfig, StorageConfigs,
AzureStorageConfig, ChecksumAlgorithm, FileStorageConfig, GoogleCloudStorageConfig,
RamStorageConfig, S3StorageConfig, StorageBackend, StorageBackendFlavor, StorageConfig,
StorageConfigs,
};

/// Returns true if the ingest API v2 is enabled.
Expand Down
35 changes: 27 additions & 8 deletions quickwit/quickwit-config/src/storage_config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,20 @@ pub enum StorageBackend {
S3,
}

/// Strategy used to checksum object-storage uploads.
#[derive(Debug, Clone, Copy, Default, Eq, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ChecksumAlgorithm {
/// CRC32C, computed and validated by the AWS SDK. Native S3 default.
#[default]
Crc32c,
/// MD5 (Content-MD5 header), computed client-side. Used by S3-compatible
/// implementations that predate the SDK's `x-amz-checksum-*` headers.
Md5,
/// No upload checksum is sent and no response checksum is validated.
Disabled,
}

#[derive(Debug, Clone, Copy, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum StorageBackendFlavor {
Expand Down Expand Up @@ -330,7 +344,10 @@ pub struct S3StorageConfig {
#[serde(default)]
pub disable_multipart_upload: bool,
#[serde(default)]
pub disable_checksums: bool,
Copy link
Copy Markdown
Collaborator

@fulmicoton-dd fulmicoton-dd May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that does not look backward compatible. I think noone uses it, and they will get a clear error message, so it is probably ok if it is a calculated risk.

Copy link
Copy Markdown
Member

@guilload guilload May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nadav, you can make this backward compatible with some serde trick. We can use an alias and then deserialize false to disabled and true to the new default.

pub checksum_algorithm: ChecksumAlgorithm,
/// Deprecated: applies into `checksum_algorithm: disabled`.
#[serde(default, skip_serializing)]
pub disable_checksums: Option<bool>,
#[serde(default)]
pub disable_stalled_stream_protection_upload: bool,
#[serde(default)]
Expand All @@ -343,25 +360,27 @@ impl S3StorageConfig {
Some(StorageBackendFlavor::DigitalOcean) => {
self.force_path_style_access = true;
self.disable_multi_object_delete = true;
self.disable_checksums = true;
}
Some(StorageBackendFlavor::Garage) => {
self.region = Some("garage".to_string());
self.force_path_style_access = true;
self.disable_checksums = true;
}
Some(StorageBackendFlavor::Gcs) => {
self.disable_multi_object_delete = true;
self.disable_multipart_upload = true;
self.disable_checksums = true;
// doesnt support CRC32C via the S3 SDK
self.checksum_algorithm = ChecksumAlgorithm::Disabled;
}
Some(StorageBackendFlavor::MinIO) => {
self.region = Some("minio".to_string());
self.force_path_style_access = true;
self.disable_checksums = true;
}
_ => {}
}
// Legacy: honor `disable_checksums: true` from older configs.
if matches!(self.disable_checksums, Some(true)) {
self.checksum_algorithm = ChecksumAlgorithm::Disabled;
}
}

pub fn redact(&mut self) {
Expand Down Expand Up @@ -404,7 +423,7 @@ impl fmt::Debug for S3StorageConfig {
&self.disable_multi_object_delete,
)
.field("disable_multipart_upload", &self.disable_multipart_upload)
.field("disable_checksums", &self.disable_checksums)
.field("checksum_algorithm", &self.checksum_algorithm)
.field(
"disable_stalled_stream_protection_upload",
&self.disable_stalled_stream_protection_upload,
Expand Down Expand Up @@ -647,7 +666,7 @@ mod tests {
force_path_style_access: true
disable_multi_object_delete_requests: true
disable_multipart_upload: true
disable_checksums: true
checksum_algorithm: disabled
disable_stalled_stream_protection_upload: true
disable_stalled_stream_protection_download: true
"#;
Expand All @@ -660,7 +679,7 @@ mod tests {
force_path_style_access: true,
disable_multi_object_delete: true,
disable_multipart_upload: true,
disable_checksums: true,
checksum_algorithm: ChecksumAlgorithm::Disabled,
disable_stalled_stream_protection_upload: true,
disable_stalled_stream_protection_download: true,
..Default::default()
Expand Down
Loading
Loading