Skip to content

[FLINK-40632][s3] Annotate native-s3-fs to PublicEvolving #29159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Samrat002 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[FLINK-40632][s3] Annotate native-s3-fs to PublicEvolving #29159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 5 additions & 9 deletions docs/content.zh/docs/deployment/filesystems/s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,10 @@ Note that these examples are *not* exhaustive and you can use S3 in other places

Flink provides three independent S3 filesystem implementations:

| Implementation | Checkpointing | FileSink | Notes |
|---------------|:---:|:---:|-------|
| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
| Implementation | Checkpointing | FileSink | Notes |
|---------------|:---:|:---:|--------------------------------------------------------------------------------------------|
| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | Built on AWS SDK v2; no Hadoop dependency. Generally available since Flink 2.4. |
| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |

Previously, users had to choose between Presto (recommended for checkpointing throughput) and Hadoop (the only implementation with `RecoverableWriter`, required by the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}})). The Native S3 implementation unifies both capabilities in a single plugin and measurements show significant checkpoint throughput improvements over the Presto implementation.
Expand Down Expand Up @@ -149,11 +149,7 @@ The legacy configuration key `s3.path.style.access` is still supported as a fall

## Implementation Details

### Native S3 FileSystem (Experimental)

{{< hint warning >}}
**Experimental**: The Native S3 FileSystem is experimental in Flink 2.3. It is functionally complete and has demonstrated strong performance in benchmarks.
{{< /hint >}}
### Native S3 FileSystem

The Native S3 FileSystem is a pure-Java implementation built on the AWS SDK v2 completely removing the dependency on Hadoop. It is registered under the schemes *s3://* and *s3a://*. It provides a drop-in replacement for the Presto and Hadoop implementations, supporting checkpointing, the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}}) (via `RecoverableWriter`), server-side encryption (SSE-S3, SSE-KMS), cross-account access via IAM role assumption, entropy injection, and bulk copy via S3TransferManager.

Expand Down
14 changes: 5 additions & 9 deletions docs/content/docs/deployment/filesystems/s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,10 @@ Note that these examples are *not* exhaustive and you can use S3 in other places

Flink provides three independent S3 filesystem implementations:

| Implementation | Checkpointing | FileSink | Notes |
|---------------|:---:|:---:|-------|
| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | **Experimental** in Flink 2.3. Built on AWS SDK v2; no Hadoop dependency. |
| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
| Implementation | Checkpointing | FileSink | Notes |
|---------------|:---:|:---:|--------------------------------------------------------------------------------------------|
| **Native S3** (`flink-s3-fs-native`) | ✓ | ✓ | Built on AWS SDK v2; no Hadoop dependency. Generally available since Flink 2.4. |
| **Presto S3** (`flink-s3-fs-presto`) | ✓ | x | Production-proven for checkpointing. |
| **Hadoop S3** (`flink-s3-fs-hadoop`) | ✓ | ✓ | Mature; the only stable implementation that provides `RecoverableWriter` for the FileSink. |

Previously, users had to choose between Presto (recommended for checkpointing throughput) and Hadoop (the only implementation with `RecoverableWriter`, required by the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}})). The Native S3 implementation unifies both capabilities in a single plugin and measurements show significant checkpoint throughput improvements over the Presto implementation.
Expand Down Expand Up @@ -149,11 +149,7 @@ The legacy configuration key `s3.path.style.access` is still supported as a fall

## Implementation Details

### Native S3 FileSystem (Experimental)

{{< hint warning >}}
**Experimental**: The Native S3 FileSystem is experimental in Flink 2.3. It is functionally complete and has demonstrated strong performance in benchmarks.
{{< /hint >}}
### Native S3 FileSystem

The Native S3 FileSystem is a pure-Java implementation built on the AWS SDK v2 completely removing the dependency on Hadoop. It is registered under the schemes *s3://* and *s3a://*. It provides a drop-in replacement for the Presto and Hadoop implementations, supporting checkpointing, the [FileSink]({{< ref "docs/connectors/datastream/filesystem" >}}) (via `RecoverableWriter`), server-side encryption (SSE-S3, SSE-KMS), cross-account access via IAM role assumption, entropy injection, and bulk copy via S3TransferManager.

Expand Down
4 changes: 2 additions & 2 deletions ...k-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3AFileSystemFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

package org.apache.flink.fs.s3native;

import org.apache.flink.annotation.Experimental;
import org.apache.flink.annotation.PublicEvolving;

/**
* Factory for the native S3 file system registered for the {@code s3a://} scheme.
Expand All @@ -29,7 +29,7 @@
* <p>All configuration options are the same as for the {@code s3://} scheme. See {@link
* NativeS3FileSystemFactory} for available options.
*/
@Experimental
@PublicEvolving

@gaborgsomogyi gaborgsomogyi Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. Additionally it's not API in Hadoop connector too.

public class NativeS3AFileSystemFactory extends NativeS3FileSystemFactory {

@Override
Expand Down
4 changes: 2 additions & 2 deletions ...nk-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystemFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

package org.apache.flink.fs.s3native;

import org.apache.flink.annotation.Experimental;
import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ConfigOptions;
import org.apache.flink.configuration.Configuration;
Expand Down Expand Up @@ -61,7 +61,7 @@
* @see NativeS3FileSystem
* @see org.apache.flink.core.fs.FileSystemFactory
*/
@Experimental
@PublicEvolving

@gaborgsomogyi gaborgsomogyi Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. Additionally it's not API in Hadoop connector too.

public class NativeS3FileSystemFactory implements FileSystemFactory, MetricsAware {

private static final Logger LOG = LoggerFactory.getLogger(NativeS3FileSystemFactory.class);
Expand Down
4 changes: 2 additions & 2 deletions ...s-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

package org.apache.flink.fs.s3native.writer;

import org.apache.flink.annotation.Experimental;
import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.core.fs.Path;
import org.apache.flink.core.fs.RecoverableFsDataOutputStream;
import org.apache.flink.core.fs.RecoverableWriter;
Expand All @@ -36,7 +36,7 @@
import static org.apache.flink.util.Preconditions.checkNotNull;

/** Recoverable writer for S3 using multipart uploads for exactly-once semantics. */
@Experimental
@PublicEvolving

@gaborgsomogyi gaborgsomogyi Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why are these part of our API at all? RecoverableWriter is an API and we just implement it internally.

public class NativeS3RecoverableWriter implements RecoverableWriter, AutoCloseable {

private static final Logger LOG = LoggerFactory.getLogger(NativeS3RecoverableWriter.class);
Expand Down