[server] Respect the option of table.log.ttl for local segments#3222
Open
zuston wants to merge 2 commits intoapache:mainfrom
Open
[server] Respect the option of table.log.ttl for local segments#3222zuston wants to merge 2 commits intoapache:mainfrom
table.log.ttl for local segments#3222zuston wants to merge 2 commits intoapache:mainfrom
Conversation
zuston
changed the title
table.log.ttl for local segments
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates Fluss server log tiering so that table.log.ttl is respected for local segments as well: when the active local segment is TTL-expired, it can be rolled to become eligible for remote tiering and subsequent local cleanup, while keeping a new empty active segment ready for future appends.
Changes:
- Add TTL-based rolling for a non-empty active segment in
LogTablet, and allow TTL-expired segments to bypass the “retain N local segments” restriction during local cleanup. - Trigger TTL-based active-segment rolling at the start of each
LogTieringTaskrun so remote tiering can upload the rolled segment. - Extend remote log TTL tests to cover rolling/upload/cleanup behavior and appends after rolling.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| fluss-server/src/test/java/org/apache/fluss/server/log/remote/RemoteLogTTLTest.java | Updates TTL expectations and adds a test covering expired active-segment roll, upload, local deletion, and subsequent appends. |
| fluss-server/src/main/java/org/apache/fluss/server/log/remote/LogTieringTask.java | Rolls a TTL-expired active segment before selecting upload candidates. |
| fluss-server/src/main/java/org/apache/fluss/server/log/LogTablet.java | Introduces active-segment TTL roll and TTL-aware local deletability logic (bypassing retained-segment count when expired). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1212
to
+1221
| int tierProtectedStartIndex = logSegments.size() - tieredLogLocalSegments; | ||
| long now = clock.milliseconds(); | ||
|
|
||
| for (int i = 0; i < logSegments.size() - 1; i++) { | ||
| if (logSegments.get(i + 1).getBaseOffset() > endOffset) { | ||
| break; | ||
| } | ||
| if (i < tierProtectedStartIndex || isSegmentExpired(now, logSegments.get(i))) { | ||
| deletableSegments.add(logSegments.get(i)); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Enable TTL-expired local log segments to be rolled (bypass the limitation of num segment retained), while preserving a new empty active segment for future appends.
This PR is to reduce the partitioned log table bucket local storage capacity, especially longterm day level partition log table
Brief change log
LogTablet.Tests
API and Format
Documentation