HDDS-16060. [Ozone Versioning] [T7] Snapshot exclusion - #10965
Open
symious wants to merge 28 commits into
Open
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…equest Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Key names in OBJECT_STORE buckets contain '/' verbatim, so a '/' separator interleaves a key's versions with those of keys nested under it, breaking the single-seek promotion and the merged ListObjectVersions order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rename NULL_VERSION_ID to UNSET_VERSION_ID: 0 is the unset value of the optional proto field, not the id of the null version. A null version carries a normally generated id and is identified by isNullVersion, so that a null created between two versioned writes orders as the middle version rather than the oldest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reclaim branches skipped S3-versioned buckets only because the legacy isVersionEnabled flag is kept in sync with an ENABLED status. Depend on the status directly, so that dropping that sync cannot strand a version record by reclaiming the blocks it still refers to. OMKeyCommitRequestWithFSO is left alone: isS3VersioningEnabled() requires the OBJECT_STORE layout, so the check is structurally false on the FSO path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
KeyArgs gains versionId and nullVersion: a null version carries a normally generated id like any other version, so the null slot needs a selector of its own rather than a reserved id. The current version is checked before the versionedKeyTable, so naming it costs no extra read, and the null slot is found by a bounded scan of the key's version prefix. Addressing a delete marker by version is reported as KEY_IS_DELETE_MARKER rather than KEY_NOT_FOUND: S3 answers 405 for it and 404 only for a read that lands on a current marker without naming a version. The status mapping itself belongs to the gateway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DELETE ?versionId= is the only delete that destroys data on a versioned bucket: the version leaves the versionedKeyTable and its blocks go to the deletedTable, which stays the single path through which version blocks are reclaimed. The null slot is addressed by attribute, so it is found by the same bounded prefix scan the read path uses. Addressing the current version is rejected for now: removing it has to promote the next-newest version to keep the keyTable authoritative, which T4.3 adds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
keyTable holds the current version of every key that still has one, so removing the current version has to hand the place over: one seek on the key's version prefix yields the newest remaining version, which moves back into the keyTable in the same WriteBatch as the delete. The record travels unchanged - promotion is positional, and a version keeps the identity it was created with. When no version survives, the key disappears entirely. Deleting a current delete marker this way is exactly S3's restore-an-object flow: the version the marker superseded becomes current again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nded Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add the bucket-level maxVersions limit and the background service that enforces it. The limit counts the key's current version and its delete markers along with the noncurrent ones, so a key with maxVersions n keeps at most n - 1 noncurrent versions; 0 means unlimited, and a bucket that sets none falls back to ozone.om.versioning.max.versions. Trimming runs in the new VersionCleanupService rather than inside the write transaction, so the cost of a version write does not grow with the number of versions the key already has. The service scans only buckets that have ever been versioned and whose limit is finite, and hands the versions it selects to OM as a ReclaimObjectVersions request. That request moves them out of the versionedKeyTable and into the deletedTable, which is the single path through which version blocks are reclaimed: nothing here reclaims blocks directly, so KeyDeletingService stays the only place that decides whether a snapshot still needs them. A version may already be gone by the time the request applies - permanently deleted, or promoted into the keyTable because the current version was deleted - so anything no longer in the versionedKeyTable is skipped and a later run reselects if the key is still over its limit. The OEP also lists a REJECT policy that fails the write instead of trimming. It is deliberately left out: S3 has no error code for the condition and no client retries on it, so a default limit enforced that way would break unmodified S3 tooling after n overwrites of one key. Trimming is the behaviour S3 lifecycle rules provide, and it costs nothing on the write path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add the per-bucket noncurrentVersionExpirationDays and select expired versions in the same VersionCleanupService scan that applies maxVersions. Either control on its own is enough to reclaim a version, so a bucket can bound its version chains by count, by age, or by both. Expiration is opt-in: 0 or absent retains versions forever. The clock starts when the version that superseded this one was committed, as S3 lifecycle's NoncurrentDays does, not when the version itself was written. A version is not carrying that moment - demotion into the versionedKeyTable moves the record unchanged - so the scan takes it from the next-newer version, which is the one that demoted it. Versions of a key are contiguous and ordered newest first, so walking the key hands the scan the superseding version immediately before each version it has to judge; only the newest noncurrent one needs a lookup, of the key's current version in the keyTable. That distinction is the whole point of the feature: a key written once and superseded years later would otherwise expire the moment it became noncurrent, since its own timestamp is already older than any threshold. A key with no current version keeps its newest noncurrent version rather than expiring it. The keyTable holds the current version of every key that still has one, so this only happens if that invariant is broken, and expiring on a broken invariant destroys data. Throughput is exposed on DeletingServiceMetrics as the versions sent for reclamation and the versions actually reclaimed; the two differ by the versions permanently deleted or promoted in between. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Such a key is invisible to reads and its marker carries no versionId to address it by, so nothing a client can do removes it: it would sit in the keyTable forever once its last real version was reclaimed. The bucket property expiredDeleteMarkerCleanup, enabled unless turned off, has VersionCleanupService remove the whole key. The marker leaves the keyTable rather than the versionedKeyTable, so the reclaim request carries it in a separate list; it holds no blocks, so nothing goes to the deletedTable for it and only its namespace slot is released. Two conditions are re-checked when the request applies, not only when the scan selects. A write since the scan makes the key's current version a real object again, and there is then nothing expired here. More importantly, a noncurrent version may have appeared, and removing the marker while one survives would promote it back to current - resurrecting an object the user deleted. That check costs one seek per marker and is worth it. There is no index of delete markers, so finding them walks the keyTable. The walk is bounded by ozone.om.version.cleanup.marker.scan.budget and resumes where the previous run stopped, so no single run iterates an unbounded number of keys while successive runs still cover the table. The resume point is held in memory: a restart or a failover starts the walk over, which costs a pass and loses nothing. The bucket is read past the table cache so that this walk and the versionedKeyTable walk judge a bucket by the same state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A snapshot shares physical blocks with the active object store and keeps them only because KeyDeletingService refuses to reclaim what a previous snapshot still references. That decision is made per deletedTable record by ReclaimableKeyFilter, which resolves the key in the previous snapshot by looking it up in the keyTable alone. With versioning a key's live records span two tables, so the lookup misses and a permanently deleted noncurrent version reads as reclaimable - taking blocks a snapshot still points at. Until that lookup is version-aware, OM refuses the combination outright rather than leaving it to convention. Both directions are covered: - CreateSnapshot is rejected when the bucket's versioning status is ENABLED or SUSPENDED. SUSPENDED counts because a suspended bucket still holds the noncurrent versions it accumulated while enabled. - SetBucketProperty is rejected when it would leave the bucket in a versioned state while the bucket still has snapshots. ENABLED -> SUSPENDED stays allowed even with snapshots present, so a bucket that reached a mixed state before these checks existed can at least stop accumulating versions. A transition to the status the bucket already has is allowed for the same reason, and because an S3 client re-sending PutBucketVersioning must not fail against a bucket already in that state. Neither check is latched: once the last snapshot is purged, versioning can be enabled or resumed normally. Two things the existing code already gives us, now covered by tests rather than new code. OMSnapshotCreateRequest.preExecute resolves a linked bucket to its source before this check runs, so a link is judged by what it points at; OMBucketSetPropertyRequest rejects links outright. And a snapshot that was deleted but not yet purged is still in the snapshotInfoTable, so it counts as present - it references its blocks until the purge removes it. That is pinned by a test, since moving the removal earlier would silently weaken the check. Both checks run in validateAndUpdateCache rather than preExecute so that they are ordered by Ratis against each other: whichever applies second sees the state the first one committed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The exclusion T7.1 enforces has to be liftable, or the integration tests for snapshot-aware version reclamation cannot be written: they need a bucket that actually holds both a snapshot and object versions, and nothing else can produce one. ozone.om.snapshot.versioning.coexistence.enabled, false by default, lifts both checks. It is documented as unsafe and testing-only in ozone-default.xml, and OM logs a warning at startup when it is on, since a cluster running with it can lose data: reclaiming a noncurrent version can delete blocks a snapshot still references. With the flag off, behaviour is exactly what T7.1 established. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Please only review commits start with "T7".
This ticket includes the following tasks:
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16060
How was this patch tested?
unit test