Serialize whole-record bucket metadata updates across config types

[Vonng]

Problem

Bucket metadata is persisted as one buckets/<bucket>/.metadata.bin record, but most writers perform an unlocked load-modify-save. Different configuration types can therefore read the same old record and overwrite each other's fields.

Confirmed writer paths include:

• BucketMetadataSys.Update/Delete through updateAndParse;
• local and peer CORS transitions, currently protected only by cors-config.lock;
• legacy/bulk PeerBucketMetadataUpdateHandler;
• PeerBucketMakeWithVersioningHandler;
• ImportBucketMetadataHandler;
• erasureServerPools.MakeBucket;
• convertLegacyConfigs and migrateTargetConfig, which save from metadata load paths.

A related confirmed clobber exists in MakeBucket{ForceCreate:true}: storage-level errVolumeExists is ignored and a fresh .metadata.bin is written for the existing bucket, dropping policy, CORS, tagging, encryption, and other fields. Import can reach this path after a metadata read failure.

This is separate from #77. Issue #77 tracks site-replication source timestamps, tombstones, heal selection, diagnostics, and counters; it does not own cross-type whole-record serialization.

Required design

Introduce one distributed namespace lock:

.minio.sys / buckets/<bucket>/metadata.lock

Lock order must be:

<bucket>.lck -> metadata.lock -> .metadata.bin

Requirements:

• hold metadata.lock from the authoritative disk read through mutation and .metadata.bin save;
• never acquire it inside BucketMetadataSys.save or BucketMetadata.Save because object saves take the .metadata.bin namespace lock and locks are not re-entrant;
• do not call MakeBucket or DeleteBucket while already holding metadata.lock;
• update local cache while the committed record is known, then release metadata.lock before peer LoadBucketMetadata fan-out or BucketMetaHook;
• replace the CORS-specific lock with the shared lock;
• make ForceCreate read-or-fresh and preserve an existing record;
• parse imports off-lock, then apply per-bucket field patches to a fresh under-lock record; a non-errConfigNotFound read failure must be reported, not converted into ForceCreate;
• serialize migration saves or route unlocked load paths through a locked migration helper.

Test-first reproduction

Add a controllable ObjectLayer wrapper around .metadata.bin writes:

1. writer A reads the old record and pauses at PutObject entry;
2. writer B starts;
3. on the unfixed tree B reaches a second stale read and completes, then A overwrites B;
4. with metadata.lock, B attempts the lock but cannot read until A is released;
5. final persisted and resident metadata must contain both fields.

Required pairs, on both ObjectLayer backends:

• policy + CORS;
• tagging + SSE;
• local Update + peer/bulk direct-save variants.

Behavioral coverage:

• ForceCreate preserves policy/CORS and Created;
• peer bucket adoption preserves existing metadata;
• Import overwrites only supplied fields and preserves unrelated concurrent fields;
• migration cannot overwrite a concurrent update;
• lock acquisition is context bounded and no re-entrant deadlock occurs.

Documentation

Update docs/site-replication/CORS-LWW-DESIGN.md:

• replace cors-config.lock with metadata.lock;
• describe peer fan-out as outside the lock;
• replace the incorrect Audit site-replication source timestamps, tombstones, and per-site counters #77 references with this issue;
• keep any deferred lifecycle-merge or DeleteBucket ghost-record work explicit.

Deliberate follow-ups

These are adjacent but may be split if needed to keep the first PR reviewable:

• higher-level lifecycle XML merge atomicity outside BucketMetadataSys.Delete;
• DeleteBucket versus an in-flight metadata update (ghost-record resurrection);
• overlapping peer reloads leaving a local cache one save behind until refresh.

The first PR must not claim these follow-ups are fixed unless their reproductions and tests are included.

[Vonng]

Implementation submitted as PR #103.

Evidence:

• deterministic red tests reproduced policy+CORS and tagging+SSE lost updates, plus ForceCreate metadata clobber;
• shared metadata.lock, direct writers, import rebasing, migrations, MakeBucket/adoption, cancellation, lifecycle Delete, peer bulk, and rolling-upgrade documentation implemented;
• focused race, full cmd tests, build, vet, lint, cross-compile, DCO and all automatic PR checks PASS;
• two Opus 5 adversarial implementation rounds: final GO, no P0/P1.

Deliberately still open under this issue: higher-level site-replication lifecycle merge atomicity, DeleteBucket vs already in-flight writer, and peer cache reload ordering.

[Vonng]

PR #103 merged into main as 6586fbfd0 and closes the confirmed scope of this issue:

• cross-type whole-record lost updates are serialized by the shared metadata.lock;
• ForceCreate and site adoption preserve existing metadata;
• import, migration, peer/bulk, cancellation and lifecycle-delete paths participate in the documented lock boundary;
• focused race, full cmd, lint, generated-file, cross-build, DCO, Go CI and VulnCheck gates passed.

The three cases explicitly excluded from PR #103 have been split into #105, where they remain reproduction-first follow-up work and are not silently claimed as fixed.

Closing this issue as completed.