[SPARK-59439][CORE] Release the HistoryServerDiskManager lease reservation exactly once on a failed commit

[uros-b]

What changes were proposed in this pull request?

HistoryServerDiskManager.Lease now releases its reserved (uncommitted) usage exactly once, whether the lease is committed, fails to commit, or is rolled back. The two updateUsage(-leased) calls in commit() and rollback() are replaced by an idempotent releaseLease() helper guarded by a released flag.

A regression test forces the rename in commit() to fail, then calls rollback() (as the caller does) and asserts the usage tracker returns to zero rather than going negative, and that a subsequent lease/commit still succeeds.

Why are the changes needed?

Lease.commit() releases the reservation with updateUsage(-leased) before renaming the temporary store into place. SPARK-58985 made that rename throw an IOException on failure, which happens after the reservation has already been released. The caller then rolls the lease back -- e.g. FsHistoryProvider.createDiskStore() calls lease.rollback() on IOException -- and rollback() releases the reservation a second time.

The reservation is added once (in lease()) but subtracted twice, so the current-usage tracker is under-counted by the leased amount and can go negative, throwing:

java.lang.IllegalStateException: Disk usage tracker went negative (now = ..., delta = ...)

This is the same crash SPARK-58985 aimed to prevent; it is reachable whenever renameTo fails (I/O error, full disk, destination parent removed out of band). In createDiskStore()'s retry loop the exception also escapes the loop, so the store is never rebuilt.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New unit test in HistoryServerDiskManagerSuite (runs for both the LevelDB and RocksDB backends). It fails on the current code with IllegalStateException: Disk usage tracker went negative and passes with this change.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8

[uros-b]

@pan3793 could you please review, as a followup to #58312

[dongjoon-hyun]

Thank you for the fix. The change looks correct to me: the reservation is now returned exactly once across all failure paths in commit() (before and after releaseLease()), and it also covers FsHistoryProvider.createHybridStore(), where onSwitchToDiskStoreFail calls lease.rollback() after a failed (or even a successful) commit() on the background thread.

A minor thing in the PR description: could you record the actual model name and version in Generated-by (e.g., Generated-by: Claude Opus 4.8) instead of just Claude?