[SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode

[kete1987]

What changes were proposed in this pull request?

In unbounded memory mode (the default, boundedMemoryUsage = false), RocksDBMemoryManager creates a new LRUCache per instance:

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala

Line 185 in d7df192

(null, new LRUCache(conf.blockCacheSizeMB * 1024 * 1024))

but RocksDB.close() never calls lruCache.close():

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala

Lines 2125 to 2135 in d7df192

	def close(): Unit = {
	// Acquire DB instance lock and release at the end to allow for synchronized access
	try {
	closeDB()
	readOptions.close()
	writeOptions.close()
	flushOptions.close()
	nativeStats.close()
	rocksDbOptions.close()
	dbLogger.close()

The Java LRUCache wrapper holds a C++ shared_ptr<Cache>, so the native object is only freed when the JVM GC finalizes the wrapper — which rarely happens under low heap pressure. This causes native memory to accumulate until GC eventually runs, leading to OOM kills in long-running processes or CI runs with many RocksDB-heavy test suites.

The fix adds an explicit lruCache.close() call in RocksDB.close() for unbounded mode. In bounded mode the cache is a shared singleton managed by RocksDBMemoryManager and must not be closed per instance.

This is a separate issue from SPARK-56523 (Statistics native memory leak), which was already fixed.

Why are the changes needed?

Without explicit close(), each RocksDB instance in unbounded mode leaks one LRUCache worth of native memory (blockCacheSizeMB, default 8 MB) for as long as GC does not run. The memory is never reclaimed deterministically.

A standalone reproducer tool confirms ~8.5 MB of native memory growth per open/close cycle in leak mode vs flat memory in fixed mode:
https://github.com/kete1987/rocksdb-leak-tool

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added a test in RocksDBSuite (SPARK-57183: LRUCache is closed on RocksDB.close() in unbounded memory mode) that verifies the native handle is released after close() via LRUCache.isOwningHandle().

Was this patch authored or co-authored using generative AI tooling?

Co-authored with Claude (Anthropic), used for analysis, code generation and review assistance.
Generated-by: Claude Sonnet 4.6

I affirm that the contribution is my original work and that I license the work to the project under the project's open source license.

[HeartSaVioR]

I'll ask other folks to review the change (I'm a bit away from recent improvement of RocksDB state store provider), but I'm going to give a general suggestion.

Please do not remove the section of PR template and consider filling the section as one of the requirement/duty.

I affirm that the contribution is my original work and that I license the work to the project under the project's open source license.

This doesn't replace the requirement the PR template asks about the usage of LLM model and the clarification of the model. The template guides how to describe the model - please check it out again.

https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE

### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->

[HeartSaVioR]

cc. @anishshri-db @micheal-o

[kete1987]

@HeartSaVioR Thanks for the feedback! I've updated the PR description to include the AI tooling section.

[cloud-fan]

why it's not merged in branch 4.x? this is not a breaking change.

[cloud-fan]

@kete1987 can you open a 4.x PR?

[HeartSaVioR]

Where are we? Do we not have a fix in 4.x? Shall we just cherry-pick and push as we would do it in merge script?

[kete1987]

@HeartSaVioR @cloud-fan Already done — #56281 was merged to branch-4.2. Thanks!

[cloud-fan]

@kete1987 branch-4.2 and branch-4.x are two different branches...

[kete1987]

@cloud-fan opened #56308 for branch-4.x. Sorry for the confusion!