[BUG]: VirtualMemoryResource leaks VMM handles after successful allocation and growth

[fallintoplace]

Component

cuda.core

What happened?

VirtualMemoryResource appears to keep CUDA VMM allocation handles alive after a successful allocation or growth operation.

The successful paths call cuMemCreate() and map the returned handle, but the transaction is committed without releasing that handle. When the Buffer is closed, deallocate() retains and releases another reference to the mapping; the original handle remains unreleased.

CUDA documents that the backing allocation is only freed after the mapping and all unreleased handles are gone. This means repeated allocate/close cycles can steadily consume GPU memory and eventually fail with CUDA_ERROR_OUT_OF_MEMORY.

How to reproduce

On a CUDA VMM-capable GPU, run this against current main:

import platform

from cuda.core import Device, VirtualMemoryResource, VirtualMemoryResourceOptions
from cuda.core._utils.cuda_utils import CUDAError


device = Device(0)
device.set_current()

handle_type = "win32_kmt" if platform.system() == "Windows" else "posix_fd"
mr = VirtualMemoryResource(
    device,
    config=VirtualMemoryResourceOptions(handle_type=handle_type),
)

allocation_size = 256 * 1024 * 1024

for i in range(1000):
    try:
        buf = mr.allocate(allocation_size)
        buf.close()
        print(f"completed iteration {i}")
    except CUDAError as exc:
        print(f"failed at iteration {i}: {exc}")
        break

On the affected path, GPU memory usage keeps increasing after buf.close() and the loop eventually reaches CUDA_ERROR_OUT_OF_MEMORY. Applying the proposed fix and running the same loop should keep usage near the post-close baseline.

The same lifecycle issue is present in the successful fast- and slow-path growth operations, where newly created and retained handles are committed without being released.

Expected behavior

Closing a Buffer should release the VMM allocation completely. Repeated allocate/close and grow/close cycles should not leak GPU memory or allocation handles.

Additional context

The proposed fix in PR #2235 releases newly created and retained handles before committing the transaction, while keeping release failures inside rollback handling.