cuda.core: retain graph attachments with per-node user objects

cuda_core/cuda/core/_cpp/DESIGN.md

@@ -205,6 +205,9 @@ struct StreamBox {
 The shared pointer's custom deleter captures any additional state needed for
 destruction. This ensures resources are always destroyed in the correct order.
 
+Graph node parameters require a specialized ownership model built on
+`OpaqueHandle`; see [Graph Node Attachments](GRAPH_ATTACHMENTS.md).
+
 ### GIL Management
 
 Handle destructors may run from any thread. The implementation includes RAII guards
@@ -217,21 +220,6 @@ Handle destructors may run from any thread. The implementation includes RAII gua
 The handle API functions are safe to call with or without the GIL held. They
 will release the GIL (if necessary) before calling CUDA driver API functions.
 
-### CUDA User-Object Cleanup
-
-CUDA user-object destructors may run on an internal driver thread where CUDA
-API calls are forbidden. Graph attachment payloads can release handles whose
-deleters call CUDA or run Python finalizers, so their CUDA callback must not
-delete the payload directly.
-
-The callback enqueues a preallocated `DeferredCleanupItem` on a
-process-lifetime lock-free `DeferredCleanupQueue` and schedules one coalesced
-`Py_AddPendingCall`. The pending callback drains all queued payloads from
-Python's main thread. If scheduling fails, payloads remain intact for a later
-retry; they are intentionally leaked once interpreter finalization begins. A
-single pending call is shared by all queued payloads because CPython's
-main-thread pending-call queue is bounded.
-
 ### Static Initialization and Deadlock Hazards
 
 When writing C++ code that interacts with Python, a subtle deadlock can occur

cuda_core/cuda/core/_cpp/GRAPH_ATTACHMENTS.md

@@ -0,0 +1,201 @@
+# Graph Node Attachments
+
+## Why attachments are needed
+
+CUDA graph node parameters borrow resource handles and pointers; they do not
+own the resources they refer to. A kernel node, for example, refers to a kernel
+and to the allocations used by its arguments. A host node refers to a callback
+and its user data.
+
+Those resources must remain alive while any graph definition, graph clone,
+executable graph, or in-flight launch can use them. This includes old node
+parameters that a clone or executable inherited before the graph definition
+was changed.
+
+cuda-core keeps these resources alive with a separate attachment for each
+resource-bearing node version. A node version is the fixed set of parameters
+installed when a node is added or updated.
+
+## Core ownership model
+
+For each resource-bearing node version, cuda-core:
+
+1. Collects the required resources in a `NodeAttachment` whose owners are
+   immutable once published.
+2. Creates a CUDA user object that owns the bundle.
+3. Retains one user-object reference on the graph definition.
+
+CUDA retains the user object when it clones or instantiates the graph, and
+keeps it alive for pending launches. Replacing or deleting a node therefore
+releases only the graph definition's reference. Existing clones, executable
+graphs, and launches keep their references until they finish using the old
+bundle.
+
+```text
+CUgraph definition ─┐
+CUgraph clone ──────┼── retains ──> CUuserObject ── owns ──> NodeAttachment
+CUgraphExec ────────┤                                      │
+in-flight launch ───┘                                      └── owns resources
+
+GraphAttachmentState ── non-owning lookup ──> current attachment
+```
+
+CUDA destroys a `NodeAttachment` only after the last user-object
+reference is released. `GraphAttachmentState` points to the current attachment
+so cuda-core can find it, but the metadata does not keep the bundle alive.
+
+`NodeAttachment` has two `OpaqueHandle` owners. Their meaning depends on the
+node type:
+
+- kernel: kernel and argument storage
+- host callback: callback and copied user data
+- memcpy: destination and source
+- memset or event: destination or event in the first owner
+
+`OpaqueHandle` is a `shared_ptr<const void>` that can keep any attached
+resource alive without exposing its type. Existing cuda-core handles reuse
+their shared ownership when converted to `OpaqueHandle`. Python objects and
+copied callback data use custom deleters.
+
+## Code map
+
+The implementation is in [resource_handles.cpp](resource_handles.cpp), with
+its Cython-facing declarations in
+[resource_handles.hpp](resource_handles.hpp) and
+[_resource_handles.pxd](../_resource_handles.pxd).
+
+- `GraphBox` wraps one `CUgraph` and points to the shared
+  `GraphAttachmentState` for its graph hierarchy.
+- `GraphAttachmentState::attachments` maps `(CUgraph, CUgraphNode)` to a
+  non-owning `NodeAttachment` pointer. The attachment stores its CUDA
+  user-object handle, both resource owners, and the intrusive fields used for
+  deferred cleanup.
+- `GraphAttachmentState::owned_subgraphs` records the child and conditional
+  graphs destroyed with each owner node.
+- `GraphAttachmentsHandle` owns a prepared attachment change until it is
+  committed or discarded.
+- `GraphCloneAttachmentsHandle` owns metadata prepared while embedding a graph
+  as a child.
+
+The Cython graph code calls the attachment helpers when it adds, replaces, or
+deletes nodes. `graph_get_attachments` copies both owners from the current
+metadata so partial node updates can preserve omitted resources.
+
+## Common operations
+
+### Adding a node
+
+Before calling CUDA to add a node, `graph_prepare_attachments`:
+
+1. Builds the immutable attachment bundle.
+2. Creates its CUDA user object.
+3. Moves one user-object reference into the graph.
+4. Allocates the metadata needed to record the returned node handle.
+
+If preparation or node creation fails, dropping the prepared handle releases
+the graph's new reference. The graph definition's metadata remains unchanged.
+
+After CUDA returns the node handle, `graph_commit_attachments` records the
+attachment. Preparation has already allocated the map entry, so commit does
+not allocate after CUDA creates the node.
+
+A captured host callback has one extra failure case: CUDA can add the node
+before cuda-core recovers its handle from the capture state. If handle recovery
+fails, `graph_abandon_attachments` leaves the user object retained by the graph
+without publishing a metadata record. Releasing it would leave the captured
+node with dangling pointers.
+
+### Replacing node parameters
+
+Replacement follows the same prepare-then-commit sequence. cuda-core retains a
+complete new bundle before passing the new parameters to CUDA. After CUDA
+accepts them, cuda-core records the new attachment and releases the graph
+definition's reference to the old one.
+
+Clones, executable graphs, and in-flight launches still retain the old bundle.
+When the new parameters have no owners, no new user object is created; commit
+simply removes the old metadata and releases the old graph reference.
+
+### Cloning and instantiating
+
+CUDA retains the graph's user objects when it clones or instantiates a graph.
+Those references are enough to keep resources alive; executable graphs and raw
+driver clones do not need cuda-core metadata for ownership.
+
+For child graphs embedded through cuda-core, the clone helpers also copy the
+non-owning metadata into the parent hierarchy, as described below.
+
+### Deleting a node
+
+cuda-core asks CUDA to delete the node before changing any Python handle or
+attachment metadata. If CUDA rejects the deletion, the node, its edges, its
+attachments, and its wrappers remain unchanged.
+
+After a successful deletion, cuda-core invalidates the node wrappers, erases
+the metadata, and releases the graph definition's user-object reference. If
+the node owns internal graphs, cuda-core also removes their stale metadata.
+
+### Deferred cleanup after the final release
+
+CUDA invokes a user-object destructor on an internal thread where CUDA API
+calls are forbidden. The destructor therefore queues the complete attachment
+bundle for later cleanup instead of releasing its `OpaqueHandle` objects.
+
+`NodeAttachment` inherits from `DeferredCleanupItem`, so the attachment itself
+is the preallocated queue item and virtual destruction releases the complete
+payload. Its CUDA destructor callback only adds it to the process-lifetime,
+lock-free `DeferredCleanupQueue` and requests a `Py_AddPendingCall`. One pending
+call drains every queued attachment from Python's main thread, where
+CUDA-backed deleters and Python finalizers are safe.
+
+The queue coalesces its work because CPython's main-thread pending-call queue is
+bounded. If `Py_AddPendingCall` cannot schedule the drain, the attachments stay
+queued and a later enqueue or safe cuda-core entry retries. Once interpreter
+finalization begins, scheduling stops and any remaining attachments are
+intentionally leaked rather than invoking unsafe cleanup.
+
+## Child and conditional graph metadata
+
+The root `GraphBox` creates a `GraphAttachmentState`. Graph boxes for its child
+and conditional graphs share that state through their parent handles. This is
+needed because the [graph and node wrapper registries](REGISTRY_DESIGN.md) are
+weak. cuda-core may later recreate a wrapper for the same internal `CUgraph`;
+the shared state lets that wrapper find the existing records.
+
+One state covers the whole hierarchy, so each `AttachmentKey` includes both the
+containing `CUgraph` and its `CUgraphNode`. The graph handle separates records
+belonging to different graphs and lets cuda-core remove all records for a
+destroyed subgraph.
+
+`owned_subgraphs[(parent_graph, owner_node)]` lists the internal graphs whose
+lifetime is tied to an owner node. CUDA destroys those graphs and releases
+their user objects with the owner node. cuda-core then removes metadata for the
+entire subtree without releasing the user objects again.
+
+## Rules for changing this code
+
+1. Retain every required resource before CUDA can use the node parameters.
+2. Never modify a published `NodeAttachment`'s owners; replace the whole
+   attachment. Its intrusive cleanup fields are used only after CUDA releases
+   the final reference.
+3. Treat `GraphAttachmentState` as non-owning metadata. Erase a record before
+   releasing the user-object reference that may destroy its bundle.
+4. Preallocate metadata before a CUDA mutation that cannot be rolled back.
+5. Once CUDA accepts a mutation, keep the new metadata even if releasing the
+   old reference reports an error. Rolling back would expose dangling
+   parameters.
+6. Never release attachment handles in a CUDA user-object callback; queue the
+   complete bundle for deferred cleanup.
+7. Follow CUDA's external-synchronization requirements when mutating a graph or
+   its attachment metadata.
+
+## Scope
+
+- `GraphAttachmentState` tracks operations performed through cuda-core. Raw
+  driver mutations are not reflected in it.
+- A raw driver clone still receives the CUDA user-object references needed for
+  resource lifetime, but cuda-core does not import metadata for its nodes.
+- `GraphAttachmentState` covers graph definitions. Owners added directly to an
+  executable graph require separate append-only ownership for that executable.
+- Stream capture explicitly retains host callbacks. Other captured operations
+  continue to follow their documented caller-owned lifetime contract.