Generic RT/NRT shared resource system (with MultiChannelBuffer as first class)

[kaoskorobase]

Status

Implemented in #152. Scope split:

• Generic resource pool, OSC verbs (/resource/new, /resource/free), lifecycle notifications (/resource/ready, /resource/error, /resource/destroyed), Methcla_ResourceDef registration API, RT acquire/release primitives, perform_with_resources, and C++ ResourceRef<T> / ResourceDef<R,O> wrappers — Add generic RT/NRT shared resource system (#147) #152.
• AudioBuffer (first concrete Resource class) — AudioBuffer Resource class and built-in plugin #153.
• Synth-side kMethcla_ResourcePort integration — Synth resource port integration (kMethcla_ResourcePort) #151.

Final resource state machine and threading reference live in docs/architecture.md.

---

Motivation

The engine currently has no general-purpose mechanism for sharing data between the RT audio thread and NRT context that can be allocated/freed at runtime via OSC. AudioBus is preallocated at engine init; synth instance memory is RT-owned. There's no story for, e.g., a sample buffer that:

• is allocated/freed via OSC,
• is written by a record synth in RT,
• is read by play synths in RT,
• can be analyzed (onset detection) or processed (heavy DSP) in NRT.

This issue specifies a generic resource protocol — plugins can register new resource types. Resources are identified by integer id; consumers obtain a type-checked pointer via URI match (similar to a dynamic cast). The first concrete Resource class is tracked separately in #153.

Scope

In scope:

• Generic resource definition registration via plugin API.
• OSC verbs for resource lifecycle.
• Synth integration: new kMethcla_ResourcePort port kind. Deferred to Synth resource port integration (kMethcla_ResourcePort) #151.
• NRT-side use via a dispatch primitive that brackets the use with refcount.
• Extension of /node/set to accept i:value for resource ports. Deferred to Synth resource port integration (kMethcla_ResourcePort) #151.

Out of scope (future):

• Concrete Resource classes (first one — AudioBuffer — tracked in AudioBuffer Resource class and built-in plugin #153).
• Per-plugin unload while resources of its class are live.
• Atomic refcount, dying flag, sweeper-style reclamation — unnecessary given methcla's threading topology.
• Reader/writer locks; engine-enforced acquire granularity.
• Mid-life resource swapping under a running synth without going through connect().
• Adding a return value to Methcla_SynthDef.construct so pure-C plugins can signal construct failure — useful but independent.

Prerequisite (separate issue): dead-code cleanup in src/Methcla/Audio/Resource.hpp plus ResourceIdAllocator → IdAllocator rename — tracked in #148. The names freed by that cleanup are reused in this design.

Design decisions

#	Decision
1	Payload-mutability is per-class; the resource pointer published by the engine is itself immutable. ResourceDef declares Mutable or Immutable. Both values are informational (see docs/architecture.md).
2	Default acquire granularity is per-synth-lifetime — acquire at construct, release at destroy, dereference the cached Methcla_Resource* per pass. The protocol equally permits per-pass acquire/release for synths that need tighter free-time bounds or hold the resource only intermittently. Refcount is single-writer RT in either pattern.
3	Resource refs are a new port kind kMethcla_ResourcePort; runtime re-map via /node/set i:node-id i:port-index i:resource-id. Synth options may also carry resource ids for shape-hint cases consumed by port_descriptor.
4	Engine owns a fixed-size resource pool (Options::maxNumResources). Per resource id it tracks {state, uri, resource: Methcla_Resource*, destroy_fn, refcount, free_pending}. Plugin owns the resource's heap allocation.
5	Interface contract: a resource URI names a C struct layout declared in a plugin-supplied header. The engine treats Methcla_Resource* as opaque (mirroring Methcla_Synth* opacity for synth instance bodies). Consumers cast the pointer after URI match. The struct may be POD, function-pointer-heavy, or both — the engine doesn't care.
6	All per-resource state mutation, refcount mutation, and lifecycle dispatch run on RT. NRT is a pure callback path for construct / destroy / perform_with_resources. No atomics needed (see docs/architecture.md).
7	NRT-side use is bracketed by methcla_world_perform_with_resources (RT-side refcount++ on dispatch, auto-release after NRT callback returns). The Methcla_Resource* is valid only for the duration of one perform call.
8	Synth-side bookkeeping is the plugin's responsibility (explicit acquire/release primitives). C++ wrapper provides RAII ResourceRef<T> for the per-synth-lifetime pattern plus a try_acquire flavor returning std::optional.
9	Id encoding: plain int32 resource id; client picks ids (NodeId / AudioBusId precedent).
10	Lifecycle notifications (/resource/ready, /resource/error, /resource/destroyed) originate from RT post-state-transition, dispatched via NRT — mirrors /node/done. Required to avoid races between a NRT direct-notify and a follow-up /synth/new.
11	kMethcla_ResourcePort supports direction (Input / Output) as informational metadata; the engine does not enforce mutability against direction. Methcla_PortDescriptor stays unchanged — URI checking remains in the synth's connect() via methcla_world_resource_acquire(..., expected_uri). Deferred to #151.

Plugin C API

Opaque resource handle and definition struct (added to include/methcla/plugin.h):

typedef void Methcla_Resource;   // opaque handle to a plugin-allocated resource
                                 // (mirrors `typedef void Methcla_Synth;`)

typedef int32_t Methcla_ResourceId;

typedef enum {
    kMethcla_ResourceMutable,
    kMethcla_ResourceImmutable
} Methcla_ResourceMutability;

typedef struct Methcla_ResourceDef {
    const char* uri;
    Methcla_ResourceMutability mutability;

    // NRT context. Parse args, allocate the resource, return it.
    // Return NULL on failure; engine notifies /resource/error.
    Methcla_Resource* (*construct)(Methcla_Host* host,
                                   const void* tag_buffer, size_t tag_size,
                                   const void* arg_buffer, size_t arg_size);

    // NRT context. Free the resource. Must be infallible.
    void (*destroy)(Methcla_Host* host, Methcla_Resource* resource);
} Methcla_ResourceDef;

Methcla_Host gains one field, parallel to register_synthdef:

struct Methcla_Host {
    /* ... existing fields ... */
    void (*register_resource_def)(Methcla_Host*, const Methcla_ResourceDef*);
};

RT-side primitives

Methcla_World gains two fields:

struct Methcla_World {
    /* ... existing fields ... */

    // Returns the resource pointer after URI check; increments refcount.
    // Returns NULL on URI mismatch, resource not Live, free_pending, or out of range.
    Methcla_Resource* (*resource_acquire)(
        Methcla_World*, Methcla_ResourceId, const char* expected_uri);

    // Decrements refcount; if hits 0 and free_pending, posts destroy-cmd to NRT.
    void (*resource_release)(Methcla_World*, Methcla_ResourceId);
};

Consumers cast the returned pointer to the typed interface struct after URI match.

NRT-side primitive

A new callback signature, distinct from Methcla_HostPerformFunction, with the resolved resource pointers passed as a separate argument:

typedef void (*Methcla_PerformWithResourcesFunction)(
    Methcla_Host* host,
    Methcla_Resource* const* resources, size_t num_resources,
    void* user_data);

// On dispatch RT acquires each resource (refcount++ for each).
// If exclusive=true, dispatch fails unless every listed resource has refcount==1
// at the moment of acquire; on failure no acquires are performed.
// On NRT callback return, RT auto-releases all (refcount--).
void (*perform_with_resources)(
    Methcla_World*,
    const Methcla_ResourceId* ids, size_t num_ids,
    bool exclusive,
    Methcla_PerformWithResourcesFunction perform,
    void* user_data);

(Field on Methcla_World, alongside the resource_acquire/resource_release pair shown above.)

C++ wrappers

Two pieces, both added to include/methcla/plugin.hpp in the Methcla::Plugin namespace.

ResourceRef<T> — consumer-side RAII handle

Implements the per-synth-lifetime pattern. The type parameter is the C++ wrapper class for the resource, which carries the URI and the underlying C struct type:

// Throws methcla::ResourceAcquireError on failure.
methcla::ResourceRef<Methcla::Plugin::MyResource> r(world, opts.resourceId);

// std::optional, empty on failure.
auto r = methcla::try_acquire<Methcla::Plugin::MyResource>(world, opts.resourceId);

// In connect(): replace contents, return false on failure (no-op on failure).
bool ok = m_r.try_replace(world, new_id);

Per-pass synths use the C primitives directly without the smart pointer (acquire at process entry, release before return); the C++ wrapper does not provide a per-pass helper.

The ResourceRef<T> name reuses the symbol freed by the dead-code cleanup in #148 (the previous unused ResourceRef<T> = intrusive_ptr<T> alias).

ResourceDef<Resource, Options> — plugin-author-side definition template

Parallel to Methcla::Plugin::SynthDef<Synth, Options, Ports, Flags>. Bridges the C ABI to a C++ class so resource authors write a normal RAII class:

template <class Resource, class Options = NoOptions>
class ResourceDef
{
    static Methcla_Resource* construct(
        Methcla_Host* host,
        const void* tag_buffer, size_t tag_size,
        const void* arg_buffer, size_t arg_size)
    {
        OSCPP::Server::ArgStream args(
            OSCPP::ReadStream(tag_buffer, tag_size),
            OSCPP::ReadStream(arg_buffer, arg_size));
        typename Options::Type opts(args);
        void* mem = methcla_host_alloc(host, sizeof(Resource));
        try {
            new (mem) Resource(HostContext(host), opts);
        } catch (...) {
            methcla_host_free(host, mem);
            return nullptr;
        }
        return static_cast<Methcla_Resource*>(mem);
    }

    static void destroy(Methcla_Host* host, Methcla_Resource* resource) {
        Resource* r = static_cast<Resource*>(resource);
        r->~Resource();
        methcla_host_free(host, r);
    }

public:
    void operator()(Methcla_Host* host, const char* uri,
                    Methcla_ResourceMutability mutability)
    {
        static const Methcla_ResourceDef kDef = {
            uri, mutability, construct, destroy
        };
        host->register_resource_def(host, &kDef);
    }
};

Resource-class authors then register with:

Methcla::Plugin::ResourceDef<MyResourceImpl, MyResourceOptions>()(
    host, METHCLA_MY_RESOURCE_URI, kMethcla_ResourceMutable);

— mirroring the existing Methcla::Plugin::SynthDef<...>()(host, uri) registration idiom.

OSC API

Verb	Args	Direction	Notes
/resource/new	s:definition-name i:resource-id [args]	client → engine	[args] is a single OSC array (matches /synth/new's [synth-options] convention). Parsed by plugin in NRT.
/resource/free	i:resource-id	client → engine	Destroy now if refcount==0; else mark free_pending.
/resource/ready	i:resource-id	engine → client	Resource is Live.
/resource/error	i:resource-id s:reason	engine → client	Construct failed; id returned to Free.
/resource/destroyed	i:resource-id	engine → client	Id returned to Free; reusable.

Synth integration via /node/set — this issue extends the existing verb to accept i:value for the new resource-port case:

Port kind	OSC arg	Notes
kMethcla_ResourcePort	i:value only	deferred to #151

Resource state machine

                 /resource/new (validated, id reserved)
        Free ──────────────────────────────────────────→ Constructing
         ▲                                                    │
         │                                                    │  NRT construct returns
         │                                          ┌─────────┴─────────┐
         │                                          │                   │
         │                                       NULL                 ptr
         │                                          │                   │
         │ NRT destroy returns                      ▼                   ▼
         │ (state → Free)                       (notify              Live
         │                                       /resource/error)     │
         │                                          │                  │ /resource/free
         │                                          │                  │   refcount==0 → Destroying
         │                                          ▼                  │   refcount>0  → free_pending=true,
         └──────────────────────────────────────  Free                 │                  drain via release path
                                                                       ▼
                                                                  Destroying
                                                                       │
                                                                       │ NRT destroy returns
                                                                       └────────→ Free

free_pending is set on a Live id when /resource/free arrives but refcount > 0; the last release to drain refcount then initiates Destroying. The same flag is set on a Constructing id if free arrives mid-construct; transition to Destroying happens immediately after Constructing → Live, so the id never sits Live-and-doomed beyond one transition.

Threading

Event	Thread	Touches
/resource/new arrives	RT	resource Free → Constructing; post NRT construct
NRT construct returns	NRT → posts back	sends publish-cmd to RT
RT drains publish-cmd	RT	resource → Live; post ResourceReadyNotification to NRT
NRT dispatches /resource/ready	NRT	calls packet handler
Synth bind in /synth/new or /node/set	RT	refcount++
Synth destruction (Node::free)	RT	refcount-- for each held resource
/resource/free arrives	RT	if refcount==0 → Destroying + post NRT destroy; else free_pending=true
NRT destroy returns	NRT → posts back	RT marks resource Free; notifies via NRT
perform_with_resources dispatch	RT	per-resource refcount++; post NRT perform
NRT perform returns	NRT → posts back	RT auto-releases all (refcount--)

Critical ordering: EnvironmentImpl::process drains the OSC request queue before m_worker->perform(). Lifecycle notifications must therefore originate from RT after the state transition, dispatched via NRT. Sending notifications directly from NRT would let a client's follow-up /synth/new race the publish-cmd.

Failure modes

Where	Surfaces as
/resource/new validate (id range / id in use / definition unknown)	replyError, RT-synchronous
Plugin construct returns NULL	/resource/error i:id s:reason notification; id returned to Free
/resource/free validate (id invalid / resource not Live)	replyError
Plugin destroy	Contractually infallible
Resource acquire fails in C++ synth's construct	ResourceRef<T> throws → processMessage replyError on the /synth/new
Resource acquire fails in pure-C synth's construct	Plugin self-handles: log via methcla_world_log_line, store a "no-resource" state on the synth instance; synth produces benign output (e.g., silence) until destroyed. Cleaning this up would need a construct return value (out of scope here).
Resource acquire fails in synth connect (re-map)	Retain prior binding, log warning, silent to client
perform_with_resources dispatch with exclusive=true and refcount>1	Synchronous fail; no acquires performed; client-visible error

Mutation contract published to plugin authors

A ResourceDef declares its mutability as either Mutable (suits buffers) or Immutable (suits wavetables, lookup tables, FFT plans). Both values are informational. The engine does not enforce write rejection; the declaration documents the class's contract for consumers and for plugin authors implementing other resource classes.

For Mutable classes, NRT writes are safe only when nothing else holds the resource. Recommended pattern: process-into-fresh-slot — allocate destination Y, dispatch perform_with_resources({X, Y}, ...) reading X and writing Y, optionally re-bind synths from X to Y via /node/set, free X.

For in-place writes, use exclusive=true. The engine rejects dispatch if any other holder exists, catching lifecycle bugs at dispatch time.

RT-side mutation (record synths) on Mutable classes is the user's responsibility, controlled by node ordering. The engine does not detect this.

For Immutable classes, the contract is by convention: the published interface struct typically has no setter methods, and consumers do not cast away const to write. Engine-side enforcement was considered and rejected (see docs/architecture.md).

Engine implementation notes

• Resource pool: internally a fixed array of an engine-private struct (ResourceEntry or similar) at engine init; one entry per Methcla_ResourceId.
• All per-resource fields are plain types (no atomics); state, refcount, free_pending, and the Methcla_Resource* are RT-only.
• NRT work runs on the engine's existing worker thread pool (Utility::WorkerThread, kNumWorkerThreads = 2); the NRT↔RT command FIFOs are the existing m_worker->sendToWorker / sendFromWorker channels, which already handle multi-producer enqueue via internal locking when needsLock = true.
• Multi-worker caveat for plugin authors: Resource-def callbacks (construct, destroy, and the perform callback passed to perform_with_resources) run on the worker pool and can execute concurrently across different worker threads for different resources. The engine's own NRT-side APIs (methcla_host_alloc, methcla_host_free, the NRT→RT command channels) are thread-safe. Plugin authors who introduce plugin-side shared state (e.g., a static cache, a global file-handle pool) must synchronize that state themselves. The engine does not serialize NRT callbacks.
• Notifications follow the existing Notification / NodeNotification pattern in EngineImpl.hpp — define ResourceReadyNotification, ResourceErrorNotification, ResourceDestroyedNotification.
• Synth allocation layout is unchanged at the engine level (no engine-side per-synth resource binding table; the plugin owns its refs).
• URI comparison in resource_acquire is a strcmp; interning at register_resource_def time (so subsequent comparisons are pointer-equality) is a possible future optimization.

Engine shutdown ordering

1. Stop accepting new OSC.
2. Free all nodes (synth destructors release their resource refs).
3. For each remaining Live resource, force-destroy via NRT.
4. Wait for any in-flight Constructing resources to finish; immediately destroy.
5. Drain NRT command FIFOs.
6. Tear down resource pool.

Resource classes remain valid for the lifetime of the engine that registered them. Per-plugin unload is not supported in this iteration.

CONTEXT.md updates (apply when implementing)

• Add Resource entry: "A plugin-registered shared data object managed by the Engine, identified by an integer id and a type URI." _Avoid_: instance (a Synth is an instance; a Resource is the thing the plugin allocates).
• Add ResourceDef entry mirroring SynthDef: "A Resource type registered by a Plugin. Describes the constructor, destructor, and interface URI for one kind of shared Resource." _Avoid_: resource class, resource type.

Other documentation updates (apply when implementing)

Human-facing:

• docs/osc-api.md — document the /resource/* verbs and the /node/set i:value form for resource ports.
• docs/architecture.md — extend the runtime / routing overview to include Resources alongside Synths and AudioBuses.
• CHANGELOG.md — entry under ## [Unreleased], per the project conventions in CLAUDE.md.

Agent-facing:

• CLAUDE.md "Domain and API references" — add a pointer to wherever the resource lifecycle lands (a section of architecture.md/osc-api.md, or a new doc).

Design rationale

The design decisions around RT-only mutation of resource state and informational mutability are documented in docs/architecture.md under the "Resource system" section.

Related issues

• Delete unused refcount / Resource scaffolding in src/Methcla/Audio/Resource.hpp #148 — dead-code cleanup + IdAllocator rename (prerequisite).
• Synth resource port integration (kMethcla_ResourcePort) #151 — synth resource port integration (kMethcla_ResourcePort) (follow-up).
• AudioBuffer Resource class and built-in plugin #153 — first concrete Resource class (AudioBuffer) (follow-up).