[Feature] Add writer resource governance: memory limits, spill, cancellation, and metrics

[suxiaogang223]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Native query engines need a Paimon writer to run within the query's resource limits. Today the Rust writer and C FFI do not provide a way to connect the write path to an engine memory tracker, spill memory-heavy work to disk, stop promptly when a query is cancelled, or report writer resource metrics.

The existing write_buffer_size option is a flush threshold, not a total writer memory limit. Writer memory also includes:

• Arrow RecordBatch values retained by primary-key writers;
• active partition and bucket writers;
• file encoder and row-group buffers;
• sorting, deduplication, merging, serialization, and index-generation state;
• file writers being closed by background tasks.

Changing write_buffer_size cannot account for all of this memory or reserve bytes from the embedding query engine before an allocation. An engine such as Apache Doris therefore cannot reliably enforce its query memory limit around a Paimon Rust write.

The C FFI also has no cancellation handle or writer metrics API. A cancelled query may continue CPU work or I/O until the current operation finishes, and the engine cannot report the memory retained or data spilled by the writer.

Solution

This requires changes in both the Rust writer core and the C FFI. FFI callbacks alone are insufficient because the allocations and long-running work occur inside the Rust writer.

Rust writer core

Add an optional resource context shared by all writers in one logical write. Existing callers should retain the current behavior when no context is configured.

The context should support:

• reserving memory before retaining buffers or starting memory-heavy work;
• automatically releasing reservations when the memory is freed;
• returning a distinct resource-exhausted error when memory cannot be reserved;
• checking a cancellation signal;
• configuring a spill directory and spill budget;
• collecting writer metrics.

Memory accounting should initially cover retained Arrow batches, active file/row-group buffers, primary-key sort and merge state, index state, and background file-close tasks. Reservations should use an ownership or guard model so error and cancellation paths release them reliably.

When memory cannot be reserved, the writer should flush safe state or spill intermediate data. Primary-key sorting, deduplication, and merging need a bounded-memory spill path rather than only a lower file flush threshold. If an operation cannot spill safely, it should return a resource-exhausted error.

Spill files should be scoped to one writer or query and removed after success, error, abort, or cancellation. The spilled path must produce the same logical result as the in-memory path.

Long-running operations should check cancellation at useful boundaries, including batch ingestion, partition/bucket dispatch, sort and merge loops, file encoding, background close waits, and prepare_commit. Cancellation should return a distinct error and clean up reservations, tasks, open writers, and spill files without publishing a commit.

Expose a metrics snapshot containing at least:

• current and peak accounted memory;
• input rows and bytes;
• output files and bytes;
• spill operations, files, and bytes;
• flush count and time;
• sort/merge and prepare_commit time;
• cancellation and resource-exhausted counts.

The API should document which values are exact and which are estimates.

C FFI

Expose the resource context through a versioned C structure or an opaque thread-safe handle. An embedding engine needs to:

• provide memory reserve/release callbacks or a fixed memory budget;
• provide spill configuration;
• cancel a writer while another FFI call may be running;
• retrieve a metrics snapshot;
• distinguish cancellation and resource exhaustion from other errors.

A possible callback shape is:

typedef struct paimon_write_resource_callbacks_v1 {
    void *context;
    bool (*try_reserve)(void *context, uint64_t bytes);
    void (*release)(void *context, uint64_t bytes);
    bool (*is_cancelled)(void *context);
} paimon_write_resource_callbacks_v1;

The final design must define callback lifetime, thread-safety, ownership, and whether callbacks can be invoked from Rust runtime threads. A separate resource handle may be preferable because cancellation must work while a write call is blocked in the FFI.

Suggested implementation phases

1. Add the resource context, memory reservations, cancellation, and metrics to the Rust core.
2. Add the C FFI resource handle and error/metrics APIs.
3. Add bounded-memory primary-key sort/merge with spill.
4. Extend accounting and spill coverage to the remaining writer and index paths.

The implementation should demonstrate that:

• a configured writer stays within its accounted budget or returns a resource-exhausted error;
• a low-memory primary-key write spills and produces the same logical result as the in-memory path;
• cancellation stops a long-running write and removes reservations, tasks, and spill files;
• an embedding engine can attach its query memory tracker and read writer metrics.

Anything else?

The closest existing issues found during the search are:

• DataFusion scan planning can exhaust memory by cloning split metadata #665 reduces scan-planning memory use;
• Allow configuring DataFusion runtime resources for SQLContext #673 configures DataFusion runtime resources;
• referenced_files_size: bound memory usage for long snapshot histories #679 bounds memory used for long snapshot histories.

These issues concern scan, SQL runtime, or metadata paths rather than table writers.

Open design questions:

1. Should the core abstraction expose reserve/release callbacks, a Rust memory-pool trait, or both?
2. Which writer allocations can be accounted exactly in the first phase?
3. Should spill be limited to primary-key sort/merge initially?
4. What cancellation latency should the API guarantee?

Willingness to contribute

• I'm willing to submit a PR!

[JingsongLi]

I've opened #914 as an initial step toward a common resource-management API. The proposal is to validate the shared memory-reservation and ownership model in the Reader layer first, then extend the same foundation to the Writer. The current design discussion focuses on Rust and Parquet; C FFI integration can follow once the core contracts are clearer.

The API in #914 consists of:

• ResourceContext: a shared budget and current/peak reservation metrics, passed through ReadBuilder::with_resources or TableRead::with_resources.
• MemoryPool::try_reserve(bytes) / release(bytes): an adapter for an embedding engine's memory budget. Failed reservations leave accounting unchanged; callbacks may run on Rust runtime threads.
• MemoryReservation: an ownership guard that returns the reservation on drop. Reader output reservations follow the shared Arrow buffer owners, including cloned arrays and slices.

This first PR accounts for retained output buffers before emitting them. Decoder working memory, prefetch buffers, and merge state are still outside that coverage, so it does not yet establish a complete Reader memory bound or resolve the Writer requirements in this issue. Existing ReadBudget remains in place. A subsequent step can move its byte accounting onto the common reservation mechanism while retaining separate concurrency and prefetch controls.

We would particularly appreciate a review from Doris maintainers or contributors familiar with its memory-management design. Could a Doris expert compare this interface with Doris's existing design and assess whether it is a suitable integration boundary? In particular:

1. How should these reservations connect to Doris's query memory tracking and reservation model, including avoiding double accounting when Arrow buffers cross the engine/library boundary?
2. Are fallible, non-blocking reservation and infallible release callbacks sufficient, or is an explicit contract needed for coordinating memory reclamation, flush, and spill? Who should initiate that work when admission fails?
3. What execution-context and lifetime guarantees are needed when reservations are acquired or released by asynchronous Rust tasks, or when output buffers outlive the reader?

Feedback on these points would help us settle the common interface before extending accounting deeper into Parquet readers and implementing Writer resource governance.