High heap growth during COPY TO ... (FORMAT VORTEX) bypasses duckdb_memory() accounting and triggers OOM on memory-constrained hosts

[KazukiKandaKK]

Summary

Converting a large Parquet file to Vortex format using COPY 'hits.parquet' TO 'hits.vortex' (FORMAT vortex) triggers OOM kills on a host with ~7.6 GiB RAM. Lowering memory_limit from the ~6.0 GiB default to 4 GB did not prevent or delay the OOM (if anything, it fired slightly earlier); only reducing threads allowed the conversion to complete. In a separate instrumented run, duckdb_memory() accounted for only ~0.29 GiB out of ~7.10 GiB of host memory in use — a ~6.8 GiB accounting gap. A perf trace from another run suggests rapid heap expansion originating from inside the extension binary.

We tested against vortex-data/vortex commit d1fe2dc4 (full coordinates below), which already includes #9505 (623fd06, 145 commits earlier). That PR replaced the extension's internal worker pool with DuckDB-driven batching (prepare_batch_push/flush_batch) and cross-references #8760 (multi-threaded Vortex writes, still open). The OOM described below still reproduces on top of that architecture.

Environment

• Extension build: vortex.duckdb_extension, from duckdb-vortex commit 9a8eb67 (pinned to vortex-data/vortex commit d1fe2dc46d2b947a2304122d0d0423332fcb77a3)
• DuckDB: v1.5.5, Linux x86_64
• Host: AWS EC2 c6a.xlarge (4 vCPU, ~7.6 GiB usable RAM), Ubuntu, with swap
• Workload: ClickBench hits.parquet (~13.8 GiB / 14.8 GB), converted with duckdb -c "LOAD vortex; COPY 'hits.parquet' TO 'hits.vortex' (FORMAT vortex);", preceded by SET threads = N; / SET memory_limit = 'N GB'; for the non-default rows below

Observed Behavior

Each condition tested once (n=1):

threads	memory_limit	Peak RSS	Outcome	Time
4 (default)	default (~6.0 GiB)	~7.08 GiB	OOM-killed	1,330s to kernel OOM
4 (default)	4 GB	~7.08 GiB	OOM-killed	1,105s to kernel OOM
3	default	~7.05 GiB	OOM-killed	1,045s to kernel OOM
2	default	~7.14 GiB	Completed	1,540.3s COPY load time
1	default	~7.12 GiB	Completed	1,439.9s COPY load time

(Peak RSS is the DuckDB process's own VmRSS, sampled every 2s. "Time to kernel OOM" and "COPY load time" are different measurements — wall-clock-to-kill vs. DuckDB's own reported load time — and aren't directly comparable to each other.)

1. memory_limit did not prevent or delay the OOM: peak RSS was essentially unchanged (~7.08 GiB) whether memory_limit was default or 4 GB, and the OOM fired slightly earlier at 4 GB (1,105s vs 1,330s). Within the range tested, only reducing threads to 2 or 1 let the conversion finish.
2. Peak RSS doesn't distinguish success from failure: every run, completed or killed, pushed RSS into the same ~7.05–7.14 GiB band; what differed was whether the process ran out of swap first. With n=1 per condition we're not claiming a monotonic relationship between threads and outcome time — just that every run above threads=2 OOM'd and both threads=1 and threads=2 completed.

Memory Accounting Gap

In a separate threads=2 instrumentation run, polling duckdb_memory() at 1s intervals against free's OS-level used memory: when used peaked at 7.10 GiB, the sum of all 16 duckdb_memory() tags was only ~0.29 GiB (almost entirely EXTERNAL_FILE_CACHE) — a ~6.8 GiB gap. We didn't independently confirm this against the DuckDB process's own RSS specifically (vs. host-wide used), though DuckDB was the only significant consumer on this host.

A separate threads=2 run was profiled with perf record (syscalls:sys_enter_mmap / syscalls:sys_enter_brk, --call-graph dwarf) for its first 5 minutes:

• brk: 6,304 of 6,338 calls (99.5%) had stack frames inside vortex.duckdb_extension, growing the glibc heap (malloc -> __default_morecore -> brk) by ~3.6 GiB in that window. (The extension binary is stripped, so individual Rust symbols aren't resolvable.)
• mmap: none of 637 calls involved vortex.duckdb_extension — all traced to DuckDB's jemalloc-backed Allocator, which duckdb_memory() does account for.

Both measurements above are from separate threads=2 runs (neither is the run in the table), and the perf trace covers only the first 5 minutes — we haven't verified the remaining ~20 minutes or other thread counts.

Source Inspection

vortex-extension (in duckdb-vortex) is a thin FFI wrapper that depends on vortex-duckdb (pinned to the vortex-data/vortex commit above) and builds the vortex.duckdb_extension binary DuckDB loads:

• Neither vortex-extension nor vortex-duckdb declares a #[global_allocator], defaulting to the system allocator (glibc malloc) — consistent with the brk profile above.
• vortex-ffi defines an optional mimalloc allocator, but it was explicitly removed from vortex-python in fix[py]: remove mimalloc in library #7826 because a global allocator in a dynamically loaded library conflicted with the host process (pyarrow). A similar conflict risk likely applies to DuckDB extensions, so adding #[global_allocator] there doesn't look like the right fix.
• duckdb_malloc/duckdb_free are used in vortex-duckdb only for small C-string/blob conversions, not data chunk buffering. In copy.rs, the write is handed to an async task via SESSION.write_options().write(...), executing inside core Vortex compression routines shared across bindings (DataFusion, Python, etc.) with no direct access to DuckDB's allocator APIs.

Questions for Maintainers

1. Is write-path memory during COPY TO expected to operate outside DuckDB's memory_limit controls, with threads as the recommended lever on memory-constrained systems?
2. Is there an existing or planned mechanism to bound write-path buffer allocations in core Vortex?

Happy to share the raw perf data, duckdb_memory() CSV time series, and reproduction scripts.

[KazukiKandaKK]

Related: #9674 is the broader request for a BufferAllocator so Vortex buffer allocations are visible to the host engine. It already names a DuckDB adapter as one example.

The core plumbing (#9668, #9669, #9670, #9671) has landed, and the commit we tested (d1fe2dc4) already includes #9668-#9670. The DuckDB session is still VortexSession::default() with no host allocator (vortex-duckdb/src/lib.rs), and COPY just calls SESSION.write_options().write(...) on that session. #9670 did touch vortex-duckdb, but only to pass BufferAllocatorRef::statically_allocated() into the DuckDB-chunk-to-Vortex-array conversion step that COPY's write path uses (convert/vector.rs, called from copy.rs) — not into the compression/serialization step performed by SESSION.write_options().write(...).

The brk growth described above is consistent with that serialization step still hitting the system allocator. A DuckDB-side BufferAllocator on the session looks like the intended path to get this traffic into duckdb_memory()'s accounting. #9674 explicitly excludes back-pressure and failed allocation from its scope, so accounting alone wouldn't necessarily make memory_limit bound this write.