[Bug](High) Large ROW binlog Broker Load fails with COW::assert_mutable: use_count() > 1

[MoanasDaddyXu]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Reproduced on doris-0.0.0-1590e7e3228 in a non-cloud deployment with 1 FE and 3 BEs.

Regression case:

regression-test/suites/stress/job_and_mtmv/ivm_ab/ivm_performance_scale_matrix_test.groovy

What's Wrong?

A 107-column UNIQUE KEY Merge-on-Write table was configured with ROW binlog and historical values. A large Broker Load completed successfully in the first round, but a second load of the same source data with a different business-key prefix was cancelled after the BE hit an internal copy-on-write assertion:

[INTERNAL_ERROR][E6] COW::assert_mutable: use_count() > 1

The first failure occurred while preparing ROW binlog source data. The same assertion was later observed in the normal vertical segment flush path.

This is not a regression-test assertion failure: the load itself entered CANCELLED after the BE returned the internal error.

What You Expected?

A valid UNIQUE KEY MOW + ROW binlog Broker Load should complete successfully. If the workload exceeds a supported resource limit, Doris should return a clear resource-limit error instead of violating the copy-on-write ownership invariant in IColumn::mutate().

How to Reproduce?

Create a wide UNIQUE KEY MOW table with the following relevant properties:

enable_unique_key_merge_on_write = true
binlog.enable = true
binlog.format = ROW
binlog.need_historical_value = true
inverted_index_storage_format = V3
AUTO PARTITION BY MONTH
DISTRIBUTED BY HASH(trace_id) BUCKETS 128

Run two large Broker Loads against the table. The reproduced workload used 1,000 source files and the following load settings:

load_parallelism = 3
exec_mem_limit = 8589934592
timeout = 43200
max_filter_ratio = 0.1

Use a different business-key prefix in the second round so that it does not overwrite rows from the first round.

Current reproduction scale:

round 1: ScannedRows = 2,500,000,000, succeeded
round 2: ScannedRows = 1,052,674,279, cancelled
round 2: LoadBytes = 906,586,501,986
round 2: FilteredRows = 0

The issue has reproduced at approximately 1 TiB input scale. A smaller standalone reproduction has not been identified yet.

Regression Test Result

Primary stack:

doris::IColumn::mutate() const && at ../src/core/cow.h:325
doris::OlapBlockDataConvertor::OlapColumnDataConvertorSimple<...>::convert_to_olap()
doris::OlapBlockDataConvertor::convert_column_data(unsigned long)
doris::segment_v2::RowBinlogSourceDataWriter::prepare_by_source_block(...)
doris::segment_v2::RowBinlogSegmentWriter::append_block(...)
doris::SegmentFlusher::flush_single_block(...)
doris::SegmentCreator::flush_single_block(...)
doris::BaseBetaRowsetWriter::flush_memtable(...)
doris::FlushToken::_flush_memtable_impl(...)
doris::FlushToken::_flush_group_memtable(...)
doris::PartOfGroupMemtableFlushTask::run()
doris::ThreadPool::dispatch_thread()

The assertion was also observed through the regular segment writer path:

doris::IColumn::mutate()
doris::OlapBlockDataConvertorSimple<...>::convert_to_olap()
doris::segment_v2::VerticalSegmentWriter::write_batch()
doris::SegmentFlusher::_add_rows()
doris::SegmentFlusher::flush_single_block()

Initial Analysis

The first load used the same schema, data source, field mapping, and load configuration and completed successfully. The failed load had FilteredRows = 0. All BEs remained alive, disk usage was about 50%, and the cluster still had available memory; there was no OOM, no-space error, node loss, or connection timeout.

Under large concurrent memtable flushes, the ROW binlog source block and the regular segment writer may share a column object that is later mutated in place. This would cause IColumn::mutate() const && to observe use_count() > 1.

Please check the ownership and clone/mutate boundaries among:

RowBinlogSourceDataWriter::prepare_by_source_block
shared/group memtable flush
OlapBlockDataConvertorSimple
VerticalSegmentWriter

Anything Else?

Tracking issues:

• [Tracking][Binlog] Binlog feature umbrella — all sub-issues tracked here #65265
• [Feature] Track incremental computation with row binlog, Table Stream, and MTMV #65418

The remaining reproduction work is to reduce the data size and isolate the effect of bucket count, ROW binlog, and inverted indexes.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Doris-Breakwater]

Initial assessment: confirmed BE copy-on-write race on the reported commit, with high confidence in the root cause. The failure is not explained by a supported resource limit. The same unsafe ownership pattern is still present on current master (dc4fc151b997a6d1909e01ebdc6bcc60eed5b8a2). This should be treated as a high-priority binlog/storage stability defect. The issue currently has no labels, assignee, milestone, or linked fix.

Verified code path

1. When ROW binlog uses a GroupRowsetWriter, FlushToken::submit() creates two independently scheduled tasks, one for the normal data writer and one for the row-binlog writer. _memtable2block() constructs the block once and returns the same SharedMemtable::block to both tasks; each task then passes it to its child writer (call site).

2. SegmentFlusher makes only a shallow Block copy. With binlog disabled in the data child context it selects VerticalSegmentWriter; with binlog enabled in the binlog child context it selects RowBinlogSegmentWriter (selection). This accounts for both reported stacks reaching the same converter from the two concurrent sub-tasks.

3. OlapBlockDataConvertor::set_source_column() retains another ColumnPtr to the selected top-level column (code). For a nullable simple column, however, OlapColumnDataConvertorSimple::convert_to_olap() obtains the nested column by reference and calls:

   std::move(nullable_column->get_nested_column()).mutate()

   (code). This does not transfer or replace the ColumnNullable owner slot. Copies of the top-level ColumnPtr therefore do not raise the nested column's refcount; both flush threads can observe that nested node as uniquely owned and try to acquire a mutable pointer to it.

4. The exact exception follows from the two separate refcount observations in COW: shallow_mutate() first checks use_count(), then the unique path calls assert_mutable(), which checks it again (code). One thread can pass the first check while the count is 1; another thread then returns a mutable pointer and increments the count; the first thread reaches assert_mutable() with a count greater than 1 and emits the reported error.

   A different interleaving can let both threads pass the assertion and create simultaneous mutable aliases to the nested column. That is a genuine ownership violation, not merely an over-strict diagnostic. It is especially relevant for nullable FLOAT/DOUBLE, because this converter normalizes NaNs in place in the same function; other simple types still take an unnecessary mutable alias.

This also explains why round 1 may succeed and round 2 fail: the trigger is scheduling-dependent, and a wide schema plus many memtable flushes creates many opportunities for the race. The present evidence does not show that the business-key prefix, historical-value lookup, 128 buckets, inverted indexes, memory pressure, or input bytes are independently causal. There is no supplied evidence of persisted data corruption, but the possible shared mutable alias means the problem should not be classified as only a load-cancellation diagnostic.

Information still useful

The code and stacks are sufficient to confirm the ownership defect. The following artifacts would identify the exact runtime column and make the regression portable:

• The complete demangled stack for each occurrence, preserving the actual OlapColumnDataConvertorSimple<T> specialization, plus the failing column name/id, type, and nullability.
• The complete SHOW CREATE TABLE, especially all nullable simple columns and any FLOAT/DOUBLE columns.
• Correlated BE log excerpts for the data and row-binlog sub-tasks with timestamp, load ID, tablet ID, rowset/segment ID, and thread ID.
• The referenced ivm_performance_scale_matrix_test.groovy, its data generator, and any required source-file manifest. That path is not present in the public repository at the reported commit or on current master, so maintainers cannot currently run the stated case.

A Doris query profile is not required to establish this COW race.

Recommended next steps

1. Fix the converter ownership boundary. Do not call rvalue mutate() on a borrowed nested-column reference. Either detach through an owned ColumnPtr/the parent owner slot before accessing a writable nested node, or keep the normal conversion path const and create a private writable temporary only for types that actually require normalization. A whole-block deep copy or serializing the two flush tasks would mask the race at substantially broader cost and should not substitute for fixing the converter contract.

2. Add a deterministic BE test using the real grouped memtable flush path, a two-thread pool, and nullable fixed-width columns. Synchronize the data and row-binlog writers at conversion so both touch the same nested node. Include nullable FLOAT/DOUBLE with NaN, verify the source block remains unchanged, and verify that both the data and binlog segments finish correctly. The existing group-flush test uses non-nullable columns and mock writers, so it does not exercise this converter race.

3. Exercise both normal segment-writer variants after the fix: vertical enabled and disabled, ROW binlog with historical values enabled and disabled, and repeated concurrent flushes under TSAN if available.

4. Audit other calls that apply rvalue mutate() to a reference returned from a composite column. The safe API must either consume an owning pointer or replace the composite's owner slot.

5. Add the repository's existing binlog, area/storage, area/load, kind/stability, and kind/need-regression-test labels as appropriate, then assign a BE storage/binlog owner. Backport evaluation should be based on supported branches containing both the grouped ROW-binlog flush path and the enforced COW contract; no internal branch/tag naming should be inferred from this development build.

Breakwater-GitHub-Analysis-Slot: slot_ce2778b8bd08