[Bug] TopN Lazy Materialization hangs when ORDER BY + LIMIT queries require RowId materialization (4.1.2-rc01)

[yang-666-cn]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Apache Doris
doris-4.1.2-rc01-aec169d2025

What's Wrong?

After upgrading to Apache Doris 4.1.2-rc01, we found that queries using ORDER BY ... LIMIT may hang indefinitely when the query needs to materialize non-key columns.
The problem disappears after disabling TopN Lazy Materialization by setting:
SET GLOBAL experimental_topn_lazy_materialization_threshold = 0;
This suggests the issue is related to the TopN Lazy Materialization / RowId Lookup execution path.

What You Expected?

The query should finish normally.
TopN Lazy Materialization should not hang.

How to Reproduce?

Doris缺陷.docx

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_130eccec93a1

This content is generated by AI for reference only.

Initial assessment: the RowId materialization path is convincingly isolated, but the exact root cause is not yet proven. The strongest code-backed lead is an already-fixed branch-4.1 compatibility issue for tables with light_schema_change=false. The issue currently has no labels.

Verified facts

• The attachment gives a useful path-level reproducer: only projections that defer non-key columns hang; removing LIMIT or setting experimental_topn_lazy_materialization_threshold=0 avoids VMaterializeNode. With the default threshold, limit 10 is eligible for LazyMaterializeTopN, which replaces deferred columns with __DORIS_GLOBAL_ROWID_COL__ below TopN and fetches those columns afterward.
• The reported build is exactly commit aec169d20256a788de448f46737b5fb53ee3b4e3. At that revision, FE does not reject TopN lazy materialization for an OLAP table with light_schema_change=false.
• On that revision, BE rebuilds the scan TabletSchema from FE's columns_desc only when the first column has col_unique_id >= 0. A non-light-schema-change table uses col_unique_id=-1, so the rebuilt schema (including GLOBAL_ROWID_COL) is skipped. This is the failure mechanism documented and fixed by #64441, backported to branch-4.1 by #64634 after the reported RC commit. The fix falls back to the normal scan path for such tables.
• The RC commit also lacks #62513, backported by #64628, which propagates a non-zero RowId-fetch RPC response status from MaterializationOperator. Without it, the useful BE error can be masked instead of being returned cleanly to the client.
• Apache Doris 4.1.3 contains both branch-4.1 fixes above.

Current judgment

If t_entities_host has light_schema_change=false, this is a high-confidence match for the known RC gap, with the missing RowId-fetch status propagation explaining why the client may not receive the underlying BE failure clearly. However, the attached DDL omits the table PROPERTIES section, so that condition is unverified. The evidence does not yet establish a new deadlock or a new RowId lookup defect.

If the table has light_schema_change=true, or the same minimal case still hangs on 4.1.3, the known fix does not explain the report and the blocked runtime location must be captured before assigning a different root cause.

Missing information

Please provide:

1. Exact SHOW CREATE TABLE t_entities_host output, including all PROPERTIES, especially light_schema_change; also state whether this is an existing upgraded table or a table newly created from the posted DDL.
2. The query ID, how long it was allowed to run, the effective query/execution timeout, and whether cancellation succeeds.
3. FE and all three BE log excerpts for that query ID and time window. The most relevant messages contain field name is invalid, multiget_data_v2, Row id fetch failed, file_mapping not found, id_file_map not found, rowset_id not found, or Materialization Sink node id.
4. A live runtime profile while blocked, including the VMaterializeNode/Materialization operator, MaxRpcTime, MergeResponseTime, and any RowIDFetcher child profiles, plus the complete EXPLAIN VERBOSE plan.
5. If it still reproduces on 4.1.3, a thread dump from the coordinator BE and any BE handling multiget_data_v2 at the same timestamp.

Recommended next steps

1. First confirm the table property and retry the minimal SQL on 4.1.3 (or another build known to include branch-4.1: [fix](topn) Skip TopN lazy materialization when light_schema_change=false #64441 #64634 and branch-4.1: [Exec](status) materialization_opertor return the error status by row_id_fetcher #62513 #64628) without disabling the optimization.
2. If 4.1.3 succeeds and the table has light_schema_change=false, treat the report as covered by the existing fallback and error-propagation fixes; the current threshold setting remains a valid temporary workaround for the RC build.
3. If it persists with light_schema_change=true on 4.1.3, keep the issue open as a distinct bug and use the profile/log/thread evidence above to determine whether it is waiting in the RowId RPC scheduler, inside read_batch_doris_format_row, or after RPC completion during response merging.

[yang-666-cn]

1、"light_schema_change" = "true"
2、查询时，在查询到非key字段且增加LIMIT时，The query hangs indefinitely.
3、我并不能提供其他相关日志
4、配置文件中修改了IP地址、线程池大小、句柄，无其他特殊修改