Query fails under disaggregated arch #10211
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
Deploy a next-gen cluster and add tests using #10189
Add the following line to tests/fullstack-test-next-gen/run.sh
${COMPOSE} -f next-gen-cluster.yaml -f disagg_tiflash.yaml exec -T tiflash-wn0 bash -c 'cd /tests ; ./run-test.sh fullstack-test-index/inverted'
/tiflash/tests/fullstack-test-next-gen [feature/next-gen*]
> ENABLE_NEXT_GEN=true ./run.sh
...
[+] Running 7/7
✔ Network fullstack-test-next-gen_default Created 0.0s
✔ Container fullstack-test-next-gen-minio0-1 Started 0.4s
✔ Container fullstack-test-next-gen-pd0-1 Started 0.4s
✔ Container fullstack-test-next-gen-tikv0-1 Started 0.5s
✔ Container fullstack-test-next-gen-tiflash-cn0-1 Started 0.6s
✔ Container fullstack-test-next-gen-tidb0-1 Started 0.6s
✔ Container fullstack-test-next-gen-tiflash-wn0-1 Started 0.7s
+ wait_next_gen_env
...
fullstack-test-index/inverted/datetime-type.test: Running
parsing file: `fullstack-test-index/inverted/datetime-type.test`
16:04:58.674084 running mysql> drop table if exists test.t;
16:04:58.687991 running mysql> CREATE TABLE test.t (d date, t time, dt datetime, ts timestamp, y year);
16:04:58.721273 running mysql> INSERT INTO test.t VALUES ('2022-01-01', '10:00:00', '2022-01-01 12:00:00', '2022-01-01 12:00:00', 2022);
16:04:58.736238 running mysql> INSERT INTO test.t VALUES ('2023-01-01', '11:00:00', '2023-01-01 12:00:00', '2023-01-01 12:00:00', 2023);
16:04:58.748919 running mysql> INSERT INTO test.t VALUES ('2024-01-01', '12:00:00', '2024-01-01 12:00:00', '2024-01-01 12:00:00', 2024);
16:04:58.760686 running mysql> INSERT INTO test.t VALUES ('2025-01-01', '13:00:00', '2025-01-01 12:00:00', '2025-01-01 12:00:00', 2025);
16:04:58.772562 running mysql> INSERT INTO test.t VALUES ('2026-01-01', '14:00:00', '2026-01-01 12:00:00', '2026-01-01 12:00:00', 2026);
16:04:58.784081 running mysql> ALTER TABLE test.t SET TIFLASH REPLICA 1;
16:04:58.817993 running func> wait_table test t
16:04:59.872584 running mysql> CREATE COLUMNAR INDEX idx_d ON test.t (d) USING INVERTED;
16:04:59.987146 running mysql> CREATE COLUMNAR INDEX idx_t ON test.t (t) USING INVERTED;
16:05:00.103082 running mysql> CREATE COLUMNAR INDEX idx_dt ON test.t (dt) USING INVERTED;
16:05:00.222617 running mysql> CREATE COLUMNAR INDEX idx_ts ON test.t (ts) USING INVERTED;
16:05:00.338151 running mysql> CREATE COLUMNAR INDEX idx_y ON test.t (y) USING INVERTED;
16:05:00.458902 running mysql> alter table test.t compact tiflash replica;
16:05:00.665970 running mysql> set tidb_isolation_read_engines='tiflash'; SELECT * FROM test.t WHERE d = '2022-01-01';
16:05:00.920704 running mysql> set tidb_isolation_read_engines='tiflash'; SELECT * FROM test.t WHERE d > '2024-01-01';
16:05:01.075477 running mysql> set tidb_isolation_read_engines='tiflash'; SELECT * FROM test.t WHERE t = '12:00:00';
16:05:02.930546 running mysql> set tidb_isolation_read_engines='tiflash'; SELECT * FROM test.t WHERE t > '12:00:00';
File: fullstack-test-index/inverted/datetime-type.test
Error line: 62
Error: set tidb_isolation_read_engines='tiflash'; SELECT * FROM test.t WHERE t = '12:00:00';
Result:
ERROR 1105 (HY000) at line 1: other error for mpp stream: Code: 0, e.displayText() = DB::TiFlashException: The data type Nullable(MyDuration(0)) from tidb table scan schema is different from the data type Nullable(Int64) from tiflash storage schema,
table id is 138,
column index is 1,
column name from tidb table scan is table_scan_1,
column name from tiflash storage is exchange_receiver_1, e.what() = DB::TiFlashException,
Expected:
+------------+----------+---------------------+---------------------+------+
| d | t | dt | ts | y |
+------------+----------+---------------------+---------------------+------+
| 2024-01-01 | 12:00:00 | 2024-01-01 12:00:00 | 2024-01-01 12:00:00 | 2024 |
+------------+----------+---------------------+---------------------+------+
Error: executing /usr/libexec/docker/cli-plugins/docker-compose -f next-gen-cluster.yaml -f disagg_tiflash.yaml exec -T tiflash-wn0 bash -c cd /tests ; ./run-test.sh fullstack-test-index/inverted: exit status 1
2. What did you expect to see? (Required)
3. What did you see instead (Required)
16:05:02.930546 running mysql> set tidb_isolation_read_engines='tiflash'; SELECT * FROM test.t WHERE t > '12:00:00';
File: fullstack-test-index/inverted/datetime-type.test
Error line: 62
Error: set tidb_isolation_read_engines='tiflash'; SELECT * FROM test.t WHERE t = '12:00:00';
Result:
ERROR 1105 (HY000) at line 1: other error for mpp stream: Code: 0, e.displayText() = DB::TiFlashException: The data type Nullable(MyDuration(0)) from tidb table scan schema is different from the data type Nullable(Int64) from tiflash storage schema,
table id is 138,
column index is 1,
column name from tidb table scan is table_scan_1,
column name from tiflash storage is exchange_receiver_1, e.what() = DB::TiFlashException,
Expected:
+------------+----------+---------------------+---------------------+------+
| d | t | dt | ts | y |
+------------+----------+---------------------+---------------------+------+
| 2024-01-01 | 12:00:00 | 2024-01-01 12:00:00 | 2024-01-01 12:00:00 | 2024 |
+------------+----------+---------------------+---------------------+------+
Error: executing /usr/libexec/docker/cli-plugins/docker-compose -f next-gen-cluster.yaml -f disagg_tiflash.yaml exec -T tiflash-wn0 bash -c cd /tests ; ./run-test.sh fullstack-test-index/inverted: exit status 1
[2025/05/30 16:05:02.922 +08:00] [ERROR] [MPPTask.cpp:655] ["task running meets error: Code: 0, e.displayText() = DB::TiFlashException: The data type Nullable(MyDuration(0)) from tidb table scan schema is different from the data type Nullable(Int64) from tiflash storage schema, \n table id is 138, \n column index is 1, \n column name from tidb table scan is table_scan_1, \n column name from tiflash storage is exchange_receiver_1, e.what() = DB::TiFlashException, Stack trace:\n\n\n 0x5635fe34590e\tStackTrace::StackTrace() [tiflash+80660750]\n \tdbms/src/Common/StackTrace.cpp:23\n 0x5635fe3351b2\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+80593330]\n \tdbms/src/Common/Exception.h:46\n 0x5635fe377b87\tDB::TiFlashException::TiFlashException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::TiFlashError const&) [tiflash+80866183]\n \tdbms/src/Common/TiFlashException.h:263\n 0x56360a01597e\tDB::(anonymous namespace)::buildTableScanProjectionCols(long, std::__1::vector<DB::NameAndTypePair, std::__1::allocator<DB::NameAndTypePair>> const&, DB::Block const&) [tiflash+278645118]\n \tdbms/src/Flash/Planner/Plans/PhysicalTableScan.cpp:56\n 0x56360a0151c0\tDB::PhysicalTableScan::buildProjection(DB::PipelineExecutorContext&, DB::PipelineExecGroupBuilder&) [tiflash+278643136]\n \tdbms/src/Flash/Planner/Plans/PhysicalTableScan.cpp:172\n 0x56360a015122\tDB::PhysicalTableScan::buildPipeline(DB::PipelineBuilder&, DB::Context&, DB::PipelineExecutorContext&) [tiflash+278642978]\n \tdbms/src/Flash/Planner/Plans/PhysicalTableScan.cpp:139\n 0x5636092a5021\tDB::PhysicalPlanNode::buildPipeline(DB::PipelineBuilder&, DB::Context&, DB::PipelineExecutorContext&) [tiflash+264552481]\n \tdbms/src/Flash/Planner/PhysicalPlanNode.cpp:142\n 0x5636092a5021\tDB::PhysicalPlanNode::buildPipeline(DB::PipelineBuilder&, DB::Context&, DB::PipelineExecutorContext&) [tiflash+264552481]\n \tdbms/src/Flash/Planner/PhysicalPlanNode.cpp:142\n 0x5636092922c2\tDB::PhysicalPlan::toPipeline(DB::PipelineExecutorContext&, DB::Context&) [tiflash+264475330]\n \tdbms/src/Flash/Planner/PhysicalPlan.cpp:335\n 0x5636091ef3a4\tDB::PipelineExecutor::PipelineExecutor(std::__1::shared_ptr<MemoryTracker> const&, DB::AutoSpillTrigger*, std::__1::function<void (std::__1::shared_ptr<DB::OperatorSpillContext> const&)> const&, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [tiflash+263807908]\n \tdbms/src/Flash/Executor/PipelineExecutor.cpp:46\n 0x56360903e883\tstd::__1::__unique_if<DB::PipelineExecutor>::__unique_single std::__1::make_unique[abi:ue170006]<DB::PipelineExecutor, std::__1::shared_ptr<MemoryTracker>&, std::nullptr_t, std::nullptr_t, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(std::__1::shared_ptr<MemoryTracker>&, std::nullptr_t&&, std::nullptr_t&&, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [tiflash+262035587]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__memory/unique_ptr.h:689\n 0x563609039d22\tDB::(anonymous namespace)::executeAsPipeline(DB::Context&, bool) [tiflash+262016290]\n \tdbms/src/Flash/executeQuery.cpp:173\n 0x56360903981f\tDB::queryExecute(DB::Context&, bool) [tiflash+262015007]\n \tdbms/src/Flash/executeQuery.cpp:192\n 0x563609129ef8\tDB::MPPTask::preprocess() [tiflash+262999800]\n \tdbms/src/Flash/Mpp/MPPTask.cpp:484\n 0x56360912a9a4\tDB::MPPTask::runImpl() [tiflash+263002532]\n \tdbms/src/Flash/Mpp/MPPTask.cpp:531\n 0x56360913066d\tDB::MPPTask::run()::$_0::operator()() const [tiflash+263026285]\n \tdbms/src/Flash/Mpp/MPPTask.cpp:203\n 0x563609130645\tdecltype(std::declval<DB::MPPTask::run()::$_0&>()()) std::__1::__invoke[abi:ue170006]<DB::MPPTask::run()::$_0&>(DB::MPPTask::run()::$_0&) [tiflash+263026245]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__type_traits/invoke.h:340\n 0x563609130605\tvoid std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:ue170006]<DB::MPPTask::run()::$_0&>(DB::MPPTask::run()::$_0&) [tiflash+263026181]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__type_traits/invoke.h:415\n 0x5636091305dd\tstd::__1::__function::__alloc_func<DB::MPPTask::run()::$_0, std::__1::allocator<DB::MPPTask::run()::$_0>, void ()>::operator()[abi:ue170006]() [tiflash+263026141]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__functional/function.h:192\n 0x56360912f8d9\tstd::__1::__function::__func<DB::MPPTask::run()::$_0, std::__1::allocator<DB::MPPTask::run()::$_0>, void ()>::operator()() [tiflash+263022809]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__functional/function.h:363\n 0x5635fe47d792\tstd::__1::__function::__value_func<void ()>::operator()[abi:ue170006]() const [tiflash+81938322]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__functional/function.h:517\n 0x5635fe4707e5\tstd::__1::function<void ()>::operator()() const [tiflash+81885157]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__functional/function.h:1168\n 0x5636070f2c95\tdecltype(std::declval<std::__1::function<void ()>>()()) std::__1::__invoke[abi:ue170006]<std::__1::function<void ()>>(std::__1::function<void ()>&&) [tiflash+229219477]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__type_traits/invoke.h:340\n 0x5636070f2c79\tdecltype(auto) std::__1::__apply_tuple_impl[abi:ue170006]<std::__1::function<void ()>, std::__1::tuple<>>(std::__1::function<void ()>&&, std::__1::tuple<>&&, std::__1::__tuple_indices<>) [tiflash+229219449]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/tuple:1825\n 0x5636070f2c4d\tdecltype(auto) std::__1::apply[abi:ue170006]<std::__1::function<void ()>, std::__1::tuple<>>(std::__1::function<void ()>&&, std::__1::tuple<>&&) [tiflash+229219405]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/tuple:1834\n 0x5636070f2bfb\tauto DB::wrapInvocable<std::__1::function<void ()>>(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+229219323]\n \tdbms/src/Common/wrapInvocable.h:36\n 0x5636070f35c9\tDB::ExecutableTask<auto DB::wrapInvocable<std::__1::function<void ()>>(bool, std::__1::function<void ()>&&)::'lambda'()>::execute() [tiflash+229221833]\n \tdbms/src/Common/ExecutableTask.h:50\n 0x5635fe4c9d68\tDB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask>>&) [tiflash+82251112]\n \tdbms/src/Common/DynamicThreadPool.cpp:124\n 0x5635fe4c99ea\tDB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+82250218]\n \tdbms/src/Common/DynamicThreadPool.cpp:138\n 0x5635fe4d36c4\tdecltype(*std::declval<DB::DynamicThreadPool*>().*std::declval<void (DB::DynamicThreadPool::* const&)(unsigned long)>()(std::declval<unsigned long&>())) std::__1::__invoke[abi:ue170006]<void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*, unsigned long&, void>(void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&) [tiflash+82290372]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__type_traits/invoke.h:308\n 0x5635fe4d3605\tstd::__1::invoke_result<void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*, unsigned long&>::type std::__1::invoke[abi:ue170006]<void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&) [tiflash+82290181]\n \t/DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n 0x5635fe4d3573\tauto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, unsigned long>(auto&&...) const [tiflash+82290035]\n \tdbms/src/Common/ThreadFactory.h:48"] [source="MPP<gather_id:1, query_ts:1748592301085370548, local_query_id:18, server_id:1256, start_ts:458382980162781187,task_id:1>"] [thread_id=348]
4. What is your TiFlash version? (Required)
master