diff --git a/be/benchmark/parquet/AGENTS.md b/be/benchmark/parquet/AGENTS.md index 4c1d3cf4d5e197..da51348852002e 100644 --- a/be/benchmark/parquet/AGENTS.md +++ b/be/benchmark/parquet/AGENTS.md @@ -51,13 +51,13 @@ be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^ParquetDecoder/' # currently 228 be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -c '^ParquetKernel/' # currently 92 + | grep -c '^ParquetKernel/' # currently 292 be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^ParquetSelection/' # currently 25 be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -c '^ParquetReader/' # currently 167 + | grep -c '^ParquetReader/' # currently 169 be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^FileScannerExpr/' # currently 8 @@ -146,13 +146,19 @@ cache to manufacture a cold run. | DELTA_LENGTH_BYTE_ARRAY | BYTE_ARRAY | | DELTA_BYTE_ARRAY | BYTE_ARRAY | -`ParquetKernel` contains 92 cases across six decode and selection stages: BYTE_STREAM_SPLIT, -DELTA_PREFIX_SUM, DICTIONARY_GATHER, NULLABLE_EXPAND, RAW_PREDICATE, and NESTED_SELECTION. It covers +`ParquetKernel` contains 292 cases across seven decode and selection stages: BYTE_STREAM_SPLIT, +DELTA_PREFIX_SUM, DICTIONARY_GATHER, NULLABLE_EXPAND, NULLABLE_SELECTION, RAW_PREDICATE, and +NESTED_SELECTION. It covers the applicable four- and eight-byte types, three dictionary working-set sizes, 0% through 90% null rates with both placement patterns, 0% through 100% raw-predicate selectivities, and 1%, 10%, and 50% nested parent-row selectivities with both placement patterns. Nested selection registers the legacy and fused implementations in the same binary and validates both against an independent source-level oracle before timing. +Nullable selection contributes 200 legacy/fused cases across five selectivities, five null rates, +and independent clustered or alternating selection/null placement. Each pair is validated for +identical physical ranges and null maps before timing. Treat no-NULL, low-NULL, and clustered +level-plan cases as negative controls: production fusion is gated to batches with at least 1,024 +rows, at least 10% NULLs, and materially fragmented definition-level runs. `ParquetSelection` contains 25 cases that isolate the selection-vector work used by Parquet predicate evaluation. It measures identity initialization, one raw-row filter, and two successive @@ -183,6 +189,10 @@ Except for the axis being varied, reader cases inherit the baseline: nullable IN alternating 10% nulls, 10% selectivity, 32 columns, predicate at column zero, and predicate plus payload projection. +Two dedicated multi-column OR cases scan the same Page Index fixture and change only the Doris Page +Index switch. They retain the complete residual expression and validate the same selected row +count before reporting throughput. + ## How decoder data is generated Decoder pages are constructed in memory before the timed loop. There is no Parquet file, Python @@ -340,7 +350,7 @@ be simulated by silently changing the local reader benchmark. ## Current validation record -The current expected registration counts are 228 decoder, 92 kernel, 25 selection, 167 reader, and +The current expected registration counts are 228 decoder, 92 kernel, 25 selection, 169 reader, and 8 expression-lifecycle cases. A smoke run is an execution record only, not a reviewed performance baseline, because repetitions, host isolation, warmups, cache control, `perf` data, variance, and before/after comparison are not collected. diff --git a/be/benchmark/parquet/README.md b/be/benchmark/parquet/README.md index 156302b24238d1..b4761855f5ceae 100644 --- a/be/benchmark/parquet/README.md +++ b/be/benchmark/parquet/README.md @@ -45,14 +45,21 @@ be/output/lib/benchmark_test \ ## SIMD kernel cases -`ParquetKernel` isolates six decode and selection stages from reader setup and virtual consumer +`ParquetKernel` isolates seven decode and selection stages from reader setup and virtual consumer overhead: byte-stream-split transpose, delta prefix sum, numeric dictionary gather, nullable -expansion, raw predicate evaluation, and repeated-level sparse selection. It covers the applicable +expansion, nullable selection planning, raw predicate evaluation, and repeated-level sparse +selection. It covers the applicable 4-byte and 8-byte integer and floating-point physical types, raw-predicate selectivities from 0% through 100%, and nullable rates from 0% through 90% with clustered and alternating placement. Nested selection covers 1%, 10%, and 50% surviving parent rows with both placement patterns. Each nested-selection scenario registers both `impl_legacy` and `impl_fused`; both paths use the same source levels and are checked against an independent oracle before timing. +Nullable selection planning registers legacy and fused pairs across five selectivities, five null +rates, and independent clustered or alternating selection/null placement. Both implementations are +checked for identical physical ranges and null maps before timing. The full matrix also acts as a +negative control: production fusion is limited to batches with at least 1,024 rows, at least 10% +NULLs, and fragmented definition-level runs; no-NULL, low-NULL, and clustered pages retain the +legacy planner. Dictionary gather uses 32-, 4,096-, and 262,144-entry working sets to separate cache-resident and cache-miss-dominated behavior. @@ -130,6 +137,18 @@ be/output/lib/benchmark_test \ --benchmark_min_time=1s ``` +The multi-column OR pair scans the same ColumnIndex/OffsetIndex fixture and changes only the Doris +Page Index switch. Both variants retain the full residual expression, so the comparison measures +metadata pruning without changing result semantics: + +```shell +be/output/lib/benchmark_test \ + --benchmark_filter='^ParquetReader/multi_column_or/page_index_(off|on)$' \ + --benchmark_min_time=1s \ + --benchmark_repetitions=10 \ + --benchmark_report_aggregates_only=true +``` + Every result reports throughput plus `raw_rows`, `selected_rows`, `fixture_bytes`, `ns/raw_row`, and (when at least one row survives) `ns/selected_row`. Keep CPU frequency, build type, compiler, machine placement, and benchmark filters fixed when comparing two commits. diff --git a/be/benchmark/parquet/benchmark_parquet_kernels.hpp b/be/benchmark/parquet/benchmark_parquet_kernels.hpp index 619d58fd8cf8ac..de3a15c254b8db 100644 --- a/be/benchmark/parquet/benchmark_parquet_kernels.hpp +++ b/be/benchmark/parquet/benchmark_parquet_kernels.hpp @@ -63,6 +63,191 @@ struct NestedSelectionScratch { size_t ancestor_null_count = 0; }; +struct NullableSelectionScratch { + format::parquet::native::ColumnSelectVector legacy_selection; + ParquetSelection physical_selection; + NullMap output_nulls; + NullMap selected_nulls; + size_t num_filtered = 0; +}; + +inline void append_nullable_run(std::vector* runs, bool is_null, size_t run_length, + bool* previous_is_null) { + if (runs->empty()) { + if (is_null) { + runs->push_back(0); + } + } else if (*previous_is_null == is_null) { + runs->push_back(0); + } + while (run_length > USHRT_MAX) { + runs->push_back(USHRT_MAX); + runs->push_back(0); + run_length -= USHRT_MAX; + } + runs->push_back(static_cast(run_length)); + *previous_is_null = is_null; +} + +inline std::vector build_nullable_runs(const NullMap& nulls) { + std::vector runs; + bool previous_is_null = false; + size_t row = 0; + while (row < nulls.size()) { + const bool is_null = nulls[row] != 0; + const size_t begin = row++; + while (row < nulls.size() && (nulls[row] != 0) == is_null) { + ++row; + } + append_nullable_run(&runs, is_null, row - begin, &previous_is_null); + } + return runs; +} + +inline Status run_legacy_nullable_selection(NullableSelectionScratch* scratch, + const std::vector& null_runs, + size_t num_values, + format::parquet::native::FilterMap* filter) { + using ReadType = format::parquet::native::ColumnSelectVector::DataReadType; + scratch->output_nulls.clear(); + scratch->selected_nulls.clear(); + scratch->physical_selection.ranges.clear(); + scratch->physical_selection.total_values = 0; + scratch->physical_selection.selected_values = 0; + RETURN_IF_ERROR(scratch->legacy_selection.init(null_runs, num_values, &scratch->output_nulls, + filter, 0)); + scratch->num_filtered = scratch->legacy_selection.num_filtered(); + + size_t physical_cursor = 0; + ReadType type; + while (const size_t run_length = scratch->legacy_selection.get_next_run(&type)) { + switch (type) { + case ReadType::CONTENT: + if (!scratch->physical_selection.ranges.empty() && + scratch->physical_selection.ranges.back().first + + scratch->physical_selection.ranges.back().count == + physical_cursor) { + scratch->physical_selection.ranges.back().count += run_length; + } else { + scratch->physical_selection.ranges.push_back( + {.first = physical_cursor, .count = run_length}); + } + scratch->physical_selection.selected_values += run_length; + scratch->selected_nulls.resize_fill(scratch->selected_nulls.size() + run_length, 0); + physical_cursor += run_length; + break; + case ReadType::NULL_DATA: + scratch->selected_nulls.resize_fill(scratch->selected_nulls.size() + run_length, 1); + break; + case ReadType::FILTERED_CONTENT: + physical_cursor += run_length; + break; + case ReadType::FILTERED_NULL: + break; + } + } + scratch->physical_selection.total_values = physical_cursor; + return Status::OK(); +} + +inline Status run_nullable_selection_once(NullableSelectionScratch* scratch, + const std::vector& null_runs, size_t num_values, + size_t num_nulls, + format::parquet::native::FilterMap* filter, + NullableSelectionImplementation implementation) { + if (implementation == NullableSelectionImplementation::LEGACY) { + return run_legacy_nullable_selection(scratch, null_runs, num_values, filter); + } + scratch->output_nulls.clear(); + return format::parquet::native::build_filtered_nullable_selection( + null_runs, num_values, num_nulls, &scratch->output_nulls, filter, 0, + &scratch->physical_selection, &scratch->selected_nulls, &scratch->num_filtered); +} + +inline bool equal_selection(const ParquetSelection& lhs, const ParquetSelection& rhs) { + if (lhs.total_values != rhs.total_values || lhs.selected_values != rhs.selected_values || + lhs.ranges.size() != rhs.ranges.size()) { + return false; + } + for (size_t range = 0; range < lhs.ranges.size(); ++range) { + if (lhs.ranges[range].first != rhs.ranges[range].first || + lhs.ranges[range].count != rhs.ranges[range].count) { + return false; + } + } + return true; +} + +inline void run_nullable_selection_kernel(benchmark::State& state, + const NullableSelectionScenario& scenario) { + using format::parquet::native::FilterMap; + + std::vector filter_data(KERNEL_ROWS, 0); + const auto selected = make_selection_plan(KERNEL_ROWS, scenario.selectivity_percent, + scenario.selection_pattern); + visit_selected_rows(selected, [&](size_t row) { filter_data[row] = 1; }); + FilterMap filter; + auto status = filter.init(filter_data.data(), filter_data.size(), false); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + + NullMap nulls; + nulls.resize_fill(KERNEL_ROWS, 0); + const auto null_plan = + make_selection_plan(KERNEL_ROWS, scenario.null_percent, scenario.null_pattern); + visit_selected_rows(null_plan, [&](size_t row) { nulls[row] = 1; }); + const auto null_runs = build_nullable_runs(nulls); + + NullableSelectionScratch legacy; + NullableSelectionScratch fused; + status = run_nullable_selection_once(&legacy, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, NullableSelectionImplementation::LEGACY); + if (status.ok()) { + status = + run_nullable_selection_once(&fused, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, NullableSelectionImplementation::FUSED); + } + if (!status.ok() || !equal_selection(legacy.physical_selection, fused.physical_selection) || + legacy.output_nulls != fused.output_nulls || + legacy.selected_nulls != fused.selected_nulls || + legacy.num_filtered != fused.num_filtered) { + if (status.ok()) { + state.SkipWithError("nullable selection implementations disagree"); + } else { + state.SkipWithError(status.to_string().c_str()); + } + return; + } + + NullableSelectionScratch scratch; + status = run_nullable_selection_once(&scratch, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, scenario.implementation); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + for (auto _ : state) { + status = run_nullable_selection_once(&scratch, null_runs, KERNEL_ROWS, + null_plan.selected_rows, &filter, + scenario.implementation); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + benchmark::DoNotOptimize(scratch.physical_selection.ranges.data()); + benchmark::DoNotOptimize(scratch.selected_nulls.data()); + benchmark::ClobberMemory(); + } + + state.SetItemsProcessed(static_cast(state.iterations()) * + static_cast(KERNEL_ROWS)); + state.counters["rows"] = static_cast(KERNEL_ROWS); + state.counters["selected_rows"] = static_cast(selected.selected_rows); + state.counters["null_rows"] = static_cast(null_plan.selected_rows); +} + inline NestedSelectionOracle build_nested_selection_oracle( const std::vector& repetition_levels, const std::vector& definition_levels, @@ -493,7 +678,24 @@ inline bool register_kernel_benchmarks() { return true; } +inline bool register_nullable_selection_benchmarks() { + for (const auto& scenario : nullable_selection_scenarios()) { + const std::string name = "ParquetKernel/nullable_selection/sel_" + + std::to_string(scenario.selectivity_percent) + "/null_" + + std::to_string(scenario.null_percent) + "/selection_" + + to_string(scenario.selection_pattern) + "/nulls_" + + to_string(scenario.null_pattern) + "/impl_" + + to_string(scenario.implementation); + benchmark::RegisterBenchmark(name.c_str(), [=](benchmark::State& state) { + run_nullable_selection_kernel(state, scenario); + })->Unit(benchmark::kNanosecond); + } + return true; +} + inline const bool KERNEL_BENCHMARKS_REGISTERED = register_kernel_benchmarks(); +inline const bool NULLABLE_SELECTION_BENCHMARKS_REGISTERED = + register_nullable_selection_benchmarks(); } // namespace detail } // namespace doris::parquet_benchmark diff --git a/be/benchmark/parquet/benchmark_parquet_reader.hpp b/be/benchmark/parquet/benchmark_parquet_reader.hpp index b456244ba0e3ba..4763ffe4c9867e 100644 --- a/be/benchmark/parquet/benchmark_parquet_reader.hpp +++ b/be/benchmark/parquet/benchmark_parquet_reader.hpp @@ -34,6 +34,7 @@ #include #include +#include "common/config.h" #include "core/assert_cast.h" #include "core/block/block.h" #include "core/column/column_nullable.h" @@ -61,6 +62,19 @@ namespace reader_detail { constexpr size_t READER_ROWS = 1UL << 14; constexpr size_t READER_ROW_GROUP_ROWS = 1UL << 12; +constexpr size_t MULTI_COLUMN_OR_ROWS = 1UL << 20; +constexpr size_t MULTI_COLUMN_OR_ROW_GROUP_ROWS = 1UL << 18; + +class ScopedPageIndexConfig { +public: + explicit ScopedPageIndexConfig(bool enabled) : _previous(config::enable_parquet_page_index) { + config::enable_parquet_page_index = enabled; + } + ~ScopedPageIndexConfig() { config::enable_parquet_page_index = _previous; } + +private: + bool _previous; +}; inline void throw_if_error(const Status& status) { if (!status.ok()) { @@ -256,6 +270,8 @@ class Int32LessThanExpr final : public VExpr { _column_id(column_id), _upper_bound(upper_bound) {} + bool is_constant() const override { return false; } + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, size_t count, ColumnPtr& result_column) const override { DORIS_CHECK(block != nullptr); @@ -612,6 +628,151 @@ inline void run_reader(benchmark::State& state, ReaderScenario scenario) { } } +inline std::filesystem::path ensure_multi_column_or_fixture() { + static std::mutex fixture_mutex; + const auto directory = + std::filesystem::temp_directory_path() / "doris_parquet_reader_benchmark"; + const auto path = directory / "v2_multi_column_or_page_index_v2.parquet"; + std::lock_guard guard(fixture_mutex); + if (std::filesystem::exists(path)) { + return path; + } + + arrow::Int32Builder ascending_builder; + arrow::Int32Builder descending_builder; + arrow::Int32Builder payload_builder; + PARQUET_THROW_NOT_OK(ascending_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + PARQUET_THROW_NOT_OK(descending_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + PARQUET_THROW_NOT_OK(payload_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + for (size_t row = 0; row < MULTI_COLUMN_OR_ROWS; ++row) { + const int32_t row_in_group = static_cast(row % MULTI_COLUMN_OR_ROW_GROUP_ROWS); + PARQUET_THROW_NOT_OK(ascending_builder.Append(row_in_group)); + PARQUET_THROW_NOT_OK(descending_builder.Append( + static_cast(MULTI_COLUMN_OR_ROW_GROUP_ROWS - 1) - row_in_group)); + PARQUET_THROW_NOT_OK(payload_builder.Append(static_cast(row))); + } + auto table = arrow::Table::Make( + arrow::schema({arrow::field("ascending", arrow::int32(), true), + arrow::field("descending", arrow::int32(), true), + arrow::field("payload", arrow::int32(), true)}), + {ascending_builder.Finish().ValueOrDie(), descending_builder.Finish().ValueOrDie(), + payload_builder.Finish().ValueOrDie()}); + + std::filesystem::create_directories(directory); + const auto temporary_path = path.string() + ".tmp"; + std::filesystem::remove(temporary_path); + const auto output_result = arrow::io::FileOutputStream::Open(temporary_path); + if (!output_result.ok()) { + throw std::runtime_error(output_result.status().ToString()); + } + const auto output = *output_result; + ::parquet::WriterProperties::Builder properties; + properties.version(::parquet::ParquetVersion::PARQUET_2_6); + properties.data_page_version(::parquet::ParquetDataPageVersion::V2); + properties.compression(::parquet::Compression::UNCOMPRESSED); + properties.disable_dictionary(); + properties.encoding(::parquet::Encoding::PLAIN); + properties.enable_write_page_index(); + properties.write_batch_size(8192); + properties.data_pagesize(64 * 1024); + PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), output, + MULTI_COLUMN_OR_ROW_GROUP_ROWS, + properties.build())); + PARQUET_THROW_NOT_OK(output->Close()); + std::filesystem::rename(temporary_path, path); + return path; +} + +inline std::unique_ptr open_multi_column_or_reader( + const std::filesystem::path& path) { + auto session = std::make_unique(); + auto properties = std::make_shared(); + properties->system_type = TFileType::FILE_LOCAL; + auto description = std::make_unique(); + description->path = path.string(); + description->file_size = static_cast(std::filesystem::file_size(path)); + description->range_start_offset = 0; + description->range_size = -1; + session->reader = std::make_unique(properties, description, + nullptr, nullptr); + throw_if_error(session->reader->init(&session->runtime_state)); + throw_if_error(session->reader->get_schema(&session->schema)); + + session->request = std::make_shared(); + format::FileScanRequestBuilder request_builder(session->request.get()); + std::array predicate_positions {}; + for (int column = 0; column < 2; ++column) { + const auto column_id = format::LocalColumnId(column); + throw_if_error(request_builder.add_predicate_column(column_id)); + session->request->predicate_only_columns.push_back(column_id); + predicate_positions[column] = + static_cast(session->request->local_positions.at(column_id).value()); + } + throw_if_error(request_builder.add_non_predicate_column(format::LocalColumnId(2))); + + TExprNode node; + node.__set_node_type(TExprNodeType::COMPOUND_PRED); + node.__set_opcode(TExprOpcode::COMPOUND_OR); + node.__set_type(std::make_shared()->to_thrift()); + node.__set_num_children(2); + node.__set_is_nullable(false); + auto compound = VCompoundPred::create_shared(node); + constexpr int32_t UPPER_BOUND = static_cast(MULTI_COLUMN_OR_ROW_GROUP_ROWS / 10); + compound->add_child(std::make_shared(predicate_positions[0], UPPER_BOUND)); + compound->add_child(std::make_shared(predicate_positions[1], UPPER_BOUND)); + auto context = VExprContext::create_shared(std::move(compound)); + throw_if_error(context->prepare(&session->runtime_state, RowDescriptor())); + throw_if_error(context->open(&session->runtime_state)); + session->request->conjuncts.push_back(context); + session->opened_conjuncts.push_back(std::move(context)); + throw_if_error(session->reader->open(session->request)); + return session; +} + +inline void run_multi_column_or_reader(benchmark::State& state, bool enable_page_index) { + try { + const auto fixture = ensure_multi_column_or_fixture(); + ScopedPageIndexConfig page_index_config(enable_page_index); + size_t selected_rows = 0; + for (auto _ : state) { + state.PauseTiming(); + auto session = open_multi_column_or_reader(fixture); + state.ResumeTiming(); + const ReaderScenario scenario {.operation = ReaderOperation::PREDICATE_SCAN, + .encoding = Encoding::PLAIN, + .null_percent = 0, + .null_pattern = Pattern::CLUSTERED, + .selectivity_percent = 20, + .projection = Projection::PREDICATE_ONLY, + .schema_width = 3, + .predicate_position = 0}; + selected_rows = scan_reader(session.get(), scenario); + state.PauseTiming(); + throw_if_error(session->reader->close()); + state.ResumeTiming(); + benchmark::ClobberMemory(); + } + constexpr size_t ROW_GROUPS = MULTI_COLUMN_OR_ROWS / MULTI_COLUMN_OR_ROW_GROUP_ROWS; + constexpr size_t EXPECTED_ROWS = ROW_GROUPS * 2 * (MULTI_COLUMN_OR_ROW_GROUP_ROWS / 10); + if (selected_rows != EXPECTED_ROWS) { + state.SkipWithError("multi-column OR benchmark returned unexpected rows"); + return; + } + state.SetItemsProcessed(static_cast(state.iterations() * selected_rows)); + state.counters["raw_rows"] = static_cast(MULTI_COLUMN_OR_ROWS); + state.counters["selected_rows"] = static_cast(selected_rows); + state.counters["fixture_bytes"] = static_cast(std::filesystem::file_size(fixture)); + state.counters["ns/raw_row"] = benchmark::Counter( + static_cast(MULTI_COLUMN_OR_ROWS), + benchmark::Counter::kIsIterationInvariantRate | benchmark::Counter::kInvert); + state.counters["ns/selected_row"] = benchmark::Counter( + static_cast(selected_rows), + benchmark::Counter::kIsIterationInvariantRate | benchmark::Counter::kInvert); + } catch (const std::exception& error) { + state.SkipWithError(error.what()); + } +} + inline bool register_reader_benchmarks() { for (const auto& scenario : reader_scenarios()) { std::string name = "ParquetReader/" + reader_scenario_name(scenario); @@ -619,6 +780,14 @@ inline bool register_reader_benchmarks() { run_reader(state, scenario); })->Unit(benchmark::kNanosecond); } + benchmark::RegisterBenchmark( + "ParquetReader/multi_column_or/page_index_off", + [](benchmark::State& state) { run_multi_column_or_reader(state, false); }) + ->Unit(benchmark::kNanosecond); + benchmark::RegisterBenchmark( + "ParquetReader/multi_column_or/page_index_on", + [](benchmark::State& state) { run_multi_column_or_reader(state, true); }) + ->Unit(benchmark::kNanosecond); return true; } diff --git a/be/benchmark/parquet/parquet_benchmark_scenarios.h b/be/benchmark/parquet/parquet_benchmark_scenarios.h index a9c58c15d8cff1..f7eddf0b11a5aa 100644 --- a/be/benchmark/parquet/parquet_benchmark_scenarios.h +++ b/be/benchmark/parquet/parquet_benchmark_scenarios.h @@ -55,6 +55,7 @@ enum class Kernel { NESTED_SELECTION }; enum class NestedSelectionImplementation { LEGACY, FUSED }; +enum class NullableSelectionImplementation { LEGACY, FUSED }; struct DecoderScenario { Encoding encoding; @@ -89,6 +90,14 @@ struct SelectionScenario { Pattern pattern; }; +struct NullableSelectionScenario { + int selectivity_percent; + int null_percent; + Pattern selection_pattern; + Pattern null_pattern; + NullableSelectionImplementation implementation; +}; + struct SelectionRange { size_t first; size_t count; @@ -177,6 +186,24 @@ inline std::vector selection_scenarios() { return scenarios; } +inline std::vector nullable_selection_scenarios() { + std::vector scenarios; + for (const int selectivity : {1, 10, 50, 90, 99}) { + for (const int null_percent : {0, 1, 10, 50, 90}) { + for (const auto selection_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto null_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto implementation : {NullableSelectionImplementation::LEGACY, + NullableSelectionImplementation::FUSED}) { + scenarios.push_back({selectivity, null_percent, selection_pattern, + null_pattern, implementation}); + } + } + } + } + } + return scenarios; +} + inline std::vector reader_scenarios() { std::vector scenarios; std::set> BeExecVersionManager::_function_change_map {}; std::set BeExecVersionManager::_function_restrict_map; diff --git a/be/src/agent/be_exec_version_manager.h b/be/src/agent/be_exec_version_manager.h index c1a40e35a075e1..4582b84c61d744 100644 --- a/be/src/agent/be_exec_version_manager.h +++ b/be/src/agent/be_exec_version_manager.h @@ -27,6 +27,7 @@ namespace doris { constexpr inline int USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION = 10; constexpr inline int SUPPORT_ICEBERG_MERGE_CARDINALITY_VERSION = 11; +constexpr inline int SUPPORT_ICEBERG_VARIANT_VERSION = 12; class BeExecVersionManager { public: diff --git a/be/src/core/block/block.cpp b/be/src/core/block/block.cpp index 8b92c2e3db9156..f2315aaacff03f 100644 --- a/be/src/core/block/block.cpp +++ b/be/src/core/block/block.cpp @@ -828,6 +828,7 @@ void Block::clear() { data.clear(); } +// Both clear paths must preserve shared children even when a composite column is top-level exclusive. void Block::clear_column_data(int64_t column_size) { SCOPED_SKIP_MEMORY_CHECK(); // data.size() greater than column_size, means here have some @@ -839,7 +840,7 @@ void Block::clear_column_data(int64_t column_size) { } for (auto& d : data) { if (d.column) { - if (d.column->is_exclusive()) { + if (is_recursively_exclusive(*d.column)) { d.column->assert_mutable()->clear(); } else { d.column = d.column->clone_empty(); @@ -854,7 +855,7 @@ void Block::clear_column_data(const std::vector& columns_to_clear) { DCHECK_LT(col, data.size()); auto& column = data[col].column; if (column) { - if (column->is_exclusive()) { + if (is_recursively_exclusive(*column)) { column->assert_mutable()->clear(); } else { column = column->clone_empty(); diff --git a/be/src/core/column/column.cpp b/be/src/core/column/column.cpp index 68048b10053b2b..37a5f37c5b19b7 100644 --- a/be/src/core/column/column.cpp +++ b/be/src/core/column/column.cpp @@ -28,6 +28,28 @@ namespace doris { +namespace { + +bool contains_boolean_value_column(const IColumn& column) { + if (const auto* nullable = check_and_get_column(column)) { + return contains_boolean_value_column(nullable->get_nested_column()); + } + if (check_and_get_column(column)) { + return true; + } + + bool contains_boolean = false; + IColumn::ColumnCallback callback = [&](const IColumn& subcolumn) { + if (!contains_boolean && contains_boolean_value_column(subcolumn)) { + contains_boolean = true; + } + }; + column.for_each_subcolumn(callback); + return contains_boolean; +} + +} // namespace + std::string IColumn::dump_structure() const { std::stringstream res; res << get_name() << "(size = " << size(); @@ -64,6 +86,11 @@ bool IColumn::column_boolean_check() const { if (const auto* col_nullable = check_and_get_column(*this)) { // for column nullable, we need to skip null values check const auto& nested_col = col_nullable->get_nested_column(); + // Do not materialize complex payloads that cannot contain Boolean values; filtering a + // multi-GB nested column solely for debug validation can exhaust query memory. + if (!contains_boolean_value_column(nested_col)) { + return true; + } const auto& null_map = col_nullable->get_null_map_data(); Filter not_null_filter; not_null_filter.reserve(nested_col.size()); diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index 6a74ac091ee5cb..0d5a7fb74bb6df 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -347,7 +347,8 @@ ColumnVariantV2::ColumnVariantV2(const ColumnVariantV2& other) _meta_ids(other._meta_ids), _values(other._values), _typed(other._typed), - _typed_type(other._typed_type) {} + _typed_type(other._typed_type), + _shredded(other._shredded) {} ColumnVariantV2::MutablePtr ColumnVariantV2::create_typed(ColumnPtr column, DataTypePtr scalar_type) { @@ -359,6 +360,15 @@ ColumnVariantV2::MutablePtr ColumnVariantV2::create_typed(ColumnPtr column, return result; } +ColumnVariantV2::MutablePtr ColumnVariantV2::create_shredded( + std::shared_ptr state) { + DORIS_CHECK(state != nullptr) << "shredded ColumnVariantV2 state must not be null"; + auto result = ColumnVariantV2::create(); + result->_shredded = std::move(state); + result->_check_invariants(); + return result; +} + const IColumn& ColumnVariantV2::typed_column() const { DORIS_CHECK(_typed != nullptr) << "typed_column requires ColumnVariantV2 typed state"; return *_typed; @@ -369,7 +379,30 @@ const DataTypePtr& ColumnVariantV2::typed_type() const { return _typed_type; } +std::optional ColumnVariantV2::find_shredded_typed_value( + std::span path) const { + if (!_shredded) { + return std::nullopt; + } + return _shredded->find_typed_value(path); +} + void ColumnVariantV2::ensure_encoded() { + if (_shredded) { + const ColumnVariantV2& materialized = _shredded->materialized_column(); + DORIS_CHECK(!materialized.is_shredded()) + << "shredded state materializer returned another shredded column"; + // The shredded state may cache and share its canonical materialization across readers. + // Detach every mutable buffer before dropping that owner so later COW mutations stay legal. + _metadatas = materialized._metadatas->clone_resized(materialized._metadatas->size()); + _meta_ids = materialized._meta_ids->clone_resized(materialized._meta_ids->size()); + _values = materialized._values->clone_resized(materialized._values->size()); + _typed = materialized._typed == nullptr + ? nullptr + : materialized._typed->clone_resized(materialized._typed->size()); + _typed_type = materialized._typed_type; + _shredded.reset(); + } if (!_typed) { DCHECK(_typed_type == nullptr); return; @@ -394,6 +427,9 @@ void ColumnVariantV2::ensure_encoded() { } std::string ColumnVariantV2::get_name() const { + if (_shredded) { + return "variant_v2(shredded)"; + } if (_typed) { DORIS_CHECK(_typed_type != nullptr); return "variant_v2(typed=" + _typed_type->get_name() + ")"; @@ -403,6 +439,9 @@ std::string ColumnVariantV2::get_name() const { } size_t ColumnVariantV2::size() const { + if (_shredded) { + return _shredded->size(); + } if (_typed) { DCHECK(_typed_type != nullptr); DCHECK(_metadatas->empty()); @@ -417,6 +456,9 @@ size_t ColumnVariantV2::size() const { } size_t ColumnVariantV2::byte_size() const { + if (_shredded) { + return _shredded->byte_size(); + } if (_typed) { DCHECK(_metadatas->empty()); DCHECK(_meta_ids->empty()); @@ -428,6 +470,9 @@ size_t ColumnVariantV2::byte_size() const { } size_t ColumnVariantV2::allocated_bytes() const { + if (_shredded) { + return _shredded->allocated_bytes(); + } if (_typed) { DCHECK(_metadatas->empty()); DCHECK(_meta_ids->empty()); @@ -441,6 +486,9 @@ size_t ColumnVariantV2::allocated_bytes() const { bool ColumnVariantV2::has_enough_capacity(const IColumn& src) const { const auto& source = assert_cast(src); + if (_shredded || source._shredded) { + return false; + } if (static_cast(_typed) != static_cast(source._typed)) { return false; } @@ -460,6 +508,11 @@ bool ColumnVariantV2::structure_equals(const IColumn& rhs) const { } void ColumnVariantV2::sanity_check() const { + if (_shredded) { + _shredded->sanity_check(); + _check_invariants(); + return; + } if (_typed) { _typed->sanity_check(); } else { @@ -487,6 +540,10 @@ void ColumnVariantV2::sanity_check() const { } void ColumnVariantV2::for_each_subcolumn(ColumnCallback callback) const { + if (_shredded) { + _shredded->for_each_subcolumn(callback); + return; + } if (_typed) { callback(*static_cast(_typed)); } else { @@ -497,6 +554,11 @@ void ColumnVariantV2::for_each_subcolumn(ColumnCallback callback) const { } void ColumnVariantV2::mutate_subcolumns() { + if (_shredded) { + // Shredded state is immutable and reference-counted, so keep a partial leaf projection + // intact until an operation explicitly requires canonical bytes. + return; + } if (_typed) { mutate_subcolumn(_typed); } else { @@ -507,6 +569,11 @@ void ColumnVariantV2::mutate_subcolumns() { } void ColumnVariantV2::clear() { + if (_shredded) { + _shredded.reset(); + _check_invariants(); + return; + } if (_typed) { mutate_subcolumn(_typed); _typed->clear(); @@ -528,7 +595,7 @@ void ColumnVariantV2::clear() { // Validate the encoded batch before appending metadata, ids, and values. void ColumnVariantV2::insert_encoded_rows( // NOLINT(readability-function-size) const EncodedDataView& data) { - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } DORIS_CHECK(_typed_type == nullptr) << "encoded state cannot retain a typed data type"; @@ -606,7 +673,7 @@ void ColumnVariantV2::insert_encoded_rows( // NOLINT(readability-function-size) } void ColumnVariantV2::insert_encoded_batch(const VariantBatchBuilder& block) { - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } DORIS_CHECK(_typed_type == nullptr) << "encoded state cannot retain a typed data type"; @@ -639,6 +706,9 @@ void ColumnVariantV2::insert_encoded_batch(const VariantBatchBuilder& block) { } VariantRef ColumnVariantV2::get_value_ref(size_t row) const { + if (_shredded) { + return _shredded->materialized_column().get_value_ref(row); + } DCHECK(!_typed); DCHECK(_typed_type == nullptr); DCHECK_LT(row, size()); @@ -672,7 +742,7 @@ void ColumnVariantV2::insert_many_defaults(size_t length) { return; } - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } @@ -724,6 +794,39 @@ void ColumnVariantV2::insert_range_from( // NOLINT(readability-function-size) return; } + if (!_shredded && !_typed && empty() && _metadatas->empty() && source._shredded) { + // IColumn::cut() inserts into an empty clone. Select the physical tree directly because an + // incomplete leaf projection cannot be reconstructed merely to copy a row range. + _shredded = start == 0 && length == source.size() + ? source._shredded + : source._shredded->select_range(start, length); + _check_invariants(); + return; + } + if (_shredded && source._shredded) { + auto selected_source = start == 0 && length == source.size() + ? source._shredded + : source._shredded->select_range(start, length); + // C++20 libc++ removed shared_ptr::unique(); use_count preserves the same COW invariant + // on every supported toolchain before mutating the format-owned state. + if (_shredded.use_count() != 1) { + _shredded = _shredded->select_range(0, size()); + } + // A partial physical projection has no metadata/value pair to encode. Preserve that + // invariant by merging compatible scanner batches before the canonical fallback below. + if (_shredded->try_append(*selected_source)) { + _check_invariants(); + return; + } + } + if (_shredded) { + ensure_encoded(); + } + if (source._shredded) { + insert_range_from(source._shredded->materialized_column(), start, length); + return; + } + if (_typed && source._typed && exact_typed_identity(_typed_type, source._typed_type)) { mutate_subcolumn(_typed); _typed->insert_range_from(*source._typed, start, length); @@ -808,6 +911,21 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) return; } + if (_shredded) { + ensure_encoded(); + } + if (!_typed && empty() && _metadatas->empty() && source._shredded) { + // Gather into the native shredded representation for the same reason as range selection: + // row selection does not require, and may not have, a complete logical Variant value. + _shredded = source._shredded->select_indices(indices_begin, indices_end); + _check_invariants(); + return; + } + if (source._shredded) { + insert_indices_from(source._shredded->materialized_column(), indices_begin, indices_end); + return; + } + if (_typed && source._typed && exact_typed_identity(_typed_type, source._typed_type)) { mutate_subcolumn(_typed); _typed->insert_indices_from(*source._typed, indices_begin, indices_end); @@ -885,6 +1003,9 @@ void ColumnVariantV2::pop_back(size_t length) { if (length == 0) { return; } + if (_shredded) { + ensure_encoded(); + } if (_typed) { mutate_subcolumn(_typed); _typed->pop_back(length); @@ -1152,6 +1273,9 @@ void ColumnVariantV2::replace_column_null_data(const uint8_t* __restrict null_ma if (std::none_of(null_map, null_map + size(), [](uint8_t value) { return value != 0; })) { return; } + if (_shredded) { + ensure_encoded(); + } // Hash joins serialize the nested value even for a null-safe NULL key. Normalize those hidden // values to the canonical Variant default so build and probe keys compare byte-for-byte. @@ -1176,6 +1300,9 @@ void ColumnVariantV2::replace_column_null_data(const uint8_t* __restrict null_ma ColumnPtr ColumnVariantV2::filter(const Filter& filter, ssize_t result_size_hint) const { column_match_filter_size(size(), filter.size()); + if (_shredded) { + return ColumnVariantV2::create_shredded(_shredded->filter(filter, result_size_hint)); + } if (_typed) { ColumnPtr filtered = _typed->filter(filter, result_size_hint); auto result = ColumnVariantV2::create(); @@ -1201,6 +1328,13 @@ ColumnPtr ColumnVariantV2::filter(const Filter& filter, ssize_t result_size_hint size_t ColumnVariantV2::filter(const Filter& filter) { column_match_filter_size(size(), filter.size()); + if (_shredded) { + // Scanner-side compaction is a row-selection operation, not a request for canonical + // Variant bytes. Keep partial Parquet projections in their physical representation. + _shredded = _shredded->filter(filter, -1); + _check_invariants(); + return size(); + } if (_typed) { ColumnPtr filtered = static_cast(_typed)->filter(filter, -1); const size_t filtered_size = filtered->size(); @@ -1232,6 +1366,10 @@ MutableColumnPtr ColumnVariantV2::permute(const Permutation& permutation, size_t } } + if (_shredded) { + return _shredded->materialized_column().permute(permutation, limit); + } + if (_typed) { MutableColumnPtr permuted = _typed->permute(permutation, result_size); auto result = ColumnVariantV2::create(); @@ -1257,6 +1395,20 @@ MutableColumnPtr ColumnVariantV2::permute(const Permutation& permutation, size_t } MutableColumnPtr ColumnVariantV2::clone_resized(size_t new_size) const { + if (_shredded) { + if (new_size == 0) { + // Empty scanner placeholders carry no rows and therefore need no physical shredded + // state. Avoid forcing a partial leaf projection through full materialization. + return ColumnVariantV2::create(); + } + if (new_size == size()) { + auto result = ColumnVariantV2::create(); + result->_shredded = _shredded; + result->_check_invariants(); + return result; + } + return _shredded->materialized_column().clone_resized(new_size); + } if (_typed) { auto result = ColumnVariantV2::create(); if (new_size <= size()) { @@ -1295,6 +1447,9 @@ MutableColumnPtr ColumnVariantV2::clone_resized(size_t new_size) const { void ColumnVariantV2::resize(size_t new_size) { const size_t old_size = size(); + if (_shredded && new_size != old_size) { + ensure_encoded(); + } if (_typed) { if (new_size == old_size) { return; @@ -1357,6 +1512,7 @@ void ColumnVariantV2::_adopt_state_from(ColumnVariantV2& replacement) { _values = std::move(replacement._values); _typed = std::move(replacement._typed); _typed_type = std::move(replacement._typed_type); + _shredded = std::move(replacement._shredded); _check_invariants(); } @@ -1368,6 +1524,14 @@ void ColumnVariantV2::_detach_metadata_for_write() { } void ColumnVariantV2::_check_invariants() const { + if (_shredded) { + DORIS_CHECK(_typed == nullptr) << "shredded state cannot contain a typed column"; + DORIS_CHECK(_typed_type == nullptr) << "shredded state cannot retain a typed data type"; + DORIS_CHECK(_metadatas->empty()) << "shredded state cannot contain encoded metadata"; + DORIS_CHECK(_meta_ids->empty()) << "shredded state cannot contain encoded metadata ids"; + DORIS_CHECK(_values->empty()) << "shredded state cannot contain encoded values"; + return; + } if (_typed) { DORIS_CHECK(_typed_type != nullptr) << "typed state requires a data type"; const IColumn* typed_column = static_cast(_typed).get(); diff --git a/be/src/core/column/variant_v2/column_variant_v2.h b/be/src/core/column/variant_v2/column_variant_v2.h index e39bef3454719e..abe4df4b8223d4 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.h +++ b/be/src/core/column/variant_v2/column_variant_v2.h @@ -19,6 +19,8 @@ #include #include +#include +#include #include #include @@ -36,9 +38,58 @@ namespace doris { class DataTypeVariantV2SerDe; class VariantBatchBuilder; +class ColumnVariantV2; -// ColumnVariantV2 stores a whole column in exactly one state: encoded Variant bytes or one nullable -// typed scalar column. Mixed operations materialize the typed state as encoded bytes on demand. +struct VariantShreddedPathSegment { + enum class Kind : uint8_t { OBJECT_KEY, ARRAY_INDEX }; + + Kind kind = Kind::OBJECT_KEY; + StringRef key; + int64_t index = 0; +}; + +struct VariantShreddedTypedValue { + // The state owns the same immutable column. Keeping a ColumnPtr here lets expression results + // retain the decoded leaf without copying it or depending on scanner lifetime. + ColumnPtr column; + DataTypePtr type; +}; + +// Format readers keep their native shredded representation behind this interface. Core Variant +// code sees only logical paths and an explicit late-materialization boundary. +class VariantShreddedState { +public: + virtual ~VariantShreddedState() = default; + + virtual size_t size() const = 0; + virtual size_t byte_size() const = 0; + virtual size_t allocated_bytes() const = 0; + virtual void sanity_check() const = 0; + // Shredded columns are immutable and shared. Expose their physical tree only through the + // immutable callback contract. + virtual void for_each_subcolumn(IColumn::ColumnCallback callback) const = 0; + // Row selection must remain in the native shredded representation. A scanner may compact a + // predicate column before every logical Variant value is available for materialization. + virtual std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t result_size_hint) const = 0; + virtual std::shared_ptr select_range(size_t start, + size_t length) const = 0; + virtual std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const = 0; + // Appends another state only when both format-owned physical layouts have identical semantics. + // An incompatible source must leave this state unchanged and return false. + virtual bool try_append(const VariantShreddedState& source) = 0; + virtual std::optional find_typed_value( + std::span path) const = 0; + + // The returned column is cached and owned by this state, so borrowed VariantRef values remain + // valid for the state lifetime. Implementations must not materialize before this is called. + virtual const ColumnVariantV2& materialized_column() const = 0; +}; + +// ColumnVariantV2 stores a whole column in exactly one state: encoded Variant bytes, one nullable +// typed scalar column, or a format-owned shredded tree. Mixed operations materialize typed or +// shredded state as encoded bytes only when canonical row bytes are required. class ColumnVariantV2 final : public COWHelper { public: struct EncodedDataView { @@ -87,10 +138,14 @@ class ColumnVariantV2 final : public COWHelper { // The input must be an exact, non-Const ColumnNullable whose nested column matches the // non-nullable supported scalar type. static MutablePtr create_typed(ColumnPtr column, DataTypePtr scalar_type); + static MutablePtr create_shredded(std::shared_ptr state); bool is_typed() const noexcept { return _typed != nullptr; } + bool is_shredded() const noexcept { return _shredded != nullptr; } const IColumn& typed_column() const; const DataTypePtr& typed_type() const; + std::optional find_shredded_typed_value( + std::span path) const; void ensure_encoded(); ReadView read_view() const; @@ -197,6 +252,10 @@ class ColumnVariantV2 final : public COWHelper { // single type described by _typed_type. IColumn::WrappedPtr _typed; DataTypePtr _typed_type; + + // A non-null state owns the decoded format columns. Encoded and typed storage stay empty until + // an operation explicitly requests canonical Variant bytes. + std::shared_ptr _shredded; }; template diff --git a/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp b/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp index 3ce60aed8dd7c7..5e276fd0279220 100644 --- a/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp @@ -89,6 +89,9 @@ const DataTypePtr& ColumnVariantV2::ReadView::typed_type() const { } ColumnVariantV2::ReadView ColumnVariantV2::read_view() const { + if (_shredded) { + return _shredded->materialized_column().read_view(); + } if (_typed) { DORIS_CHECK(_typed_type != nullptr) << "typed state requires a data type"; return {static_cast(_typed).get(), &_typed_type}; diff --git a/be/src/exec/operator/file_scan_operator.cpp b/be/src/exec/operator/file_scan_operator.cpp index 370f77bed8b1c8..5290e3078fb171 100644 --- a/be/src/exec/operator/file_scan_operator.cpp +++ b/be/src/exec/operator/file_scan_operator.cpp @@ -19,8 +19,14 @@ #include +#include #include +#include "core/assert_cast.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" #include "exec/operator/olap_scan_operator.h" #include "exec/operator/scan_operator.h" #include "exec/scan/file_scanner.h" @@ -31,6 +37,29 @@ #include "storage/tablet/tablet_manager.h" namespace doris { +namespace { + +bool contains_variant_type(const DataTypePtr& input) { + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + +} // namespace PushDownType FileScanLocalState::_should_push_down_binary_predicate( VectorizedFnCall* fn_call, VExprContext* expr_ctx, Field& constant_val, @@ -163,6 +192,20 @@ Status FileScanLocalState::_init_scanners(std::list* scanners) { const bool use_file_scanner_v2 = _should_use_file_scanner_v2(state()->query_options(), is_load, *scan_params); _operator_profile->add_info_string("UseScannerV2", use_file_scanner_v2 ? "true" : "false"); + const auto* output_tuple_desc = state()->desc_tbl().get_tuple_descriptor(_output_tuple_id); + DORIS_CHECK(output_tuple_desc != nullptr); + const bool metadata_only_count = + is_count_star_pushdown() && _split_source->all_ranges_have_table_level_row_count(); + if (!is_load && !use_file_scanner_v2 && !metadata_only_count && + std::ranges::any_of(output_tuple_desc->slots(), [](const SlotDescriptor* slot) { + return contains_variant_type(slot->get_data_type_ptr()); + })) { + // A syntactic COUNT(*) alone is insufficient: every assigned range must prove that the + // legacy scanner will emit metadata counts without decoding a Variant carrier. + return Status::NotSupported( + "External VARIANT columns require FileScannerV2; the legacy file scanner does " + "not support VARIANT"); + } for (int i = 0; i < _max_scanners; ++i) { ScannerSPtr scanner; if (use_file_scanner_v2) { diff --git a/be/src/exec/operator/scan_operator.h b/be/src/exec/operator/scan_operator.h index e11eb506a772ab..cb9effb23882ce 100644 --- a/be/src/exec/operator/scan_operator.h +++ b/be/src/exec/operator/scan_operator.h @@ -77,6 +77,18 @@ class ScanLocalStateBase : public PipelineXLocalState<> { virtual TPushAggOp::type get_push_down_agg_type() = 0; virtual const std::optional>& get_push_down_count_slot_ids() const = 0; + static bool is_count_star_pushdown(TPushAggOp::type agg_type, + const std::optional>& count_slot_ids) { + // An absent argument field is an old plan with unknown semantics. Only an explicitly empty + // argument list proves COUNT(*)/COUNT(1) and permits placeholder slots to be ignored. + return agg_type == TPushAggOp::type::COUNT && count_slot_ids.has_value() && + count_slot_ids->empty(); + } + + bool is_count_star_pushdown() { + return is_count_star_pushdown(get_push_down_agg_type(), get_push_down_count_slot_ids()); + } + // If scan operator is serial operator(like topn), its real parallelism is 1. // Otherwise, its real parallelism is query_parallel_instance_num. // query_parallel_instance_num of olap table is usually equal to session var parallel_pipeline_task_num. diff --git a/be/src/exec/scan/access_path_parser.cpp b/be/src/exec/scan/access_path_parser.cpp index c294f86f10437d..0d0a5b0a547405 100644 --- a/be/src/exec/scan/access_path_parser.cpp +++ b/be/src/exec/scan/access_path_parser.cpp @@ -189,6 +189,20 @@ void insert_access_path(AccessPathNode* root, const std::vector& pa insert_access_path(&root->children[path[path_idx]], path, path_idx + 1); } +void collect_variant_access_paths(const AccessPathNode& node, std::vector* path, + std::vector>* result) { + DORIS_CHECK(path != nullptr && result != nullptr); + for (const auto& [segment, child] : node.children) { + path->push_back(segment); + if (child.project_all || child.children.empty()) { + result->push_back(*path); + } else { + collect_variant_access_paths(child, path, result); + } + path->pop_back(); + } +} + Status build_nested_children_from_access_node(format::ColumnDefinition* column, const DataTypePtr& type, const AccessPathNode& node, const std::string& path, @@ -445,6 +459,19 @@ Status build_nested_children_from_access_node(format::ColumnDefinition* column, return build_map_children_from_access_node( column, assert_cast(*nested_type), node, path, schema_column, prefer_exact_name_match); + case TYPE_VARIANT: { + // A Variant nested below STRUCT/ARRAY/MAP owns paths relative to this terminal. Keeping + // them on the nested ColumnDefinition lets ColumnMapper select the same physical leaves + // as a root Variant without flattening away the surrounding container. + column->variant_access_paths.clear(); + std::vector variant_path; + collect_variant_access_paths(node, &variant_path, &column->variant_access_paths); + std::ranges::sort(column->variant_access_paths); + column->variant_access_paths.erase(std::unique(column->variant_access_paths.begin(), + column->variant_access_paths.end()), + column->variant_access_paths.end()); + return Status::OK(); + } default: return Status::NotSupported("AccessPathParser does not support access path {} for slot {}", path, column->name); @@ -461,6 +488,44 @@ Status AccessPathParser::build_nested_children(format::ColumnDefinition* column, if (is_scanner_materialized_virtual_column(column->name)) { return Status::OK(); } + if (remove_nullable(column->type)->get_primitive_type() == TYPE_VARIANT) { + column->variant_access_paths.clear(); + for (const auto& access_path : access_paths) { + if (access_path.type != TAccessPathType::DATA || + !access_path.__isset.data_access_path) { + return Status::NotSupported( + "AccessPathParser only supports DATA access paths for Variant slot {}", + column->name); + } + const auto& path = access_path.data_access_path.path; + if (path.empty()) { + // Match the generic access-path tree: an empty DATA path denotes the whole slot + // and dominates every narrower Variant path in the same request. + column->variant_access_paths.clear(); + return Status::OK(); + } + int32_t top_level_id = -1; + if (to_lower(path.front()) != to_lower(column->name) && + (!parse_non_negative_int(path.front(), &top_level_id) || + !column->has_identifier_field_id() || + top_level_id != column->get_identifier_field_id())) { + return Status::NotSupported( + "AccessPathParser access path {} does not match Variant slot {}", + access_path_to_string(path), column->name); + } + if (path.size() == 1) { + // A whole-root access covers every subpath and must disable physical leaf pruning. + column->variant_access_paths.clear(); + return Status::OK(); + } + column->variant_access_paths.emplace_back(path.begin() + 1, path.end()); + } + std::ranges::sort(column->variant_access_paths); + column->variant_access_paths.erase(std::unique(column->variant_access_paths.begin(), + column->variant_access_paths.end()), + column->variant_access_paths.end()); + return Status::OK(); + } if (!is_complex_type(remove_nullable(column->type)->get_primitive_type())) { return Status::OK(); } @@ -506,8 +571,36 @@ Status AccessPathParser::build_nested_children(format::ColumnDefinition* column, bool prefer_exact_name_match) { DORIS_CHECK(column != nullptr); DORIS_CHECK(slot_desc != nullptr); - return build_nested_children(column, slot_desc->all_access_paths(), schema_column, + return build_nested_children(column, slot_desc->all_access_paths(), + slot_desc->predicate_access_paths(), schema_column, prefer_exact_name_match); } +Status AccessPathParser::build_nested_children( + format::ColumnDefinition* column, const std::vector& all_access_paths, + const std::vector& predicate_access_paths, + const format::ColumnDefinition* schema_column, bool prefer_exact_name_match) { + DORIS_CHECK(column != nullptr); + auto predicate_column = *column; + RETURN_IF_ERROR(build_nested_children(column, all_access_paths, schema_column, + prefer_exact_name_match)); + column->has_predicate_access_paths = !predicate_access_paths.empty(); + column->predicate_children.clear(); + column->predicate_variant_access_paths.clear(); + if (predicate_access_paths.empty()) { + return Status::OK(); + } + + predicate_column.children.clear(); + predicate_column.variant_access_paths.clear(); + predicate_column.has_predicate_access_paths = false; + predicate_column.predicate_children.clear(); + predicate_column.predicate_variant_access_paths.clear(); + RETURN_IF_ERROR(build_nested_children(&predicate_column, predicate_access_paths, schema_column, + prefer_exact_name_match)); + column->predicate_children = std::move(predicate_column.children); + column->predicate_variant_access_paths = std::move(predicate_column.variant_access_paths); + return Status::OK(); +} + } // namespace doris diff --git a/be/src/exec/scan/access_path_parser.h b/be/src/exec/scan/access_path_parser.h index 0be785a33906e8..650993aca08b27 100644 --- a/be/src/exec/scan/access_path_parser.h +++ b/be/src/exec/scan/access_path_parser.h @@ -38,6 +38,12 @@ class AccessPathParser { const std::vector& access_paths, const format::ColumnDefinition* schema_column, bool prefer_exact_name_match = true); + + static Status build_nested_children( + format::ColumnDefinition* column, + const std::vector& all_access_paths, + const std::vector& predicate_access_paths, + const format::ColumnDefinition* schema_column, bool prefer_exact_name_match = true); }; } // namespace doris diff --git a/be/src/exec/scan/file_scanner_v2.cpp b/be/src/exec/scan/file_scanner_v2.cpp index 052d465daa2005..a0840b7ae50024 100644 --- a/be/src/exec/scan/file_scanner_v2.cpp +++ b/be/src/exec/scan/file_scanner_v2.cpp @@ -36,7 +36,10 @@ #include "core/block/column_with_type_and_name.h" #include "core/column/column.h" #include "core/data_type/data_type.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" #include "core/data_type_serde/data_type_serde.h" #include "core/string_ref.h" #include "exec/common/util.hpp" @@ -79,6 +82,8 @@ namespace { constexpr int kIcebergPositionDeleteContent = 1; constexpr int kIcebergDeletionVectorContent = 3; +std::string table_format_name(const TFileRangeDesc& range); + std::string table_format_name(const TFileRangeDesc& range) { return range.__isset.table_format_params ? range.table_format_params.table_format_type : "NotSet"; @@ -89,6 +94,26 @@ TFileFormatType::type get_range_format_type(const TFileScanRangeParams& params, return range.__isset.format_type ? range.format_type : params.format_type; } +bool contains_variant_type(const DataTypePtr& input) { + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + bool is_supported_table_format(const TFileRangeDesc& range) { const auto table_format = table_format_name(range); if (table_format == "hudi" && range.__isset.table_format_params && @@ -486,6 +511,13 @@ Status FileScannerV2::_filter_output_block(Block* block) { _get_current_format_type()); } +bool FileScannerV2::_can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const { + // A Variant access expression is evaluated above the file reader. Keep each file-local + // shredded schema intact until that projection turns complete and leaf-only states into a + // common logical result column. + return !_has_variant_projection; +} + Status FileScannerV2::_contextualize_output_filter_status(Status status, TFileFormatType::type format_type) { if (!status.ok() && format_type == TFileFormatType::FORMAT_ORC) { @@ -672,8 +704,9 @@ Status FileScannerV2::_create_table_reader_for_format( Status FileScannerV2::_prepare_table_reader_split(const TFileRangeDesc& range, std::map partition_values) { + const auto format_type = get_range_format_type(*_params, range); format::FileFormat current_split_format; - RETURN_IF_ERROR(_to_file_format(get_range_format_type(*_params, range), ¤t_split_format)); + RETURN_IF_ERROR(_to_file_format(format_type, ¤t_split_format)); VExprContextSPtrs conjuncts; RETURN_IF_ERROR(_build_table_conjuncts(&conjuncts)); VExprContextSPtrs partition_prune_conjuncts; @@ -797,6 +830,7 @@ Status FileScannerV2::_build_projected_columns(const format::TableReader& table_ _projected_columns.clear(); _projected_columns.reserve(_params->required_slots.size()); _need_global_rowid_column = false; + _has_variant_projection = false; format::ProjectedColumnBuildContext build_context { .scan_params = _params, .range = &_current_range, @@ -814,6 +848,7 @@ Status FileScannerV2::_build_projected_columns(const format::TableReader& table_ slot_info.slot_id); } auto column = _build_table_column(it->second); + _has_variant_projection = _has_variant_projection || contains_variant_type(column.type); build_context.slot_desc = it->second; if (column.name.starts_with(BeConsts::GLOBAL_ROWID_COL)) { _need_global_rowid_column = true; diff --git a/be/src/exec/scan/file_scanner_v2.h b/be/src/exec/scan/file_scanner_v2.h index 03e5f4d6bbc1a0..3ccc4e075ef209 100644 --- a/be/src/exec/scan/file_scanner_v2.h +++ b/be/src/exec/scan/file_scanner_v2.h @@ -114,6 +114,7 @@ class FileScannerV2 final : public Scanner { protected: Status _get_block_impl(RuntimeState* state, Block* block, bool* eof) override; + bool _can_merge_padding_blocks(const Block& left, const Block& right) const override; Status _filter_output_block(Block* block) override; void _collect_profile_before_close() override; bool _should_update_load_counters() const override; @@ -190,6 +191,7 @@ class FileScannerV2 final : public Scanner { // the reader is format-specific, so it is rebuilt whenever this stops matching the range. std::string _table_reader_format; std::vector _projected_columns; + bool _has_variant_projection = false; // File formats without embedded schema, such as CSV, still need the FE slot descriptors in // file-column order. This mirrors old FileScanner::_file_slot_descs and is passed only to // readers that cannot derive their schema from file metadata. diff --git a/be/src/exec/scan/scanner.cpp b/be/src/exec/scan/scanner.cpp index cbdf5fa7eeec55..e30543b9343613 100644 --- a/be/src/exec/scan/scanner.cpp +++ b/be/src/exec/scan/scanner.cpp @@ -88,6 +88,15 @@ Status Scanner::get_block_after_projects(RuntimeState* state, Block* block, bool auto& row_descriptor = _local_state->_parent->row_descriptor(); if (_output_row_descriptor) { _origin_block.clear_column_data(row_descriptor.num_materialized_slots()); + if (!_can_merge_padding_blocks(_padding_block, _origin_block)) { + DORIS_CHECK(_padding_block.empty()) + << "padding policy must remain stable for one scanner"; + // Some physical columns carry file-local state that an upper projection must consume + // before the next split is read. Padding those blocks first would make correctness + // depend on whether two file tails happen to share one output batch. + RETURN_IF_ERROR(get_block(state, &_origin_block, eos)); + return _do_projections(&_origin_block, block); + } const auto min_batch_size = std::max(state->batch_size() / 2, 1); const auto block_max_bytes = state->preferred_block_size_bytes(); while (_padding_block.rows() < min_batch_size && _padding_block.bytes() < block_max_bytes && diff --git a/be/src/exec/scan/scanner.h b/be/src/exec/scan/scanner.h index 1bf4cdb5bb2ea1..75e583ecdfce61 100644 --- a/be/src/exec/scan/scanner.h +++ b/be/src/exec/scan/scanner.h @@ -123,6 +123,10 @@ class Scanner { // Subclass should implement this to return data. virtual Status _get_block_impl(RuntimeState* state, Block* block, bool* eof) = 0; + virtual bool _can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const { + return true; + } + Status _merge_padding_block() { if (_padding_block.empty()) { _padding_block.swap(_origin_block); diff --git a/be/src/exec/scan/split_source_connector.h b/be/src/exec/scan/split_source_connector.h index 5926baff303cbf..f13190cf0641c2 100644 --- a/be/src/exec/scan/split_source_connector.h +++ b/be/src/exec/scan/split_source_connector.h @@ -17,6 +17,8 @@ #pragma once +#include + #include "common/config.h" #include "core/custom_allocator.h" #include "runtime/runtime_state.h" @@ -45,6 +47,8 @@ class SplitSourceConnector { virtual TFileScanRangeParams* get_params() = 0; + virtual bool all_ranges_have_table_level_row_count() const { return false; } + protected: template , typename V2 = std::vector> requires(std::is_same_v, @@ -125,6 +129,19 @@ class LocalSplitSourceConnector : public SplitSourceConnector { throw Exception( Status::FatalError("Unreachable, params is got by file_scan_range_params_map")); } + + bool all_ranges_have_table_level_row_count() const override { + // Every assigned range must carry a proven count; one fallback range would still require + // decoding the projected carrier through the selected scanner. + return !_scan_ranges.empty() && std::ranges::all_of(_scan_ranges, [](const auto& params) { + const auto& ranges = params.scan_range.ext_scan_range.file_scan_range.ranges; + return !ranges.empty() && std::ranges::all_of(ranges, [](const auto& range) { + return range.__isset.table_format_params && + range.table_format_params.__isset.table_level_row_count && + range.table_format_params.table_level_row_count >= 0; + }); + }); + } }; /** diff --git a/be/src/exec/sink/viceberg_merge_sink.cpp b/be/src/exec/sink/viceberg_merge_sink.cpp index 5ff5a0a1f28150..0b1e77ee1a47be 100644 --- a/be/src/exec/sink/viceberg_merge_sink.cpp +++ b/be/src/exec/sink/viceberg_merge_sink.cpp @@ -55,13 +55,15 @@ VIcebergMergeSink::~VIcebergMergeSink() = default; Status VIcebergMergeSink::init_properties(ObjectPool* pool, const RowDescriptor& row_desc) { RETURN_IF_ERROR(_build_inner_sinks()); - _table_writer = std::make_unique(_table_sink, _table_output_expr_ctxs, - nullptr, nullptr); - _table_writer->defer_file_cleanup_until_outer_close(); + if (_writes_data_files) { + _table_writer = std::make_unique(_table_sink, _table_output_expr_ctxs, + nullptr, nullptr); + _table_writer->defer_file_cleanup_until_outer_close(); + RETURN_IF_ERROR(_table_writer->init_properties(pool, row_desc)); + } _delete_writer = std::make_unique(_delete_sink, _delete_output_expr_ctxs, nullptr, nullptr); _delete_writer->defer_file_cleanup_until_outer_close(); - RETURN_IF_ERROR(_table_writer->init_properties(pool, row_desc)); RETURN_IF_ERROR(_delete_writer->init_properties(pool)); return Status::OK(); } @@ -69,6 +71,14 @@ Status VIcebergMergeSink::init_properties(ObjectPool* pool, const RowDescriptor& Status VIcebergMergeSink::open(RuntimeState* state, RuntimeProfile* profile) { _state = state; + if (!_writes_data_files && state->be_exec_version() < SUPPORT_ICEBERG_VARIANT_VERSION) { + // The query-wide version keeps delete-only writer omission all-or-nothing; an older BE + // would ignore writes_data_files and parse the unsupported Variant data-writer schema. + return Status::NotSupported( + "Delete-only Iceberg MERGE requires backend execution version {}", + SUPPORT_ICEBERG_VARIANT_VERSION); + } + _written_rows_counter = ADD_COUNTER(profile, "RowsWritten", TUnit::UNIT); _insert_rows_counter = ADD_COUNTER(profile, "InsertRows", TUnit::UNIT); _delete_rows_counter = ADD_COUNTER(profile, "DeleteRows", TUnit::UNIT); @@ -88,10 +98,13 @@ Status VIcebergMergeSink::open(RuntimeState* state, RuntimeProfile* profile) { RETURN_IF_ERROR(_prepare_output_layout()); - RuntimeProfile* table_profile = profile->create_child("IcebergMergeTableWriter", true, true); RuntimeProfile* delete_profile = profile->create_child("IcebergMergeDeleteWriter", true, true); - RETURN_IF_ERROR(_table_writer->open(state, table_profile)); + if (_table_writer) { + RuntimeProfile* table_profile = + profile->create_child("IcebergMergeTableWriter", true, true); + RETURN_IF_ERROR(_table_writer->open(state, table_profile)); + } RETURN_IF_ERROR(_delete_writer->open(state, delete_profile)); return Status::OK(); @@ -153,6 +166,13 @@ Status VIcebergMergeSink::write(RuntimeState* state, Block& block) { _delete_row_count += delete_rows; _insert_row_count += insert_rows; + // A delete-only plan deliberately omits the data writer so Variant target schemas never enter + // the unsupported Iceberg data-write path. Reject a mismatched FE plan before dereferencing it. + if (has_insert && !_writes_data_files) { + return Status::InternalError( + "Iceberg delete-only merge sink received a data insert operation"); + } + bool skip_io = false; #ifdef BE_TEST skip_io = _skip_io; @@ -338,6 +358,8 @@ Status VIcebergMergeSink::_build_inner_sinks() { } const auto& merge_sink = _t_sink.iceberg_merge_sink; + // An old FE cannot produce delete-only plans, so an unset flag retains its data-writer path. + _writes_data_files = !merge_sink.__isset.writes_data_files || merge_sink.writes_data_files; // Missing means an old FE plan, which predates SQL MERGE cardinality validation. _require_merge_cardinality_check = merge_sink.__isset.require_merge_cardinality_check && merge_sink.require_merge_cardinality_check; diff --git a/be/src/exec/sink/viceberg_merge_sink.h b/be/src/exec/sink/viceberg_merge_sink.h index f3733a3318230e..88e9ef89121b74 100644 --- a/be/src/exec/sink/viceberg_merge_sink.h +++ b/be/src/exec/sink/viceberg_merge_sink.h @@ -78,6 +78,7 @@ class VIcebergMergeSink final : public AsyncResultWriter { std::vector _data_column_indices; std::map _matched_row_positions; size_t _matched_row_id_state_size = sizeof(std::map); + bool _writes_data_files = true; bool _require_merge_cardinality_check = false; VExprContextSPtrs _table_output_expr_ctxs; diff --git a/be/src/exprs/function/function_variant_element_v2.cpp b/be/src/exprs/function/function_variant_element_v2.cpp index 644a033f4af356..90863fe86c5c72 100644 --- a/be/src/exprs/function/function_variant_element_v2.cpp +++ b/be/src/exprs/function/function_variant_element_v2.cpp @@ -131,6 +131,42 @@ Status extract_encoded_variant_element(const ColumnVariantV2& source, const ResolvedVariantElementV2Path& path, std::span outer_nulls, ColumnPtr* output); +std::optional extract_shredded_typed_variant_element( + const ColumnVariantV2& source, const ResolvedVariantElementV2Path& path, + std::span outer_nulls) { + DorisVector shredded_path; + shredded_path.reserve(path.size()); + for (size_t position = 0; position < path.size(); ++position) { + VariantShreddedPathSegment segment; + if (path.kind_at(position) == VariantElementV2PathSegment::Kind::OBJECT_KEY) { + segment.kind = VariantShreddedPathSegment::Kind::OBJECT_KEY; + segment.key = path.object_key_at(position); + } else { + segment.kind = VariantShreddedPathSegment::Kind::ARRAY_INDEX; + segment.index = path.array_index_at(position); + } + shredded_path.push_back(segment); + } + + auto match = source.find_shredded_typed_value(shredded_path); + if (!match.has_value()) { + return std::nullopt; + } + const auto& leaf = assert_cast(*match->column); + auto nulls = leaf.get_null_map_column().clone_resized(source.size()); + auto& null_data = assert_cast(*nulls).get_data(); + for (size_t row = 0; row < source.size(); ++row) { + null_data[row] = + static_cast(null_data[row] != 0 || is_outer_null(outer_nulls, row)); + } + + // The typed ColumnVariantV2 retains the exact decoded Parquet leaf. Only the SQL result null + // map is produced here, so predicates and casts can consume the leaf without reconstructing + // canonical Variant rows. + auto values = ColumnVariantV2::create_typed(match->column, match->type); + return ColumnNullable::create(std::move(values), std::move(nulls)); +} + Status make_all_null_variant_element_result(size_t rows, ColumnPtr* output); } // namespace @@ -217,7 +253,14 @@ Status extract_variant_element_v2(const ColumnVariantV2& source, ColumnPtr candidate; try { - if (!source.is_typed()) { + if (source.is_shredded()) { + if (auto typed = extract_shredded_typed_variant_element(source, path, outer_nulls)) { + candidate = std::move(*typed); + } else { + RETURN_IF_ERROR( + extract_encoded_variant_element(source, path, outer_nulls, &candidate)); + } + } else if (!source.is_typed()) { RETURN_IF_ERROR(extract_encoded_variant_element(source, path, outer_nulls, &candidate)); } else { // A typed Variant is one scalar root value per row. String payloads are strings, not diff --git a/be/src/format_v2/column_data.h b/be/src/format_v2/column_data.h index e122b48555113a..40f360e29d01f2 100644 --- a/be/src/format_v2/column_data.h +++ b/be/src/format_v2/column_data.h @@ -261,6 +261,17 @@ struct ColumnDefinition { // Full table-schema identity subtree before access-path pruning. ID-less physical complex // wrappers must be discovered from this view without adding unrequested children to output. std::vector identity_children {}; + // Logical object-key paths requested from a Variant column. An empty collection means the + // whole Variant is required; non-empty paths may be resolved to format-specific shredded + // physical children after the per-file schema is known. + std::vector> variant_access_paths {}; + // Predicate access paths are kept separately from the final union projection. File Scanner V2 + // can lower this smaller semantic tree to an eager predicate projection while deferring the + // final children until rows survive. The flag distinguishes no predicate metadata from a + // whole-root predicate, whose child/path collections are intentionally empty. + bool has_predicate_access_paths = false; + std::vector predicate_children {}; + std::vector> predicate_variant_access_paths {}; // Expression used to materialize missing/default/generated values when the column is not read // directly from the file. VExprContextSPtr default_expr = nullptr; @@ -371,6 +382,19 @@ struct LocalColumnIndex { std::string debug_string() const; }; +inline bool same_local_column_index(const LocalColumnIndex& lhs, const LocalColumnIndex& rhs) { + if (lhs.index != rhs.index || lhs.project_all_children != rhs.project_all_children || + lhs.children.size() != rhs.children.size()) { + return false; + } + for (size_t i = 0; i < lhs.children.size(); ++i) { + if (!same_local_column_index(lhs.children[i], rhs.children[i])) { + return false; + } + } + return true; +} + inline bool is_full_projection(const LocalColumnIndex* projection) { return projection == nullptr || projection->project_all_children; } diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp index 6da3b9b9cd6198..5c37bc873d1bb0 100644 --- a/be/src/format_v2/column_mapper.cpp +++ b/be/src/format_v2/column_mapper.cpp @@ -18,6 +18,7 @@ #include "format_v2/column_mapper.h" #include +#include #include #include #include @@ -1278,6 +1279,12 @@ static bool needs_projected_file_type_rebuild(const ColumnMapping& mapping) { remove_nullable(mapping.table_type)->get_primitive_type()) { return true; } + if (remove_nullable(mapping.file_type)->get_primitive_type() == TYPE_STRUCT && + mapping.child_mappings.size() != mapping.original_file_children.size()) { + // A predicate access path keeps the parent Struct type but intentionally carries only the + // referenced child descriptors; type equality alone must not restore the pruned siblings. + return true; + } if (!mapping.table_type->equals(*mapping.file_type)) { return true; } @@ -1646,17 +1653,24 @@ static bool has_projected_file_children(const ColumnMapping& mapping) { return false; } -static bool needs_nested_file_projection(const ColumnMapping& mapping) { +static bool needs_nested_file_projection(const ColumnMapping& mapping, + bool include_variant_access_paths = false) { if (has_projected_file_children(mapping)) { // Return True if the projected child column is missing / re-ordered return true; } - return std::ranges::any_of(mapping.child_mappings, [](const ColumnMapping& child_mapping) { - return needs_nested_file_projection(child_mapping); - }); + if (include_variant_access_paths && !mapping.variant_access_paths.empty()) { + return true; + } + return std::ranges::any_of( + mapping.child_mappings, [include_variant_access_paths](const ColumnMapping& child) { + return needs_nested_file_projection(child, include_variant_access_paths); + }); } -static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection); +static bool build_variant_projection(const ColumnMapping& mapping, LocalColumnIndex* projection); +static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection, + bool enable_variant_leaf_projection = false); // Build the projected file children/type according to the pruned complex projection. For example, // if we have a struct column `s` with children `id` and `name`, and the projection only keeps @@ -1694,11 +1708,15 @@ static Status rebuild_projected_file_children_and_type( // projected output shape; file readers still read full keys to construct ColumnMap offsets and keep // key semantics unchanged. If a caller tries to project only/prune the key child, the common schema // projection helper rejects it. -static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection) { +static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection, + bool enable_variant_leaf_projection) { if (projection == nullptr) { return Status::InvalidArgument("projection is null"); } DORIS_CHECK(mapping.file_local_id.has_value()); + if (enable_variant_leaf_projection && build_variant_projection(mapping, projection)) { + return Status::OK(); + } *projection = LocalColumnIndex::local(*mapping.file_local_id); projection->project_all_children = mapping.child_mappings.empty(); projection->children.clear(); @@ -1712,7 +1730,8 @@ static Status build_complex_projection(const ColumnMapping& mapping, LocalColumn } for (const auto* child_mapping : present_children) { LocalColumnIndex child_projection; - RETURN_IF_ERROR(build_complex_projection(*child_mapping, &child_projection)); + RETURN_IF_ERROR(build_complex_projection(*child_mapping, &child_projection, + enable_variant_leaf_projection)); projection->children.push_back(std::move(child_projection)); } if (!projection->project_all_children && projection->children.empty()) { @@ -1722,8 +1741,6 @@ static Status build_complex_projection(const ColumnMapping& mapping, LocalColumn return Status::OK(); } -using FilterProjectionMap = std::map; - // Update the mapping's file type according to the projection, and determine whether the projection // is trivial (i.e. the projected file type is the same as the table type, so no need to // rematerialize the complex value back to table layout after reading from file). @@ -1748,50 +1765,148 @@ static Status apply_projection_to_mapping_file_type(const LocalColumnIndex& proj return Status::OK(); } -static Status merge_filter_projection(const FilterProjectionMap* filter_projections, - LocalColumnIndex* projection) { - DORIS_CHECK(projection != nullptr); - if (filter_projections == nullptr) { - return Status::OK(); +static const ColumnDefinition* find_file_child_by_name( + const std::vector& children, std::string_view name) { + const auto child_it = std::ranges::find_if( + children, [name](const ColumnDefinition& child) { return child.name == name; }); + return child_it == children.end() ? nullptr : &*child_it; +} + +static bool variant_leaf_type_preserves_physical_identity(const ColumnDefinition& leaf) { + if (!leaf.children.empty() || leaf.type == nullptr) { + return false; } - const auto filter_projection_it = filter_projections->find(projection->column_id()); - if (filter_projection_it == filter_projections->end()) { - return Status::OK(); + // ColumnDefinition does not transport Parquet's raw-binary/UUID and timestamp-unit tags. + // Limit direct leaves to identities fully described by the Doris scalar type; every ambiguous + // identity must retain the complete wrapper so reconstruction can inspect its physical schema. + switch (remove_nullable(leaf.type)->get_primitive_type()) { + case TYPE_BOOLEAN: + case TYPE_TINYINT: + case TYPE_SMALLINT: + case TYPE_INT: + case TYPE_BIGINT: + case TYPE_FLOAT: + case TYPE_DOUBLE: + case TYPE_DECIMAL128I: + case TYPE_DATEV2: + return true; + default: + return false; } - // Merge predicate-only nested paths into the root projection that is about to be scanned. - // Example: `SELECT s.a WHERE s.b > 1` first builds the output projection `s -> a` from - // ColumnMapping, while build_nested_struct_filter_projection_map() records `s -> b`. This merge - // produces one file scan projection `s -> a,b`. - RETURN_IF_ERROR(merge_local_column_index(projection, filter_projection_it->second)); - return Status::OK(); } -static bool table_root_is_map(const ColumnMapping& mapping) { - if (mapping.table_type == nullptr) { +static bool build_variant_leaf_path_projection(const ColumnMapping& mapping, + const std::vector& path, + LocalColumnIndex* root_projection) { + DORIS_CHECK(root_projection != nullptr); + const auto is_numeric_selector = [](std::string_view value) { + if (value.empty()) { + return false; + } + const size_t digits_begin = value.front() == '+' || value.front() == '-' ? 1 : 0; + return digits_begin < value.size() && + std::ranges::all_of(value.substr(digits_begin), + [](unsigned char c) { return std::isdigit(c); }); + }; + if (path.size() != 1 || path[0].empty() || path[0] == "NULL" || + path[0].find('.') != std::string::npos || is_numeric_selector(path[0]) || + !mapping.file_local_id.has_value()) { + // Thrift currently carries access paths as strings without segment-kind or escaping + // metadata. Signed numeric tokens are therefore also ambiguous between an array selector + // and an object key, so only a single unambiguous key can be mapped losslessly to a leaf. + return false; + } + *root_projection = LocalColumnIndex::partial_local(*mapping.file_local_id); + const auto* root_typed = find_file_child_by_name(mapping.original_file_children, "typed_value"); + if (root_typed == nullptr || root_typed->children.empty() || root_typed->type == nullptr || + remove_nullable(root_typed->type)->get_primitive_type() != TYPE_STRUCT) { + return false; + } + root_projection->children.push_back( + LocalColumnIndex::partial_local(root_typed->file_local_id())); + auto* current_projection = &root_projection->children.back(); + const auto* typed_children = &root_typed->children; + for (size_t position = 0; position < path.size(); ++position) { + const auto* wrapper = find_file_child_by_name(*typed_children, path[position]); + if (wrapper == nullptr) { + return false; + } + current_projection->children.push_back( + LocalColumnIndex::partial_local(wrapper->file_local_id())); + current_projection = ¤t_projection->children.back(); + const auto* typed = find_file_child_by_name(wrapper->children, "typed_value"); + if (typed == nullptr) { + return false; + } + auto typed_projection = LocalColumnIndex::partial_local(typed->file_local_id()); + const bool leaf = position + 1 == path.size(); + if (leaf) { + // Only primitive typed values can be returned as a direct vector. Complex shredded + // values still need their wrapper shape and therefore keep the full Variant fallback. + if (!variant_leaf_type_preserves_physical_identity(*typed)) { + return false; + } + typed_projection.project_all_children = true; + } + current_projection->children.push_back(std::move(typed_projection)); + current_projection = ¤t_projection->children.back(); + typed_children = &typed->children; + } + return true; +} + +static bool build_variant_projection(const ColumnMapping& mapping, LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); + if (mapping.table_type == nullptr || mapping.variant_access_paths.empty() || + remove_nullable(mapping.table_type)->get_primitive_type() != TYPE_VARIANT) { return false; } - return remove_nullable(mapping.table_type)->get_primitive_type() == TYPE_MAP; + std::optional merged; + for (const auto& path : mapping.variant_access_paths) { + LocalColumnIndex path_projection; + if (!build_variant_leaf_path_projection(mapping, path, &path_projection)) { + return false; + } + if (!merged.has_value()) { + merged = std::move(path_projection); + } else if (!merge_local_column_index(&*merged, path_projection).ok()) { + return false; + } + } + if (!merged.has_value()) { + return false; + } + *projection = std::move(*merged); + return true; } -static Status add_scan_column(FileScanRequest* file_request, ColumnMapping* mapping, - bool is_predicate_column, bool force_full_complex_scan_projection, - const FilterProjectionMap* filter_projections = nullptr) { +static Status build_scan_projection(ColumnMapping* mapping, bool force_full_complex_scan_projection, + bool enable_variant_leaf_projection, + LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); const auto file_column_id = LocalColumnId(mapping->file_local_id.value()); - LocalColumnIndex projection = LocalColumnIndex::top_level(file_column_id); + *projection = LocalColumnIndex::top_level(file_column_id); // Columnar readers can turn a complex mapping into a nested file projection, but // row-oriented readers must scan the full top-level complex field because all children are // encoded in the same text cell. - if (!force_full_complex_scan_projection && needs_nested_file_projection(*mapping)) { - RETURN_IF_ERROR(build_complex_projection(*mapping, &projection)); - } - if (is_predicate_column && !force_full_complex_scan_projection) { - DCHECK(filter_projections != nullptr); - // If a projected complex root is also used by a predicate, rebuild the predicate scan - // projection from the output mapping before merging predicate-only children. For - // `SELECT s.a WHERE s.b > 1`, build_complex_projection() produces `s -> a` and - // merge_filter_projection() adds `s -> b`, so the predicate column reads both children. - RETURN_IF_ERROR(merge_filter_projection(filter_projections, &projection)); + if (enable_variant_leaf_projection && !force_full_complex_scan_projection && + build_variant_projection(*mapping, projection)) { + // The per-file Parquet reader will validate residual-value statistics before honoring this + // physical leaf projection; unsafe files atomically fall back to the complete Variant. + } else if (!force_full_complex_scan_projection && + needs_nested_file_projection(*mapping, enable_variant_leaf_projection)) { + RETURN_IF_ERROR( + build_complex_projection(*mapping, projection, enable_variant_leaf_projection)); } + return Status::OK(); +} + +static Status add_scan_column(FileScanRequest* file_request, ColumnMapping* mapping, + bool is_predicate_column, bool force_full_complex_scan_projection, + bool enable_variant_leaf_projection) { + LocalColumnIndex projection; + RETURN_IF_ERROR(build_scan_projection(mapping, force_full_complex_scan_projection, + enable_variant_leaf_projection, &projection)); FileScanRequestBuilder builder(file_request); if (is_predicate_column) { return builder.add_predicate_column(std::move(projection)); @@ -1812,18 +1927,21 @@ static const LocalColumnIndex* find_scan_projection( // mapping.file_type/projected_file_children from the original file schema to the exact shape that // FileReader will return. // -// Example: for `SELECT s.a WHERE s.b > 1`, add_scan_column() keeps only one predicate scan -// projection `s -> a,b`. Applying that projection changes the mapping's file type from the full -// file struct `s` to the projected file struct `s`, so later filter rewrite and -// TableReader final materialization use the same column shape as the file-local block. +// Applying the selected projection changes a mapping's file type to the exact nested shape exposed +// by FileReader, so later filter rewrite and TableReader materialization agree with the file block. static Status apply_scan_projection_to_mapping_file_type(const FileScanRequest& file_request, - ColumnMapping* mapping) { + ColumnMapping* mapping, + bool predicate_mapping = false) { DORIS_CHECK(mapping != nullptr); DORIS_CHECK(mapping->file_local_id.has_value()); const auto file_column_id = LocalColumnId(*mapping->file_local_id); - // Predicate columns are the actual scan projection when a column is used by row-level filters: - // add_scan_column() removes the duplicate non-predicate projection in that case. - const auto* projection = find_scan_projection(file_request.predicate_columns, file_column_id); + const LocalColumnIndex* projection = nullptr; + if (!predicate_mapping && file_request.has_deferred_non_predicate_column(file_column_id)) { + projection = find_scan_projection(file_request.non_predicate_columns, file_column_id); + } + if (projection == nullptr) { + projection = find_scan_projection(file_request.predicate_columns, file_column_id); + } if (projection == nullptr) { projection = find_scan_projection(file_request.non_predicate_columns, file_column_id); } @@ -1831,76 +1949,6 @@ static Status apply_scan_projection_to_mapping_file_type(const FileScanRequest& return apply_projection_to_mapping_file_type(*projection, mapping); } -// Build extra scan projections required only by row-level filters on nested struct children. -// -// Example: for `SELECT s.a FROM t WHERE s.b.c > 1`, the output projection may only contain `s.a`, -// but the file reader must also read `s.b.c` to evaluate the predicate. This function collects the -// table-side filter path, resolves it through ColumnMapping first, and records the corresponding -// file-side projection in filter_projections. This keeps renamed fields consistent between the scan -// projection and row-level conjunct rewrite. Example: -// table filter path: s -> renamed_b -> c -// old file path: s -> b -> c -// recorded path: s -> b -> c -// When add_scan_column() adds the same root as a predicate column, it rebuilds that root from the -// output mapping, merges this filter-only projection into it, and removes the duplicate -// non-predicate root entry. -static Status build_nested_struct_filter_projection_map( - const std::vector& table_filters, const std::vector& mappings, - FilterProjectionMap* filter_projections) { - DORIS_CHECK(filter_projections != nullptr); - filter_projections->clear(); - for (const auto& table_filter : table_filters) { - if (table_filter.conjunct == nullptr) { - continue; - } - // Collect all nested struct paths in the table filter. For example, for - // `s.id > 5 AND element_at(s, 'renamed_name') = 'abc'`, collect the table paths - // `s -> id` and `s -> renamed_name`, then resolve each one to its file-side projection. - std::vector paths; - collect_nested_struct_paths(table_filter.conjunct->root(), &paths); - for (const auto& path : paths) { - auto mapping_it = std::ranges::find_if(mappings, [&](const ColumnMapping& mapping) { - return mapping.global_index == path.root_global_index; - }); - if (mapping_it == mappings.end() || !mapping_it->file_local_id.has_value() || - path.selectors.empty()) { - continue; - } - - ResolvedNestedStructPath resolved; - LocalColumnIndex root_projection; - if (!resolve_nested_struct_path_for_file(path, mappings, &resolved)) { - if (!table_root_is_map(*mapping_it)) { - continue; - } - // Direct map value filters such as `m.value.a > 1` need the value leaf for row - // evaluation even when the query only projects another value child. This is only a - // scan projection fallback; complex map/array expressions are still not rewritten - // into file-local conjuncts. - LocalColumnIndex child_projection; - RETURN_IF_ERROR(build_file_child_projection_from_schema( - mapping_it->original_file_children, path.selectors, &child_projection)); - if (child_projection.local_id() < 0) { - continue; - } - root_projection = LocalColumnIndex::partial_local(*mapping_it->file_local_id); - root_projection.children.push_back(std::move(child_projection)); - } else { - root_projection = std::move(resolved.file_projection); - } - auto filter_projection_it = filter_projections->find(root_projection.column_id()); - if (filter_projection_it == filter_projections->end()) { - filter_projections->emplace(root_projection.column_id(), - std::move(root_projection)); - continue; - } - RETURN_IF_ERROR( - merge_local_column_index(&filter_projection_it->second, root_projection)); - } - } - return Status::OK(); -} - static void rebuild_projection(ColumnMapping* mapping, LocalIndex block_position) { DORIS_CHECK(mapping->file_local_id.has_value()); if (mapping->is_trivial || needs_complex_rematerialize(*mapping)) { @@ -1993,6 +2041,7 @@ Status TableColumnMapper::_create_mapping_for_column(const ColumnDefinition& tab mapping->global_index = global_index; mapping->table_column_name = table_column.name; mapping->table_type = table_column.type; + mapping->variant_access_paths = table_column.variant_access_paths; // Row-lineage names are Iceberg metadata contracts, not reserved names in generic Hive, // Hudi, or Paimon schemas. Only the Iceberg reader may opt into virtual synthesis. const auto row_lineage_type = @@ -2117,6 +2166,19 @@ Status TableColumnMapper::create_mapping(const std::vector& pr RETURN_IF_ERROR(_create_mapping_for_column(projected_columns[column_idx], GlobalIndex(column_idx), &mapping)); _mappings.push_back(std::move(mapping)); + if (enable_independent_predicate_projection() && + projected_columns[column_idx].has_predicate_access_paths) { + auto predicate_column = projected_columns[column_idx]; + predicate_column.children = predicate_column.predicate_children; + predicate_column.variant_access_paths = predicate_column.predicate_variant_access_paths; + predicate_column.has_predicate_access_paths = false; + predicate_column.predicate_children.clear(); + predicate_column.predicate_variant_access_paths.clear(); + ColumnMapping predicate_mapping; + RETURN_IF_ERROR(_create_mapping_for_column(predicate_column, GlobalIndex(column_idx), + &predicate_mapping)); + _predicate_mappings.push_back(std::move(predicate_mapping)); + } } return Status::OK(); } @@ -2124,7 +2186,13 @@ Status TableColumnMapper::create_mapping(const std::vector& pr std::vector TableColumnMapper::_filter_visible_mappings() const { std::vector mappings; mappings.reserve(_mappings.size() + _hidden_mappings.size()); - mappings.insert(mappings.end(), _mappings.begin(), _mappings.end()); + for (const auto& mapping : _mappings) { + const auto predicate_it = std::ranges::find_if( + _predicate_mappings, [&](const ColumnMapping& predicate_mapping) { + return predicate_mapping.global_index == mapping.global_index; + }); + mappings.push_back(predicate_it == _predicate_mappings.end() ? mapping : *predicate_it); + } mappings.insert(mappings.end(), _hidden_mappings.begin(), _hidden_mappings.end()); return mappings; } @@ -2153,7 +2221,8 @@ Status TableColumnMapper::create_scan_request( const std::vector& table_filters, const std::vector& projected_columns, FileScanRequest* file_request, RuntimeState* runtime_state, - const std::map* fixed_local_positions) { + const std::map* fixed_local_positions, + const std::map* fixed_non_predicate_positions) { // FileReader evaluates expressions against a file-local block. This mapper owns the // table-column to file-column conversion, so it also owns the file-local block positions. file_request->predicate_columns.clear(); @@ -2166,7 +2235,14 @@ Status TableColumnMapper::create_scan_request( // continues to address the same physical column. file_request->local_positions = *fixed_local_positions; } + file_request->non_predicate_positions.clear(); + if (fixed_non_predicate_positions != nullptr) { + // Deferred output slots are part of the active reader's immutable block layout, just like + // eager slots; retaining only local_positions can shift a later complex root out of bounds. + file_request->non_predicate_positions = *fixed_non_predicate_positions; + } file_request->conjuncts.clear(); + file_request->metadata_pruning_safe_conjunct_count = 0; file_request->delete_conjuncts.clear(); _filter_entries.clear(); // 1. Build referenced non-predicate columns @@ -2188,7 +2264,8 @@ Status TableColumnMapper::create_scan_request( } if (!used_by_filter || !enable_lazy_materialization()) { RETURN_IF_ERROR(add_scan_column(file_request, mapping, false, - force_full_complex_scan_projection())); + force_full_complex_scan_projection(), + enable_variant_leaf_projection())); } } } @@ -2227,12 +2304,12 @@ Status TableColumnMapper::create_scan_request( if (!mapping.file_local_id.has_value()) { continue; } - auto position_it = - file_request->local_positions.find(LocalColumnId(*mapping.file_local_id)); + const auto local_id = LocalColumnId(*mapping.file_local_id); + const auto position_it = file_request->local_positions.find(local_id); DORIS_CHECK(position_it != file_request->local_positions.end()) << file_request->local_positions.size() << " " << *mapping.file_local_id << " " << mapping.file_column_name; - rebuild_projection(&mapping, position_it->second); + rebuild_projection(&mapping, file_request->non_predicate_position(local_id)); } return Status::OK(); } @@ -2246,7 +2323,19 @@ ColumnMapping* TableColumnMapper::_find_mapping(GlobalIndex global_index) { return nullptr; } +ColumnMapping* TableColumnMapper::_find_predicate_mapping(GlobalIndex global_index) { + for (auto& mapping : _predicate_mappings) { + if (mapping.global_index == global_index) { + return &mapping; + } + } + return nullptr; +} + ColumnMapping* TableColumnMapper::_find_filter_mapping(GlobalIndex global_index) { + if (auto* mapping = _find_predicate_mapping(global_index); mapping != nullptr) { + return mapping; + } if (auto* mapping = _find_mapping(global_index); mapping != nullptr) { return mapping; } @@ -2262,10 +2351,7 @@ Status TableColumnMapper::localize_filters(const std::vector& table FileScanRequest* file_request, RuntimeState* runtime_state) { std::set localized_predicate_columns; - FilterProjectionMap filter_projections; auto filter_mappings = _filter_visible_mappings(); - RETURN_IF_ERROR(build_nested_struct_filter_projection_map(table_filters, filter_mappings, - &filter_projections)); for (const auto& table_filter : table_filters) { for (const auto& global_index : table_filter.global_indices) { auto* mapping = _find_filter_mapping(global_index); @@ -2273,9 +2359,29 @@ Status TableColumnMapper::localize_filters(const std::vector& table !filter_conversion_has_local_source(mapping->filter_conversion)) { continue; } + // Nested eager projection is an FE contract. Without predicate_access_paths the + // all-access-path mapping is read as one unit instead of inferring another subtree + // from VExpr and risking a shape that disagrees with final materialization. RETURN_IF_ERROR(add_scan_column(file_request, mapping, enable_lazy_materialization(), force_full_complex_scan_projection(), - &filter_projections)); + enable_variant_leaf_projection())); + auto* output_mapping = _find_mapping(global_index); + if (!enable_independent_predicate_projection() || output_mapping == nullptr || + mapping == output_mapping || !output_mapping->file_local_id.has_value()) { + continue; + } + LocalColumnIndex output_projection; + RETURN_IF_ERROR( + build_scan_projection(output_mapping, force_full_complex_scan_projection(), + enable_variant_leaf_projection(), &output_projection)); + const auto* predicate_projection = find_scan_projection(file_request->predicate_columns, + output_projection.column_id()); + DORIS_CHECK(predicate_projection != nullptr); + if (!same_local_column_index(*predicate_projection, output_projection)) { + FileScanRequestBuilder builder(file_request); + RETURN_IF_ERROR( + builder.add_deferred_non_predicate_column(std::move(output_projection))); + } } } // Rebuild the file type for every scan-local mapping before expression rewrite. Predicate-only @@ -2286,6 +2392,13 @@ Status TableColumnMapper::localize_filters(const std::vector& table RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request, &mapping)); } } + for (auto& mapping : _predicate_mappings) { + if (mapping.file_local_id.has_value() && + file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id))) { + RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request, &mapping, + /*predicate_mapping=*/true)); + } + } for (auto& mapping : _hidden_mappings) { if (mapping.file_local_id.has_value() && file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id))) { @@ -2356,6 +2469,9 @@ Status TableColumnMapper::localize_filters(const std::vector& table auto localized_conjunct = VExprContext::create_shared(std::move(localized_root)); RETURN_IF_ERROR(rewrite_context.prepare_created_exprs(localized_conjunct.get())); file_request->conjuncts.push_back(std::move(localized_conjunct)); + if (table_filter.metadata_pruning_safe) { + ++file_request->metadata_pruning_safe_conjunct_count; + } for (const auto global_index : table_filter.global_indices) { const auto* mapping = _find_filter_mapping(global_index); if (mapping != nullptr && mapping->file_local_id.has_value() && @@ -2368,14 +2484,15 @@ Status TableColumnMapper::localize_filters(const std::vector& table // Candidate columns are added before expression rewriting because their file-block positions // are needed to localize slot refs. If rewriting rejects every filter that references a visible - // column, move its already-merged output/filter projection to the lazy non-predicate set - // instead of forcing it through the eager predicate path. + // column, move its all-access-path projection to the lazy non-predicate set instead of forcing + // it through the eager predicate path. for (auto& mapping : _mappings) { if (!mapping.file_local_id.has_value()) { continue; } const auto local_id = LocalColumnId(*mapping.file_local_id); - if (localized_predicate_columns.contains(local_id)) { + if (localized_predicate_columns.contains(local_id) || + file_request->has_deferred_non_predicate_column(local_id)) { continue; } const auto predicate_it = std::ranges::find_if( @@ -2419,6 +2536,9 @@ Status TableColumnMapper::_create_direct_mapping(const ColumnDefinition& table_c mapping->original_file_children = file_field.children; mapping->projected_file_children = file_field.children; mapping->file_type = file_field.type; + // Access paths are relative to the Variant terminal, so recursive complex mappings must carry + // them instead of leaving them only on the top-level table column. + mapping->variant_access_paths = table_column.variant_access_paths; mapping->is_trivial = mapping_can_use_file_column_directly(*mapping); mapping->filter_conversion = direct_filter_conversion(*mapping); mapping->child_mappings.clear(); @@ -2484,6 +2604,7 @@ Status TableColumnMapper::_create_direct_mapping(const ColumnDefinition& table_c child_mapping.table_type = table_child.type; child_mapping.file_type = table_child.type; child_mapping.default_expr = table_child.default_expr; + child_mapping.variant_access_paths = table_child.variant_access_paths; child_mapping.filter_conversion = FilterConversionType::FINALIZE_ONLY; mapping->child_mappings.push_back(std::move(child_mapping)); continue; diff --git a/be/src/format_v2/column_mapper.h b/be/src/format_v2/column_mapper.h index a3f743378e4fc4..daa628e1b63a3d 100644 --- a/be/src/format_v2/column_mapper.h +++ b/be/src/format_v2/column_mapper.h @@ -128,6 +128,8 @@ struct ColumnMapping { // schema, not table child order. TableReader uses this to map table-output children back to the // file-local block layout when projection, predicate-only children, and schema evolution mix. std::vector projected_file_children; + // Table-side Variant object-key paths retained until the physical shredding schema is known. + std::vector> variant_access_paths; // Split/file-local constant entry when this mapping is produced from partition/default/virtual // expression instead of physical file data. std::optional constant_index; @@ -202,7 +204,8 @@ class TableColumnMapper { const std::vector& table_filters, const std::vector& projected_columns, FileScanRequest* file_request, RuntimeState* runtime_state = nullptr, - const std::map* fixed_local_positions = nullptr); + const std::map* fixed_local_positions = nullptr, + const std::map* fixed_non_predicate_positions = nullptr); // Localize table-level filters to the file schema. // Trivial mappings can copy structured predicates directly. Type changes may be localized with @@ -213,6 +216,7 @@ class TableColumnMapper { RuntimeState* runtime_state = nullptr); void clear() { _mappings.clear(); + _predicate_mappings.clear(); _hidden_mappings.clear(); _constant_map.clear(); _filter_entries.clear(); @@ -233,6 +237,12 @@ class TableColumnMapper { // delimited text field. They must scan the whole complex top-level field and let TableReader // rematerialize the requested table child after row-level filters have run. virtual bool force_full_complex_scan_projection() const { return false; } + // Only Parquet currently has a Variant physical shredding schema and a reader that can + // validate residual-value completeness before honoring a typed-leaf projection. + virtual bool enable_variant_leaf_projection() const { return false; } + // Parquet can keep two independent readers/cursors for the same complex root: a narrow eager + // predicate subtree and the final output subtree materialized only for surviving rows. + virtual bool enable_independent_predicate_projection() const { return false; } const ColumnDefinition* _find_file_field( const ColumnDefinition& table_column, @@ -255,6 +265,7 @@ class TableColumnMapper { std::vector _filter_visible_mappings() const; ColumnMapping* _find_mapping(GlobalIndex global_index); + ColumnMapping* _find_predicate_mapping(GlobalIndex global_index); ColumnMapping* _find_filter_mapping(GlobalIndex global_index); TableColumnMapperOptions _options; @@ -262,6 +273,9 @@ class TableColumnMapper { // describes how to get one table/global column from file-local sources, and carries metadata // for filter localization and result finalize. std::vector _mappings; + // Optional mappings built from SlotDescriptor::predicate_access_paths. They deliberately keep + // a different file type/projection shape from the final output mappings above. + std::vector _predicate_mappings; // Predicate-only top-level columns are not output projection columns, so keep their mappings // here. They are visible only to filter localization and file-reader predicate construction. std::vector _hidden_mappings; @@ -278,6 +292,10 @@ class TableColumnMapper { class ParquetColumnMapper final : public TableColumnMapper { public: using TableColumnMapper::TableColumnMapper; + +protected: + bool enable_variant_leaf_projection() const override { return true; } + bool enable_independent_predicate_projection() const override { return true; } }; // Mapper for readers that always materialize every required file column before filtering. The diff --git a/be/src/format_v2/column_mapper_nested.cpp b/be/src/format_v2/column_mapper_nested.cpp index b172c6ecc874d7..f3cb37f8403c96 100644 --- a/be/src/format_v2/column_mapper_nested.cpp +++ b/be/src/format_v2/column_mapper_nested.cpp @@ -30,87 +30,12 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/primitive_type.h" #include "exprs/vexpr.h" -#include "format_v2/expr/cast.h" #include "gen_cpp/Exprs_types.h" namespace doris::format { namespace { -static bool is_cast_expr(const VExprSPtr& expr) { - return dynamic_cast(expr.get()) != nullptr; -} - -static bool is_signed_integer_type(PrimitiveType type) { - switch (type) { - case TYPE_TINYINT: - case TYPE_SMALLINT: - case TYPE_INT: - case TYPE_BIGINT: - case TYPE_LARGEINT: - return true; - default: - return false; - } -} - -static int primitive_integer_width(PrimitiveType type) { - switch (type) { - case TYPE_TINYINT: - return 1; - case TYPE_SMALLINT: - return 2; - case TYPE_INT: - return 4; - case TYPE_BIGINT: - return 8; - case TYPE_LARGEINT: - return 16; - default: - return 0; - } -} - -static bool is_decimal_type(PrimitiveType type) { - switch (type) { - case TYPE_DECIMAL32: - case TYPE_DECIMAL64: - case TYPE_DECIMALV2: - case TYPE_DECIMAL128I: - case TYPE_DECIMAL256: - return true; - default: - return false; - } -} - -static bool is_order_preserving_safe_cast(const DataTypePtr& from_type, - const DataTypePtr& to_type) { - if (from_type == nullptr || to_type == nullptr) { - return false; - } - const auto from_nested_type = remove_nullable(from_type); - const auto to_nested_type = remove_nullable(to_type); - if (from_nested_type->equals(*to_nested_type)) { - return true; - } - - const auto from_primitive_type = from_nested_type->get_primitive_type(); - const auto to_primitive_type = to_nested_type->get_primitive_type(); - if (is_signed_integer_type(from_primitive_type) && is_signed_integer_type(to_primitive_type)) { - return primitive_integer_width(to_primitive_type) >= - primitive_integer_width(from_primitive_type); - } - if (from_primitive_type == TYPE_FLOAT && to_primitive_type == TYPE_DOUBLE) { - return true; - } - if (is_decimal_type(from_primitive_type) && is_decimal_type(to_primitive_type)) { - return from_nested_type->get_scale() == to_nested_type->get_scale() && - to_nested_type->get_precision() >= from_nested_type->get_precision(); - } - return false; -} - static bool parse_struct_child_selector(const VExprSPtr& expr, StructChildSelector* selector) { DORIS_CHECK(selector != nullptr); if (expr == nullptr || !expr->is_literal()) { @@ -191,27 +116,6 @@ static bool extract_nested_struct_path(const VExprSPtr& expr, NestedStructPath* return true; } -static bool extract_nested_struct_path_for_pruning(const VExprSPtr& expr, NestedStructPath* path) { - DORIS_CHECK(path != nullptr); - // Simple `ELEMENT_AT` - if (extract_nested_struct_path(expr, path)) { - return true; - } - - // `ELEMENT_AT` with `CAST` - if (!is_cast_expr(expr) || expr->get_num_children() != 1) { - return false; - } - const auto& child = expr->children()[0]; - if (!is_order_preserving_safe_cast(child->data_type(), expr->data_type())) { - return false; - } - // A safe widening cast is null-preserving and keeps the comparison ordering of the nested - // primitive leaf, so file-layer pruning can target the original leaf statistics. The row-level - // filter still evaluates the original cast expression after read. - return extract_nested_struct_path_for_pruning(child, path); -} - static const ColumnDefinition* resolve_file_child(const std::vector& children, const StructChildSelector& selector) { if (selector.by_name) { @@ -493,26 +397,6 @@ bool resolve_nested_struct_expr_for_file(const VExprSPtr& expr, return resolve_nested_struct_path_for_file(path, mappings, resolved, true); } -// Collect nested struct leaf references that can be turned into file-reader projections. For -// example, from `s.a > 1 AND element_at(s, 'b') = 2`, this records two paths rooted at `s`: -// `s -> a` and `s -> b`. Non-struct expressions are traversed recursively, while a recognized -// struct path is emitted once so the caller can merge it into the scan projection for that -// top-level file column. -void collect_nested_struct_paths(const VExprSPtr& expr, std::vector* paths) { - DORIS_CHECK(paths != nullptr); - if (expr == nullptr) { - return; - } - NestedStructPath path; - if (extract_nested_struct_path_for_pruning(expr, &path)) { - paths->push_back(std::move(path)); - return; - } - for (const auto& child : expr->children()) { - collect_nested_struct_paths(child, paths); - } -} - std::vector present_child_mappings_in_file_order( const std::vector& child_mappings) { std::vector result; diff --git a/be/src/format_v2/column_mapper_nested.h b/be/src/format_v2/column_mapper_nested.h index 7b2e8cb1513cb4..ab96512e1709ed 100644 --- a/be/src/format_v2/column_mapper_nested.h +++ b/be/src/format_v2/column_mapper_nested.h @@ -86,8 +86,6 @@ bool resolve_nested_struct_expr_for_file(const VExprSPtr& expr, const std::vector& mappings, ResolvedNestedStructPath* resolved); -void collect_nested_struct_paths(const VExprSPtr& expr, std::vector* paths); - std::vector present_child_mappings_in_file_order( const std::vector& child_mappings); diff --git a/be/src/format_v2/file_reader.cpp b/be/src/format_v2/file_reader.cpp index 9bbf7c3a8fc066..a2ca4894044404 100644 --- a/be/src/format_v2/file_reader.cpp +++ b/be/src/format_v2/file_reader.cpp @@ -65,6 +65,14 @@ std::string FileScanRequest::debug_string() const { } out << column_id << ":" << block_position; } + out << "}, non_predicate_positions={"; + position_idx = 0; + for (const auto& [column_id, block_position] : non_predicate_positions) { + if (position_idx++ > 0) { + out << ", "; + } + out << column_id << ":" << block_position; + } out << "}, conjunct_count=" << conjuncts.size() << ", delete_conjunct_count=" << delete_conjuncts.size() << ", count_star_placeholder_columns={"; diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index 65a2d03417c605..ddbabf6329864d 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -68,8 +69,8 @@ struct FileScanRequest { // Columns that must be read before row-level filtering. They are materialized eagerly because // conjuncts/delete_conjuncts need them to decide the selected rows. std::vector predicate_columns; - // Columns read after row-level filtering. Predicate columns are also available for output and - // should not be duplicated here. + // Columns read after row-level filtering. A complex root may intentionally also appear in + // predicate_columns when its eager predicate subtree is smaller than its final output subtree. std::vector non_predicate_columns; // Predicate columns introduced only to evaluate hidden filter slots. Their values are dead // after all file-local predicates run, although the shared file block still needs row-shaped @@ -77,8 +78,15 @@ struct FileScanRequest { std::vector predicate_only_columns; // file-local column id -> file-local output block position. std::map local_positions; + // Optional output position for a root that has independent eager-predicate and deferred-output + // projections. local_positions continues to identify the position referenced by localized + // predicate expressions. + std::map non_predicate_positions; // Row-level filters converted to file-local expressions from table-level predicates. VExprContextSPtrs conjuncts; + // Only this leading subset may participate in footer/page metadata pruning. The boundary is + // inherited from table-conjunct order so an omitted slotless unsafe expression remains a fence. + size_t metadata_pruning_safe_conjunct_count = std::numeric_limits::max(); // Delete predicates converted to file-local expressions. A TRUE result means that the row is // deleted, so readers must invert each result when building their keep filter. VExprContextSPtrs delete_conjuncts; @@ -98,6 +106,26 @@ struct FileScanRequest { bool is_predicate_only(LocalColumnId column_id) const { return std::ranges::find(predicate_only_columns, column_id) != predicate_only_columns.end(); } + + LocalIndex non_predicate_position(LocalColumnId column_id) const { + const auto it = non_predicate_positions.find(column_id); + return it == non_predicate_positions.end() ? local_positions.at(column_id) : it->second; + } + + bool has_deferred_non_predicate_column(LocalColumnId column_id) const { + return non_predicate_positions.contains(column_id); + } + + size_t block_column_count() const { + size_t count = 0; + for (const auto& [_, position] : local_positions) { + count = std::max(count, position.value() + 1); + } + for (const auto& [_, position] : non_predicate_positions) { + count = std::max(count, position.value() + 1); + } + return count; + } }; // Helper for constructing the scan-column layout in FileScanRequest. @@ -105,9 +133,10 @@ struct FileScanRequest { // as Parquet can read predicate columns first, filter rows, and then lazily read the remaining // projected columns. The two lists still share one file-local output block, whose positions are // stored in local_positions. This builder centralizes the mechanical rules for that shared layout: -// - each root file column gets one stable block position; +// - each root file column gets one stable predicate block position; // - predicate columns dominate non-predicate columns because they are already returned in the file // block and can be reused for final materialization; +// - a smaller complex predicate subtree may get a second deferred output position; // - repeated nested projections for the same root are merged instead of duplicated. // TableColumnMapper should still own table-to-file semantic resolution. This helper only owns the // FileScanRequest layout contract after a file-local projection has been produced. @@ -127,6 +156,38 @@ class FileScanRequestBuilder { /*is_predicate_column=*/false); } + Status add_deferred_non_predicate_column(LocalColumnIndex projection) { + const auto file_column_id = projection.column_id(); + DORIS_CHECK(file_column_id != LocalColumnId::invalid()); + DORIS_CHECK(_request->local_positions.contains(file_column_id)); + DORIS_CHECK(std::ranges::any_of(_request->predicate_columns, + [&](const LocalColumnIndex& predicate) { + return predicate.column_id() == file_column_id; + })); + + if (!_request->non_predicate_positions.contains(file_column_id)) { + _request->non_predicate_positions.emplace(file_column_id, + _next_block_position(*_request)); + } + _sort_projection_children_by_file_id(&projection); + auto existing = std::ranges::find_if(_request->non_predicate_columns, + [&](const LocalColumnIndex& output) { + return output.column_id() == file_column_id; + }); + if (existing == _request->non_predicate_columns.end()) { + _request->non_predicate_columns.push_back(std::move(projection)); + } else { + RETURN_IF_ERROR(merge_local_column_index(&*existing, projection)); + _sort_projection_children_by_file_id(&*existing); + } + if (!_request->is_predicate_only(file_column_id)) { + // The eager complex value has a different physical shape from the final value and + // must never leak into table materialization after its predicates have run. + _request->predicate_only_columns.push_back(file_column_id); + } + return Status::OK(); + } + Status add_predicate_column(LocalColumnId column_id) { return add_predicate_column(LocalColumnIndex::top_level(column_id)); } @@ -141,6 +202,9 @@ class FileScanRequestBuilder { for (const auto& [_, block_position] : request.local_positions) { next_position = std::max(next_position, block_position.value() + 1); } + for (const auto& [_, block_position] : request.non_predicate_positions) { + next_position = std::max(next_position, block_position.value() + 1); + } return LocalIndex(next_position); } @@ -164,9 +228,11 @@ class FileScanRequestBuilder { const auto file_column_id = projection.column_id(); DORIS_CHECK(file_column_id != LocalColumnId::invalid()); if (!is_predicate_column && - std::ranges::find_if(_request->predicate_columns, [&](const LocalColumnIndex& p) { - return p.column_id() == file_column_id; - }) != _request->predicate_columns.end()) { + std::ranges::find_if(_request->predicate_columns, + [&](const LocalColumnIndex& p) { + return p.column_id() == file_column_id; + }) != _request->predicate_columns.end() && + !_request->has_deferred_non_predicate_column(file_column_id)) { return Status::OK(); } if (!_request->local_positions.contains(file_column_id)) { @@ -184,7 +250,7 @@ class FileScanRequestBuilder { _sort_projection_children_by_file_id(&*existing_projection_it); } - if (is_predicate_column) { + if (is_predicate_column && !_request->has_deferred_non_predicate_column(file_column_id)) { auto it = std::ranges::find_if( _request->non_predicate_columns, [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); diff --git a/be/src/format_v2/parquet/native_schema_desc.cpp b/be/src/format_v2/parquet/native_schema_desc.cpp index 9eba6d8feaa55a..b56afb6f7170fb 100644 --- a/be/src/format_v2/parquet/native_schema_desc.cpp +++ b/be/src/format_v2/parquet/native_schema_desc.cpp @@ -20,7 +20,9 @@ #include #include +#include #include +#include #include #include "common/cast_set.h" @@ -55,6 +57,412 @@ static bool is_map_node(const tparquet::SchemaElement& schema) { (schema.__isset.logicalType && schema.logicalType.__isset.MAP); } +static bool is_variant_node(const tparquet::SchemaElement& schema) { + return schema.__isset.logicalType && schema.logicalType.__isset.VARIANT; +} + +enum class VariantPrimitiveAnnotation : uint8_t { + NONE, + INT8, + INT16, + DECIMAL, + DATE, + TIME_MICROS, + TIMESTAMP_MICROS, + TIMESTAMP_NANOS, + STRING, + UUID, + UNSUPPORTED, +}; + +static VariantPrimitiveAnnotation variant_logical_annotation( + const tparquet::SchemaElement& schema) { + if (!schema.__isset.logicalType) { + return VariantPrimitiveAnnotation::NONE; + } + const auto& logical = schema.logicalType; + if (logical.__isset.INTEGER) { + if (!logical.INTEGER.isSigned) { + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.INTEGER.bitWidth == 8) { + return VariantPrimitiveAnnotation::INT8; + } + if (logical.INTEGER.bitWidth == 16) { + return VariantPrimitiveAnnotation::INT16; + } + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.DECIMAL) { + return VariantPrimitiveAnnotation::DECIMAL; + } + if (logical.__isset.DATE) { + return VariantPrimitiveAnnotation::DATE; + } + if (logical.__isset.TIME) { + return !logical.TIME.isAdjustedToUTC && logical.TIME.unit.__isset.MICROS + ? VariantPrimitiveAnnotation::TIME_MICROS + : VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.TIMESTAMP) { + if (logical.TIMESTAMP.unit.__isset.MICROS) { + return VariantPrimitiveAnnotation::TIMESTAMP_MICROS; + } + if (logical.TIMESTAMP.unit.__isset.NANOS) { + return VariantPrimitiveAnnotation::TIMESTAMP_NANOS; + } + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.STRING) { + return VariantPrimitiveAnnotation::STRING; + } + if (logical.__isset.UUID) { + return VariantPrimitiveAnnotation::UUID; + } + const bool empty = !logical.__isset.MAP && !logical.__isset.LIST && !logical.__isset.ENUM && + !logical.__isset.UNKNOWN && !logical.__isset.JSON && !logical.__isset.BSON && + !logical.__isset.FLOAT16 && !logical.__isset.GEOMETRY && + !logical.__isset.GEOGRAPHY && !logical.__isset.VARIANT; + return empty ? VariantPrimitiveAnnotation::NONE : VariantPrimitiveAnnotation::UNSUPPORTED; +} + +static VariantPrimitiveAnnotation variant_converted_annotation( + const tparquet::SchemaElement& schema) { + if (!schema.__isset.converted_type) { + return VariantPrimitiveAnnotation::NONE; + } + switch (schema.converted_type) { + case tparquet::ConvertedType::INT_8: + return VariantPrimitiveAnnotation::INT8; + case tparquet::ConvertedType::INT_16: + return VariantPrimitiveAnnotation::INT16; + case tparquet::ConvertedType::DECIMAL: + return VariantPrimitiveAnnotation::DECIMAL; + case tparquet::ConvertedType::DATE: + return VariantPrimitiveAnnotation::DATE; + case tparquet::ConvertedType::TIME_MICROS: + return VariantPrimitiveAnnotation::TIME_MICROS; + case tparquet::ConvertedType::TIMESTAMP_MICROS: + return VariantPrimitiveAnnotation::TIMESTAMP_MICROS; + case tparquet::ConvertedType::UTF8: + return VariantPrimitiveAnnotation::STRING; + default: + return VariantPrimitiveAnnotation::UNSUPPORTED; + } +} + +static Status validate_variant_decimal(const tparquet::SchemaElement& schema, + tparquet::Type::type physical_type) { + int32_t precision = -1; + int32_t scale = -1; + if (schema.__isset.logicalType && schema.logicalType.__isset.DECIMAL) { + precision = schema.logicalType.DECIMAL.precision; + scale = schema.logicalType.DECIMAL.scale; + if ((schema.__isset.precision && schema.precision != precision) || + (schema.__isset.scale && schema.scale != scale)) { + return Status::Corruption( + "Parquet Variant DECIMAL logical and converted parameters disagree"); + } + } else if (schema.__isset.precision && schema.__isset.scale) { + precision = schema.precision; + scale = schema.scale; + } + if (precision <= 0 || precision > 38 || scale < 0 || scale > precision) { + return Status::Corruption("Parquet Variant DECIMAL({}, {}) is invalid", precision, scale); + } + + if ((physical_type == tparquet::Type::INT32 && precision > 9) || + (physical_type == tparquet::Type::INT64 && (precision < 10 || precision > 18)) || + ((physical_type == tparquet::Type::BYTE_ARRAY || + physical_type == tparquet::Type::FIXED_LEN_BYTE_ARRAY) && + precision < 19)) { + return Status::Corruption( + "Parquet Variant DECIMAL precision {} does not match physical type {}", precision, + physical_type); + } + if (physical_type == tparquet::Type::FIXED_LEN_BYTE_ARRAY) { + static constexpr int32_t MAX_PRECISION_BY_LENGTH[] = {2, 4, 6, 9, 11, 14, 16, 18, + 21, 23, 26, 28, 31, 33, 35, 38}; + if (!schema.__isset.type_length || schema.type_length <= 0 || schema.type_length > 16 || + precision > MAX_PRECISION_BY_LENGTH[schema.type_length - 1]) { + return Status::Corruption( + "Parquet Variant DECIMAL precision {} does not fit fixed length {}", precision, + schema.__isset.type_length ? schema.type_length : -1); + } + } + return Status::OK(); +} + +static Status validate_variant_primitive_type(const NativeFieldSchema& typed) { + const auto& schema = typed.parquet_schema; + auto logical = variant_logical_annotation(schema); + auto converted = variant_converted_annotation(schema); + if (logical == VariantPrimitiveAnnotation::UNSUPPORTED || + converted == VariantPrimitiveAnnotation::UNSUPPORTED || + (logical != VariantPrimitiveAnnotation::NONE && + converted != VariantPrimitiveAnnotation::NONE && logical != converted)) { + return Status::Corruption( + "Parquet Variant typed value {} has an unsupported logical annotation", typed.name); + } + const auto annotation = logical != VariantPrimitiveAnnotation::NONE ? logical : converted; + const auto physical = schema.type; + bool valid = false; + switch (physical) { + case tparquet::Type::BOOLEAN: + valid = annotation == VariantPrimitiveAnnotation::NONE; + break; + case tparquet::Type::INT32: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::INT8 || + annotation == VariantPrimitiveAnnotation::INT16 || + annotation == VariantPrimitiveAnnotation::DECIMAL || + annotation == VariantPrimitiveAnnotation::DATE; + break; + case tparquet::Type::INT64: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::DECIMAL || + annotation == VariantPrimitiveAnnotation::TIME_MICROS || + annotation == VariantPrimitiveAnnotation::TIMESTAMP_MICROS || + annotation == VariantPrimitiveAnnotation::TIMESTAMP_NANOS; + break; + case tparquet::Type::FLOAT: + case tparquet::Type::DOUBLE: + valid = annotation == VariantPrimitiveAnnotation::NONE; + break; + case tparquet::Type::BYTE_ARRAY: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::STRING || + annotation == VariantPrimitiveAnnotation::DECIMAL; + break; + case tparquet::Type::FIXED_LEN_BYTE_ARRAY: + valid = annotation == VariantPrimitiveAnnotation::DECIMAL || + (annotation == VariantPrimitiveAnnotation::UUID && schema.__isset.type_length && + schema.type_length == 16); + break; + default: + valid = false; + break; + } + if (!valid) { + return Status::Corruption( + "Parquet Variant typed value {} has unsupported physical/logical type pair", + typed.name); + } + if (annotation == VariantPrimitiveAnnotation::DECIMAL) { + RETURN_IF_ERROR(validate_variant_decimal(schema, physical)); + } + return Status::OK(); +} + +class ScopedBoolOverride { +public: + ScopedBoolOverride(bool& target, bool value) : _target(target), _original(target) { + _target = value; + } + ~ScopedBoolOverride() { _target = _original; } + +private: + bool& _target; + bool _original; +}; + +static Status validate_variant_layout(const tparquet::SchemaElement& group_schema, + const NativeFieldSchema& group_field) { + const auto& annotation = group_schema.logicalType.VARIANT; + if (annotation.__isset.specification_version && annotation.specification_version != 1) { + return Status::NotSupported("Parquet Variant specification version {} is not supported", + annotation.specification_version); + } + if (group_field.children.size() < 2 || group_field.children.size() > 3) { + return Status::Corruption( + "Parquet Variant {} must contain metadata, value, and optional typed_value", + group_schema.name); + } + + const NativeFieldSchema* metadata = nullptr; + const NativeFieldSchema* value = nullptr; + const NativeFieldSchema* typed_value = nullptr; + for (const auto& child : group_field.children) { + const NativeFieldSchema** target = nullptr; + if (child.name == "metadata") { + target = &metadata; + } else if (child.name == "value") { + target = &value; + } else if (child.name == "typed_value") { + target = &typed_value; + } else { + return Status::Corruption("Parquet Variant {} has unexpected child {}", + group_schema.name, child.name); + } + if (*target != nullptr) { + return Status::Corruption("Parquet Variant {} has duplicate child {}", + group_schema.name, child.name); + } + *target = &child; + } + if (metadata == nullptr || value == nullptr) { + return Status::Corruption("Parquet Variant {} requires metadata and value children", + group_schema.name); + } + if (!metadata->children.empty() || metadata->physical_type != tparquet::Type::BYTE_ARRAY || + metadata->parquet_schema.repetition_type != tparquet::FieldRepetitionType::REQUIRED) { + return Status::Corruption("Parquet Variant {} metadata must be a required BYTE_ARRAY", + group_schema.name); + } + const auto expected_value_repetition = typed_value == nullptr + ? tparquet::FieldRepetitionType::REQUIRED + : tparquet::FieldRepetitionType::OPTIONAL; + // SQL nullability belongs to the outer Variant group. Only shredding makes value optional, + // because typed_value may carry all or part of the logical value instead. + if (!value->children.empty() || value->physical_type != tparquet::Type::BYTE_ARRAY || + value->parquet_schema.repetition_type != expected_value_repetition) { + return Status::Corruption("Parquet Variant {} value must be a {} BYTE_ARRAY", + group_schema.name, + typed_value == nullptr ? "required" : "optional"); + } + if (typed_value != nullptr && + typed_value->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL) { + return Status::Corruption("Parquet Variant {} typed_value must be optional", + group_schema.name); + } + + enum class WrapperContext : uint8_t { OBJECT_FIELD, ARRAY_ELEMENT }; + std::function validate_typed_value; + std::function validate_wrapper; + validate_wrapper = [&](const NativeFieldSchema& wrapper, WrapperContext context) -> Status { + if (!wrapper.parquet_schema.__isset.repetition_type || + wrapper.parquet_schema.repetition_type != tparquet::FieldRepetitionType::REQUIRED) { + return Status::Corruption("Parquet Variant shredded wrapper {} must be required", + wrapper.name); + } + const NativeFieldSchema* fallback = nullptr; + const NativeFieldSchema* typed = nullptr; + for (const auto& child : wrapper.children) { + if (child.name == "value") { + if (fallback != nullptr) { + return Status::Corruption( + "Parquet Variant wrapper {} has duplicate value child", wrapper.name); + } + fallback = &child; + } else if (child.name == "typed_value") { + if (typed != nullptr) { + return Status::Corruption( + "Parquet Variant wrapper {} has duplicate typed_value child", + wrapper.name); + } + typed = &child; + } else { + return Status::Corruption("Parquet Variant wrapper {} has unexpected child {}", + wrapper.name, child.name); + } + } + if (fallback == nullptr && typed == nullptr) { + return Status::Corruption( + "Parquet Variant shredded wrapper {} requires at least one of value or " + "typed_value", + wrapper.name); + } + // Object fields always retain the fallback value carrier; only typed_value is optional. + // Array elements may omit either carrier when every element uses the remaining one. + if (context == WrapperContext::OBJECT_FIELD && fallback == nullptr) { + return Status::Corruption( + "Parquet Variant object wrapper {} requires an optional value child", + wrapper.name); + } + if (fallback != nullptr && + (!fallback->children.empty() || fallback->physical_type != tparquet::Type::BYTE_ARRAY || + !fallback->parquet_schema.__isset.repetition_type || + fallback->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL)) { + return Status::Corruption( + "Parquet Variant wrapper {} value must be an optional BYTE_ARRAY", + wrapper.name); + } + if (typed != nullptr) { + if (!typed->parquet_schema.__isset.repetition_type || + typed->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL) { + return Status::Corruption("Parquet Variant wrapper {} typed_value must be optional", + wrapper.name); + } + return validate_typed_value(*typed); + } + return Status::OK(); + }; + validate_typed_value = [&](const NativeFieldSchema& typed) -> Status { + if (!typed.unsupported_reason.empty()) { + return Status::NotSupported("Parquet Variant typed value {} is not supported: {}", + typed.name, typed.unsupported_reason); + } + if (typed.children.empty()) { + const auto& physical = typed.parquet_schema; + if (physical.__isset.logicalType && physical.logicalType.__isset.INTEGER && + !physical.logicalType.INTEGER.isSigned) { + return Status::Corruption( + "Parquet Variant unsigned integers are not valid typed values"); + } + if (physical.__isset.converted_type && + (physical.converted_type == tparquet::ConvertedType::UINT_8 || + physical.converted_type == tparquet::ConvertedType::UINT_16 || + physical.converted_type == tparquet::ConvertedType::UINT_32 || + physical.converted_type == tparquet::ConvertedType::UINT_64)) { + return Status::Corruption( + "Parquet Variant unsigned integers are not valid typed values"); + } + if (physical.__isset.logicalType && physical.logicalType.__isset.TIME) { + const auto& time = physical.logicalType.TIME; + // Variant v1 has one canonical TIME representation: local wall-clock MICROS. + // Accepting adjusted or lower-precision forms would make projection-dependent + // reconstruction disagree with the canonical Variant value. + if (time.isAdjustedToUTC) { + return Status::Corruption( + "Parquet Variant TIME must have isAdjustedToUTC=false"); + } + if (!time.unit.__isset.MICROS) { + return Status::Corruption( + "Parquet Variant TIME(MILLIS) is not supported; use TIME(MICROS)"); + } + } + if (physical.__isset.converted_type && + physical.converted_type == tparquet::ConvertedType::TIME_MILLIS) { + return Status::Corruption( + "Parquet Variant TIME(MILLIS) is not supported; use TIME(MICROS)"); + } + if (physical.__isset.logicalType && physical.logicalType.__isset.TIMESTAMP && + physical.logicalType.TIMESTAMP.unit.__isset.NANOS) { + // Reject at schema open so full reconstruction and direct typed-leaf access have + // the same precision contract instead of diverging after projection planning. + return Status::NotSupported("Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + // Preserve precise diagnostics above, then enforce the complete Variant matrix before + // generic Parquet inference can re-encode a value with another logical identity. + RETURN_IF_ERROR(validate_variant_primitive_type(typed)); + return Status::OK(); + } + + const PrimitiveType primitive = remove_nullable(typed.data_type)->get_primitive_type(); + if (primitive == TYPE_STRUCT) { + std::unordered_set field_names; + for (const auto& child : typed.children) { + if (!field_names.insert(child.name).second) { + // Name lookup selects one physical wrapper, so duplicates would otherwise make + // full reconstruction and leaf projection observe different logical values. + return Status::Corruption( + "Parquet Variant object has duplicate shredded field {}", child.name); + } + RETURN_IF_ERROR(validate_wrapper(child, WrapperContext::OBJECT_FIELD)); + } + return Status::OK(); + } + if (primitive == TYPE_ARRAY && typed.children.size() == 1) { + return validate_wrapper(typed.children[0], WrapperContext::ARRAY_ELEMENT); + } + return Status::Corruption("Invalid Parquet Variant typed_value schema {}", typed.name); + }; + if (typed_value != nullptr) { + RETURN_IF_ERROR(validate_typed_value(*typed_value)); + } + return Status::OK(); +} + static bool has_primitive_only_annotation(const tparquet::SchemaElement& schema) { if (schema.__isset.logicalType) { const auto& logical = schema.logicalType; @@ -265,6 +673,11 @@ Status NativeFieldDescriptor::parse_node_field( // nested structure or nullable list return parse_group_field(t_schemas, curr_pos, node_field); } + if (is_variant_node(t_schema)) { + return Status::InvalidArgument( + "Parquet Variant logical type requires a group node, got primitive {}", + t_schema.name); + } if (is_repeated_node(t_schema)) { // repeated (LIST) // produce required list @@ -517,6 +930,27 @@ Status NativeFieldDescriptor::parse_group_field( const std::vector& t_schemas, size_t curr_pos, NativeFieldSchema* group_field) { auto& group_schema = t_schemas[curr_pos]; + group_field->parquet_schema = group_schema; + if (is_variant_node(group_schema)) { + if (is_repeated_node(group_schema)) { + // A repeated annotated group needs an ARRAY carrier and Dremel-level remapping; treating + // it as a scalar Variant would expose the wrong row shape. + return Status::NotSupported("repeated Parquet Variant group {} is not supported", + group_schema.name); + } + // UTC-adjusted timestamps inside Variant carry an instant, independent of the catalog's + // presentation mapping. Parsing them as DATETIMEV2 would apply the session timezone and + // lose that instant when the shredded value is re-encoded into ColumnVariantV2. + { + ScopedBoolOverride timestamp_tz_mapping(_enable_mapping_timestamp_tz, true); + RETURN_IF_ERROR(parse_struct_field(t_schemas, curr_pos, group_field)); + } + RETURN_IF_ERROR(validate_variant_layout(group_schema, *group_field)); + group_field->variant_physical_type = group_field->data_type; + // Native page readers dispatch groups from data_type, so preserve the physical STRUCT + // here. The public Parquet schema maps it to logical Variant without losing this shape. + return Status::OK(); + } if ((group_schema.__isset.logicalType && group_schema.logicalType.__isset.ENUM) || (group_schema.__isset.converted_type && group_schema.converted_type == tparquet::ConvertedType::ENUM)) { diff --git a/be/src/format_v2/parquet/native_schema_desc.h b/be/src/format_v2/parquet/native_schema_desc.h index dfb669559fc71a..918be6d2c65c58 100644 --- a/be/src/format_v2/parquet/native_schema_desc.h +++ b/be/src/format_v2/parquet/native_schema_desc.h @@ -47,6 +47,10 @@ struct NativeFieldSchema { // Used to identify whether this field is a nested field. DataTypePtr data_type; + + // VARIANT is logically exposed as DataTypeVariantV2, while native page readers still need the + // physical STRUCT shape formed by metadata/value/typed_value. + DataTypePtr variant_physical_type; // Schema construction keeps a physical fallback so unprojected columns and metadata-only // queries remain readable, while projection validation reports the original logical failure. std::string unsupported_reason; diff --git a/be/src/format_v2/parquet/native_schema_node.cpp b/be/src/format_v2/parquet/native_schema_node.cpp index 052df93e4b4143..e1f573951fbbfc 100644 --- a/be/src/format_v2/parquet/native_schema_node.cpp +++ b/be/src/format_v2/parquet/native_schema_node.cpp @@ -64,7 +64,8 @@ Status build_native_schema_node(const DataTypePtr& projected_type, const auto type = remove_nullable(projected_type); switch (type->get_primitive_type()) { case TYPE_STRUCT: { - if (file_schema.kind != ParquetColumnSchemaKind::STRUCT) { + if (file_schema.kind != ParquetColumnSchemaKind::STRUCT && + file_schema.kind != ParquetColumnSchemaKind::VARIANT) { return Status::Corruption("Parquet column {} is not a STRUCT", file_schema.name); } const auto* struct_type = assert_cast(type.get()); diff --git a/be/src/format_v2/parquet/parquet_column_schema.cpp b/be/src/format_v2/parquet/parquet_column_schema.cpp index 7e541b57da2780..71416e17dc9209 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.cpp +++ b/be/src/format_v2/parquet/parquet_column_schema.cpp @@ -20,7 +20,11 @@ #include #include +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "format_v2/parquet/native_schema_desc.h" #include "format_v2/parquet/parquet_type.h" @@ -63,7 +67,11 @@ void fill_native_type_descriptor(const NativeFieldSchema& field, ParquetTypeDesc result->fixed_length = schema.__isset.type_length ? schema.type_length : -1; if (schema.__isset.logicalType) { const auto& logical = schema.logicalType; - if (logical.__isset.DECIMAL) { + if (logical.__isset.STRING) { + result->is_string_annotation = true; + } else if (logical.__isset.UUID) { + result->is_uuid = true; + } else if (logical.__isset.DECIMAL) { result->is_decimal = true; result->decimal_precision = logical.DECIMAL.precision; result->decimal_scale = logical.DECIMAL.scale; @@ -87,6 +95,9 @@ void fill_native_type_descriptor(const NativeFieldSchema& field, ParquetTypeDesc } } else if (schema.__isset.converted_type) { switch (schema.converted_type) { + case tparquet::ConvertedType::UTF8: + result->is_string_annotation = true; + break; case tparquet::ConvertedType::DECIMAL: result->is_decimal = true; result->decimal_precision = schema.__isset.precision ? schema.precision : -1; @@ -173,7 +184,15 @@ std::unique_ptr build_native_node_schema(const NativeFieldS result->local_id = local_id; result->parquet_field_id = field.field_id; result->name = field.name; - result->type = field.data_type; + result->variant_physical_type = field.variant_physical_type; + if (field.variant_physical_type != nullptr) { + DataTypePtr variant_type = std::make_shared(); + result->type = field.variant_physical_type->is_nullable() + ? make_nullable(std::move(variant_type)) + : std::move(variant_type); + } else { + result->type = field.data_type; + } result->definition_level = field.definition_level; result->repetition_level = field.repetition_level; result->max_definition_level = field.definition_level; @@ -191,7 +210,10 @@ std::unique_ptr build_native_node_schema(const NativeFieldS fill_native_type_descriptor(field, &result->type_descriptor); return result; } - if (primitive_type == TYPE_ARRAY) { + if (field.variant_physical_type != nullptr) { + result->kind = ParquetColumnSchemaKind::VARIANT; + result->contains_variant = true; + } else if (primitive_type == TYPE_ARRAY) { result->kind = ParquetColumnSchemaKind::LIST; } else if (primitive_type == TYPE_MAP) { result->kind = ParquetColumnSchemaKind::MAP; @@ -202,6 +224,33 @@ std::unique_ptr build_native_node_schema(const NativeFieldS for (size_t child_idx = 0; child_idx < field.children.size(); ++child_idx) { result->children.push_back( build_native_node_schema(field.children[child_idx], cast_set(child_idx))); + result->contains_variant |= result->children.back()->contains_variant; + } + // A nested Variant changes its public child type from the physical STRUCT carrier. Rebuild + // every enclosing complex type so file-block columns keep the same logical shape as readers. + if (result->kind != ParquetColumnSchemaKind::VARIANT && result->contains_variant) { + DataTypePtr logical_type; + if (result->kind == ParquetColumnSchemaKind::LIST) { + DORIS_CHECK(result->children.size() == 1); + logical_type = std::make_shared(result->children[0]->type); + } else if (result->kind == ParquetColumnSchemaKind::MAP) { + DORIS_CHECK(result->children.size() == 2); + logical_type = std::make_shared(make_nullable(result->children[0]->type), + make_nullable(result->children[1]->type)); + } else { + DataTypes child_types; + Strings child_names; + child_types.reserve(result->children.size()); + child_names.reserve(result->children.size()); + for (const auto& child : result->children) { + child_types.push_back(child->type); + child_names.push_back(child->name); + } + logical_type = std::make_shared(std::move(child_types), + std::move(child_names)); + } + result->type = result->type->is_nullable() ? make_nullable(std::move(logical_type)) + : std::move(logical_type); } propagate_native_max_levels(result.get()); return result; diff --git a/be/src/format_v2/parquet/parquet_column_schema.h b/be/src/format_v2/parquet/parquet_column_schema.h index 697bcd498382b1..3103b97cfc253c 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.h +++ b/be/src/format_v2/parquet/parquet_column_schema.h @@ -32,6 +32,7 @@ enum class ParquetColumnSchemaKind { STRUCT, // Parquet group with STRUCT semantics LIST, // Parquet group with LIST semantics MAP, // Parquet group with MAP semantics + VARIANT, // Parquet Variant logical group }; // ============================================================================ @@ -46,12 +47,20 @@ struct ParquetColumnSchema { DataTypePtr type = nullptr; + // Set only for VARIANT. The public file type is DataTypeVariantV2, while this type describes + // the metadata/value/typed_value STRUCT consumed by the native decoder. + DataTypePtr variant_physical_type = nullptr; + int leaf_column_id = -1; ParquetTypeDescriptor type_descriptor {}; ParquetColumnSchemaKind kind = ParquetColumnSchemaKind::PRIMITIVE; + // Cached during schema construction so readers created per row group do not repeatedly walk + // ordinary nested schemas to discover whether Variant-specific planning is needed. + bool contains_variant = false; + // ======== Dremel Levels ======== int16_t max_definition_level = 0; diff --git a/be/src/format_v2/parquet/parquet_file_context.cpp b/be/src/format_v2/parquet/parquet_file_context.cpp index c178e491b8ac21..8ba8cf94662f9a 100644 --- a/be/src/format_v2/parquet/parquet_file_context.cpp +++ b/be/src/format_v2/parquet/parquet_file_context.cpp @@ -273,6 +273,7 @@ Status ParquetFileContext::open(io::FileReaderSPtr input_file_reader, io::IOCont bool enable_page_cache, const io::FileDescription& file_description, bool enable_mapping_timestamp_tz, bool enable_mapping_varbinary) { DORIS_CHECK(input_file_reader != nullptr); + contains_variant = false; if (detail::should_stage_small_http_file(input_file_reader->path().native(), input_file_reader->size(), config::in_memory_file_size)) { @@ -602,6 +603,7 @@ Status ParquetFileContext::close() { native_io_ctx = nullptr; native_page_cache_enabled = false; native_page_cache_file_key.clear(); + contains_variant = false; return Status::OK(); } diff --git a/be/src/format_v2/parquet/parquet_file_context.h b/be/src/format_v2/parquet/parquet_file_context.h index 0cd413e10557ac..38e78438e39b29 100644 --- a/be/src/format_v2/parquet/parquet_file_context.h +++ b/be/src/format_v2/parquet/parquet_file_context.h @@ -139,6 +139,9 @@ struct ParquetFileContext { int64_t native_footer_cache_hits = 0; bool native_page_cache_enabled = false; std::string native_page_cache_file_key; + // Set once after the logical file schema is built. Per-request planning uses this guard so + // ordinary files never enter Variant projection or shredded-statistics paths. + bool contains_variant = false; Status open(io::FileReaderSPtr input_file_reader, io::IOContext* io_ctx, bool enable_page_cache, const io::FileDescription& file_description, diff --git a/be/src/format_v2/parquet/parquet_profile.cpp b/be/src/format_v2/parquet/parquet_profile.cpp index e6717505cea898..5fd3100a2e5bcf 100644 --- a/be/src/format_v2/parquet/parquet_profile.cpp +++ b/be/src/format_v2/parquet/parquet_profile.cpp @@ -56,6 +56,8 @@ void ParquetProfile::init(RuntimeProfile* profile) { parquet_profile, 1); filtered_page_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "FilteredRowsByPage", TUnit::UNIT, parquet_profile, 1); + variant_leaf_projections = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantLeafProjections", + TUnit::UNIT, parquet_profile, 1); pages_skipped_by_data_page_filter = ADD_CHILD_COUNTER_WITH_LEVEL( profile, "PagesSkippedByDataPageFilter", TUnit::UNIT, parquet_profile, 1); data_page_filter_skip_bytes = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "DataPageFilterSkipBytes", @@ -88,6 +90,18 @@ void ParquetProfile::init(RuntimeProfile* profile) { ADD_CHILD_TIMER_WITH_LEVEL(profile, "LevelOnlySkipTime", parquet_profile, 1); materialization_time = ADD_CHILD_TIMER_WITH_LEVEL(profile, "MaterializationTime", parquet_profile, 1); + variant_reconstruction_time = + ADD_CHILD_TIMER_WITH_LEVEL(profile, "VariantReconstructionTime", parquet_profile, 1); + variant_reconstructed_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantReconstructedRows", + TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantDirectLeafRows", + TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_path_misses = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafPathMisses", TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_residual_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafResidualFallbacks", TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_unsupported_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafUnsupportedFallbacks", TUnit::UNIT, parquet_profile, 1); hybrid_selection_batches = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "HybridSelectionBatches", TUnit::UNIT, parquet_profile, 1); hybrid_selection_ranges = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "HybridSelectionRanges", @@ -298,6 +312,12 @@ ParquetColumnReaderProfile ParquetProfile::column_reader_profile() const { .level_only_read_time = level_only_read_time, .level_only_skip_time = level_only_skip_time, .materialization_time = materialization_time, + .variant_reconstruction_time = variant_reconstruction_time, + .variant_reconstructed_rows = variant_reconstructed_rows, + .variant_direct_leaf_rows = variant_direct_leaf_rows, + .variant_direct_leaf_path_misses = variant_direct_leaf_path_misses, + .variant_direct_leaf_residual_fallbacks = variant_direct_leaf_residual_fallbacks, + .variant_direct_leaf_unsupported_fallbacks = variant_direct_leaf_unsupported_fallbacks, .hybrid_selection_batches = hybrid_selection_batches, .hybrid_selection_ranges = hybrid_selection_ranges, .hybrid_selection_null_fallback_batches = hybrid_selection_null_fallback_batches, diff --git a/be/src/format_v2/parquet/parquet_profile.h b/be/src/format_v2/parquet/parquet_profile.h index 57c03c79336f06..ed1faa8f935134 100644 --- a/be/src/format_v2/parquet/parquet_profile.h +++ b/be/src/format_v2/parquet/parquet_profile.h @@ -38,6 +38,12 @@ struct ParquetColumnReaderProfile { RuntimeProfile::Counter* level_only_read_time = nullptr; RuntimeProfile::Counter* level_only_skip_time = nullptr; RuntimeProfile::Counter* materialization_time = nullptr; // value materialization time (ns) + RuntimeProfile::Counter* variant_reconstruction_time = nullptr; + RuntimeProfile::Counter* variant_reconstructed_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_path_misses = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_residual_fallbacks = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_unsupported_fallbacks = nullptr; RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; @@ -145,6 +151,8 @@ struct ParquetProfile { RuntimeProfile::Counter* selected_row_ranges = nullptr; RuntimeProfile::Counter* filtered_group_rows = nullptr; RuntimeProfile::Counter* filtered_page_rows = nullptr; + // File-level Variant access paths that safely retained a physical typed-leaf projection. + RuntimeProfile::Counter* variant_leaf_projections = nullptr; // ======== Page Skip ======== RuntimeProfile::Counter* pages_skipped_by_data_page_filter = nullptr; @@ -166,6 +174,12 @@ struct ParquetProfile { RuntimeProfile::Counter* level_only_read_time = nullptr; RuntimeProfile::Counter* level_only_skip_time = nullptr; RuntimeProfile::Counter* materialization_time = nullptr; + RuntimeProfile::Counter* variant_reconstruction_time = nullptr; + RuntimeProfile::Counter* variant_reconstructed_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_path_misses = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_residual_fallbacks = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_unsupported_fallbacks = nullptr; RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp index 065adc63b66e1e..8c6efe27558bbc 100644 --- a/be/src/format_v2/parquet/parquet_reader.cpp +++ b/be/src/format_v2/parquet/parquet_reader.cpp @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -57,6 +58,120 @@ struct ParquetReaderScanState { bool enable_strict_mode = false; }; +const ParquetColumnSchema* projected_schema_child(const ParquetColumnSchema& schema, + int32_t local_id) { + const auto child_it = std::ranges::find_if( + schema.children, [local_id](const auto& child) { return child->local_id == local_id; }); + return child_it == schema.children.end() ? nullptr : child_it->get(); +} + +const ParquetColumnSchema* schema_child_by_name(const ParquetColumnSchema& schema, + std::string_view name) { + const auto child_it = std::ranges::find_if( + schema.children, [name](const auto& child) { return child->name == name; }); + return child_it == schema.children.end() ? nullptr : child_it->get(); +} + +bool collect_variant_residual_leaf_ids(const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection, + std::vector* residual_leaf_ids) { + DORIS_CHECK(residual_leaf_ids != nullptr); + const auto* value = schema_child_by_name(schema, "value"); + const auto* typed_value = schema_child_by_name(schema, "typed_value"); + if (value != nullptr && typed_value != nullptr) { + if (value->kind != ParquetColumnSchemaKind::PRIMITIVE || value->leaf_column_id < 0) { + return false; + } + residual_leaf_ids->push_back(value->leaf_column_id); + } + for (const auto& child_projection : projection.children) { + const auto* child = projected_schema_child(schema, child_projection.local_id()); + if (child == nullptr || + !collect_variant_residual_leaf_ids(*child, child_projection, residual_leaf_ids)) { + return false; + } + } + return true; +} + +bool detail::variant_projection_is_fully_shredded(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection) { + if (schema.kind != ParquetColumnSchemaKind::VARIANT || schema.max_repetition_level != 0 || + !format::is_partial_projection(&projection)) { + return false; + } + std::vector residual_leaf_ids; + if (!collect_variant_residual_leaf_ids(schema, projection, &residual_leaf_ids)) { + return false; + } + std::ranges::sort(residual_leaf_ids); + residual_leaf_ids.erase(std::unique(residual_leaf_ids.begin(), residual_leaf_ids.end()), + residual_leaf_ids.end()); + for (const auto& row_group : metadata.row_groups) { + for (const int leaf_id : residual_leaf_ids) { + if (leaf_id < 0 || leaf_id >= static_cast(row_group.columns.size())) { + return false; + } + const auto& chunk = row_group.columns[leaf_id]; + if (!chunk.__isset.meta_data || !chunk.meta_data.__isset.statistics || + !chunk.meta_data.statistics.__isset.null_count || + chunk.meta_data.statistics.null_count != row_group.num_rows) { + return false; + } + } + } + return true; +} + +size_t detail::finalize_variant_leaf_projection(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + format::LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); + if (!format::is_partial_projection(projection)) { + return 0; + } + if (schema.kind == ParquetColumnSchemaKind::VARIANT) { + if (variant_projection_is_fully_shredded(metadata, schema, *projection)) { + return 1; + } + // Unknown residual completeness must restore this Variant wrapper atomically. For a + // repeated ancestor, footer null_count is in the leaf-value domain rather than Variant + // instances, so variant_projection_is_fully_shredded() deliberately takes this fallback. + projection->project_all_children = true; + projection->children.clear(); + return 0; + } + + size_t retained = 0; + for (auto& child_projection : projection->children) { + const auto* child_schema = projected_schema_child(schema, child_projection.local_id()); + DORIS_CHECK(child_schema != nullptr); + retained += finalize_variant_leaf_projection(metadata, *child_schema, &child_projection); + } + return retained; +} + +size_t finalize_variant_leaf_projections( + const NativeParquetMetadata& metadata, + const std::vector>& file_schema, + std::vector* projections) { + DORIS_CHECK(projections != nullptr); + size_t retained = 0; + for (auto& projection : *projections) { + const int32_t local_id = projection.local_id(); + if (local_id < 0 || local_id >= static_cast(file_schema.size())) { + continue; + } + if (!file_schema[local_id]->contains_variant) { + continue; + } + retained += detail::finalize_variant_leaf_projection(metadata.to_thrift(), + *file_schema[local_id], &projection); + } + return retained; +} + Status validate_all_projected_leaves_supported(const ParquetColumnSchema& column_schema) { if (column_schema.kind == ParquetColumnSchemaKind::PRIMITIVE) { if (!column_schema.type_descriptor.unsupported_reason.empty()) { @@ -355,6 +470,11 @@ Status ParquetReader::init(RuntimeState* state) { SCOPED_TIMER(_parquet_profile.parse_meta_time); RETURN_IF_ERROR(build_parquet_column_schema(_state->file_context.native_metadata->schema(), &_state->file_schema)); + _state->file_context.contains_variant = + std::ranges::any_of(_state->file_schema, [](const auto& column) { + DORIS_CHECK(column != nullptr); + return column->contains_variant; + }); if (_enable_mapping_timestamp_tz) { for (auto& column_schema : _state->file_schema) { apply_timestamp_tz_mapping(column_schema.get()); @@ -406,6 +526,20 @@ Status ParquetReader::open(std::shared_ptr request) { } auto request_snapshot = request; DORIS_CHECK(request_snapshot != nullptr); + size_t retained_variant_leaf_projections = 0; + if (_state->file_context.contains_variant) { + retained_variant_leaf_projections = + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->predicate_columns) + + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->non_predicate_columns); + } + if (_parquet_profile.variant_leaf_projections != nullptr) { + COUNTER_UPDATE(_parquet_profile.variant_leaf_projections, + retained_variant_leaf_projections); + } RETURN_IF_ERROR(format::FileReader::open(std::move(request))); // `local_positions.empty()` means all columns are needed by table reader diff --git a/be/src/format_v2/parquet/parquet_reader.h b/be/src/format_v2/parquet/parquet_reader.h index fe95b93a9e0101..0a2b3791502736 100644 --- a/be/src/format_v2/parquet/parquet_reader.h +++ b/be/src/format_v2/parquet/parquet_reader.h @@ -36,6 +36,15 @@ namespace doris::format::parquet { struct ParquetReaderScanState; +namespace detail { +bool variant_projection_is_fully_shredded(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection); +size_t finalize_variant_leaf_projection(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + format::LocalColumnIndex* projection); +} // namespace detail + // ============================================================================ // ============================================================================ // init() -> get_schema() -> open(request) -> get_block() [loop] -> close() diff --git a/be/src/format_v2/parquet/parquet_scan.cpp b/be/src/format_v2/parquet/parquet_scan.cpp index 18b42d90a96368..630e4d8121c970 100644 --- a/be/src/format_v2/parquet/parquet_scan.cpp +++ b/be/src/format_v2/parquet/parquet_scan.cpp @@ -259,7 +259,7 @@ void materialize_count_star_placeholders(const format::FileScanRequest& request, if (!request.is_count_star_placeholder(column.column_id())) { continue; } - const auto block_position = request.local_positions.at(column.column_id()).value(); + const auto block_position = request.non_predicate_position(column.column_id()).value(); auto placeholder = file_block->get_by_position(block_position).column->assert_mutable(); DCHECK(placeholder->empty()); placeholder->insert_many_defaults(rows); @@ -475,8 +475,9 @@ Status finalize_native_row_group_read_plan( std::vector page_selected_ranges; std::map page_skip_plans; RETURN_IF_ERROR(select_row_group_ranges_by_native_page_index( - thrift, page_indexes, file_schema, request, row_group_plan->row_group_rows, - &page_selected_ranges, &page_skip_plans, pruning_stats, timezone, runtime_state)); + thrift, thrift.row_groups[row_group_plan->row_group_id], page_indexes, file_schema, + request, row_group_plan->row_group_rows, &page_selected_ranges, &page_skip_plans, + pruning_stats, timezone, runtime_state)); row_group_plan->selected_ranges = intersect_row_ranges(row_group_plan->selected_ranges, page_selected_ranges); row_group_plan->page_skip_plans = std::move(page_skip_plans); @@ -1176,9 +1177,17 @@ Status ParquetScanScheduler::open_next_row_group( RETURN_IF_ERROR(detail::build_native_prefetch_ranges( thrift_metadata, file_schema, request_scan_columns(request), row_group_idx, file_context.native_file->size(), compat.parquet_816_padding, &native_ranges)); - _current_merge_range_active = file_context.set_native_random_access_ranges( - native_ranges, detail::average_prefetch_range_size(native_ranges), _profile, - _merge_read_slice_size); + if (request.non_predicate_positions.empty()) { + _current_merge_range_active = file_context.set_native_random_access_ranges( + native_ranges, detail::average_prefetch_range_size(native_ranges), _profile, + _merge_read_slice_size); + } else { + // Independent predicate/output readers may revisit the same physical leaf at different + // cursors. MergeRangeFileReader has one consumptive cache per range, so use the random + // access reader for this layout instead of sharing one sequential range cache. + _current_merge_range_active = file_context.set_native_random_access_ranges( + {}, 0, _profile, _merge_read_slice_size); + } for (const auto& col : request.predicate_columns) { const auto local_id = col.column_id(); @@ -2624,9 +2633,7 @@ Status ParquetScanScheduler::read_current_row_group_batch( // selection vector. This also merges pending range gaps with fully filtered batches. RETURN_IF_ERROR(flush_pending_non_predicate_skip_rows()); for (const auto& [fid, column_reader] : _current_non_predicate_columns) { - auto position_it = request.local_positions.find(fid); - DORIS_CHECK(position_it != request.local_positions.end()); - const auto block_position = position_it->second.value(); + const auto block_position = request.non_predicate_position(fid).value(); auto column = file_block->get_by_position(block_position).column->assert_mutable(); DCHECK_EQ(file_block->get_by_position(block_position).type->get_primitive_type(), column_reader->type()->get_primitive_type()) @@ -2694,9 +2701,7 @@ Status ParquetScanScheduler::materialize_pending_predicate_batch( SCOPED_TIMER(_scan_profile.column_read_time); RETURN_IF_ERROR(flush_pending_non_predicate_skip_rows()); for (const auto& [fid, column_reader] : _current_non_predicate_columns) { - auto position_it = request.local_positions.find(fid); - DORIS_CHECK(position_it != request.local_positions.end()); - const auto block_position = position_it->second.value(); + const auto block_position = request.non_predicate_position(fid).value(); auto column = file_block->get_by_position(block_position).column->assert_mutable(); [[maybe_unused]] const auto old_size = column->size(); RETURN_IF_ERROR(column_reader->select(_pending_output_selection, diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index 7f35b087910353..38a7a03c194784 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -37,7 +37,10 @@ #include "core/data_type_serde/data_type_serde.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" +#include "exprs/vectorized_fn_call.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" +#include "exprs/vslot_ref.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_file_context.h" #include "format_v2/parquet/reader/native/block_split_bloom_filter.h" @@ -452,18 +455,339 @@ std::optional file_column_id_by_block_position( return std::nullopt; } +enum class VariantComparisonOp { EQ, NE, LT, LE, GT, GE }; + +struct VariantShreddedPredicate { + int slot_index = -1; + std::vector path; + DataTypePtr comparison_type; + DataTypePtr literal_type; + Field literal; + VariantComparisonOp op = VariantComparisonOp::EQ; +}; + +std::string callable_name(const VExprSPtr& expr) { + if (const auto function = std::dynamic_pointer_cast(expr); + function != nullptr) { + return function->function_name(); + } + return expr == nullptr ? std::string {} : expr->expr_name(); +} + +std::optional variant_comparison_op(std::string_view name) { + if (name == "eq") { + return VariantComparisonOp::EQ; + } + if (name == "ne") { + return VariantComparisonOp::NE; + } + if (name == "lt") { + return VariantComparisonOp::LT; + } + if (name == "le") { + return VariantComparisonOp::LE; + } + if (name == "gt") { + return VariantComparisonOp::GT; + } + if (name == "ge") { + return VariantComparisonOp::GE; + } + return std::nullopt; +} + +VariantComparisonOp reverse_variant_comparison(VariantComparisonOp op) { + switch (op) { + case VariantComparisonOp::EQ: + case VariantComparisonOp::NE: + return op; + case VariantComparisonOp::LT: + return VariantComparisonOp::GT; + case VariantComparisonOp::LE: + return VariantComparisonOp::GE; + case VariantComparisonOp::GT: + return VariantComparisonOp::LT; + case VariantComparisonOp::GE: + return VariantComparisonOp::LE; + } + __builtin_unreachable(); +} + +std::optional> variant_literal(const VExprSPtr& expr) { + const auto literal = std::dynamic_pointer_cast(expr); + if (literal == nullptr || !literal->get_column_ptr() || literal->get_column_ptr()->empty()) { + return std::nullopt; + } + Field value; + literal->get_column_ptr()->get(0, value); + if (value.is_null()) { + return std::nullopt; + } + return std::make_pair(std::move(value), literal->get_data_type()); +} + +std::optional extract_variant_shredded_predicate( + const VExprContextSPtr& conjunct) { + if (conjunct == nullptr || conjunct->root() == nullptr || + conjunct->root()->get_num_children() != 2) { + return std::nullopt; + } + auto op = variant_comparison_op(callable_name(conjunct->root())); + if (!op.has_value()) { + return std::nullopt; + } + + VExprSPtr value_expr; + std::optional> literal; + if ((literal = variant_literal(conjunct->root()->get_child(1))).has_value()) { + value_expr = conjunct->root()->get_child(0); + } else if ((literal = variant_literal(conjunct->root()->get_child(0))).has_value()) { + value_expr = conjunct->root()->get_child(1); + op = reverse_variant_comparison(*op); + } else { + return std::nullopt; + } + + const auto comparison_type = value_expr->data_type(); + while (value_expr->node_type() == TExprNodeType::CAST_EXPR && + value_expr->get_num_children() == 1) { + if (!expr_zonemap::data_types_compatible(value_expr->data_type(), comparison_type)) { + // Every removed cast must preserve the comparison domain. Otherwise bounds for the + // raw typed leaf could skip rows whose value changes in an intermediate narrowing cast. + return std::nullopt; + } + value_expr = value_expr->get_child(0); + } + + std::vector reverse_path; + while (callable_name(value_expr) == "element_at" && value_expr->get_num_children() == 2) { + const auto key = variant_literal(value_expr->get_child(1)); + if (!key.has_value() || key->first.get_type() != TYPE_STRING) { + // Repeated array shredding has no single scalar page range, so only object keys are + // eligible for this file-level optimization. + return std::nullopt; + } + reverse_path.push_back(key->first.get()); + value_expr = value_expr->get_child(0); + } + const auto slot = std::dynamic_pointer_cast(value_expr); + if (slot == nullptr || reverse_path.empty() || comparison_type == nullptr || + remove_nullable(slot->data_type())->get_primitive_type() != TYPE_VARIANT || + !expr_zonemap::data_types_compatible(comparison_type, literal->second)) { + return std::nullopt; + } + std::ranges::reverse(reverse_path); + return VariantShreddedPredicate {.slot_index = slot->column_id(), + .path = std::move(reverse_path), + .comparison_type = comparison_type, + .literal_type = literal->second, + .literal = std::move(literal->first), + .op = *op}; +} + +VExprContextSPtrs metadata_pruning_conjuncts(const format::FileScanRequest& request) { + const size_t safe_count = + std::min(request.metadata_pruning_safe_conjunct_count, request.conjuncts.size()); + return VExprContextSPtrs(request.conjuncts.begin(), request.conjuncts.begin() + safe_count); +} + +bool has_variant_shredded_filter(const format::FileScanRequest& request) { + const auto conjuncts = metadata_pruning_conjuncts(request); + return std::ranges::any_of(conjuncts, [](const auto& conjunct) { + return extract_variant_shredded_predicate(conjunct).has_value(); + }); +} + +const ParquetColumnSchema* child_named(const ParquetColumnSchema& parent, std::string_view name) { + const auto it = std::ranges::find_if(parent.children, [&](const auto& child) { + return child != nullptr && child->name == name; + }); + return it == parent.children.end() ? nullptr : it->get(); +} + +struct ResolvedVariantShredding { + const ParquetColumnSchema* fallback_value = nullptr; + const ParquetColumnSchema* typed_value = nullptr; +}; + +bool metadata_cast_is_order_preserving(const DataTypePtr& source, const DataTypePtr& target) { + if (expr_zonemap::data_types_compatible(source, target)) { + return true; + } + const auto source_type = remove_nullable(source); + const auto target_type = remove_nullable(target); + const auto source_primitive = source_type->get_primitive_type(); + const auto target_primitive = target_type->get_primitive_type(); + // Metadata bounds may cross only exact widening domains. This mirrors the residual CAST while + // excluding rounding, overflow, and narrowing cases that could reverse a pruning decision. + if (source_primitive == TYPE_FLOAT && target_primitive == TYPE_DOUBLE) { + return true; + } + if (is_int(source_primitive) && source_primitive != TYPE_LARGEINT && + is_decimalv3(target_primitive)) { + const uint32_t required_integer_digits = source_primitive == TYPE_TINYINT ? 3 + : source_primitive == TYPE_SMALLINT ? 5 + : source_primitive == TYPE_INT ? 10 + : 19; + return target_type->get_precision() >= target_type->get_scale() && + target_type->get_precision() - target_type->get_scale() >= required_integer_digits; + } + if (is_decimalv3(source_primitive) && is_decimalv3(target_primitive)) { + const uint32_t source_integer_digits = + source_type->get_precision() - source_type->get_scale(); + const uint32_t target_integer_digits = + target_type->get_precision() - target_type->get_scale(); + return target_integer_digits >= source_integer_digits && + target_type->get_scale() >= source_type->get_scale(); + } + return false; +} + +std::optional cast_metadata_field(const Field& value, const DataTypePtr& source, + const DataTypePtr& target) { + if (expr_zonemap::data_types_compatible(source, target)) { + return value; + } + const auto source_type = remove_nullable(source); + const auto target_type = remove_nullable(target); + if (source_type->get_primitive_type() == TYPE_FLOAT && + target_type->get_primitive_type() == TYPE_DOUBLE) { + return Field::create_field(static_cast(value.get())); + } + try { + auto source_column = source_type->create_column(); + source_column->insert(value); + DataTypeSerDe::FormatOptions options = DataTypeSerDe::get_default_format_options(); + options.converted_from_string = true; + std::string text = source_type->to_string(*source_column, 0, options); + StringRef input(text.data(), text.size()); + auto target_column = target_type->create_column(); + if (!target_type->get_serde() + ->from_string_strict_mode(input, *target_column, options) + .ok() || + target_column->size() != 1) { + return std::nullopt; + } + Field result; + target_column->get(0, result); + return result; + } catch (...) { + return std::nullopt; + } +} + +std::optional normalize_variant_statistics( + const VariantShreddedPredicate& predicate, const ParquetColumnSchema& typed_value, + const ParquetColumnStatistics& statistics) { + if (!statistics.has_min_max || + expr_zonemap::data_types_compatible(typed_value.type, predicate.comparison_type)) { + return statistics; + } + auto min_value = + cast_metadata_field(statistics.min_value, typed_value.type, predicate.comparison_type); + auto max_value = + cast_metadata_field(statistics.max_value, typed_value.type, predicate.comparison_type); + if (!min_value.has_value() || !max_value.has_value()) { + return std::nullopt; + } + auto normalized = statistics; + normalized.min_value = std::move(*min_value); + normalized.max_value = std::move(*max_value); + return normalized; +} + +std::optional resolve_variant_shredding( + const std::vector>& file_schema, + const format::FileScanRequest& request, const VariantShreddedPredicate& predicate) { + const auto local_id = file_column_id_by_block_position(request, predicate.slot_index); + if (!local_id.has_value() || local_id->value() < 0 || + local_id->value() >= static_cast(file_schema.size())) { + return std::nullopt; + } + const ParquetColumnSchema* wrapper = file_schema[local_id->value()].get(); + if (wrapper == nullptr || wrapper->kind != ParquetColumnSchemaKind::VARIANT) { + return std::nullopt; + } + for (const auto& component : predicate.path) { + const auto* typed_object = child_named(*wrapper, "typed_value"); + if (typed_object == nullptr || typed_object->kind != ParquetColumnSchemaKind::STRUCT) { + return std::nullopt; + } + wrapper = child_named(*typed_object, component); + if (wrapper == nullptr || wrapper->kind != ParquetColumnSchemaKind::STRUCT) { + return std::nullopt; + } + } + const auto* fallback = child_named(*wrapper, "value"); + const auto* typed = child_named(*wrapper, "typed_value"); + const auto typed_primitive = typed == nullptr || typed->type == nullptr + ? INVALID_TYPE + : remove_nullable(typed->type)->get_primitive_type(); + if (fallback == nullptr || typed == nullptr || + fallback->kind != ParquetColumnSchemaKind::PRIMITIVE || + typed->kind != ParquetColumnSchemaKind::PRIMITIVE || typed->max_repetition_level != 0 || + // Parquet float statistics do not prove that a page contains no NaN. Min/max pruning in + // the presence of NaN is not order preserving, so keep those pages until such proof exists. + typed_primitive == TYPE_FLOAT || typed_primitive == TYPE_DOUBLE || + !metadata_cast_is_order_preserving(typed->type, predicate.comparison_type) || + !expr_zonemap::data_types_compatible(predicate.comparison_type, predicate.literal_type)) { + return std::nullopt; + } + return ResolvedVariantShredding {.fallback_value = fallback, .typed_value = typed}; +} + +bool fallback_is_all_null(const tparquet::RowGroup& row_group, + const ParquetColumnSchema& fallback) { + if (fallback.max_repetition_level != 0 || fallback.leaf_column_id < 0 || + fallback.leaf_column_id >= static_cast(row_group.columns.size())) { + return false; + } + const auto& chunk = row_group.columns[fallback.leaf_column_id]; + return row_group.num_rows >= 0 && chunk.__isset.meta_data && + chunk.meta_data.num_values == row_group.num_rows && chunk.meta_data.__isset.statistics && + chunk.meta_data.statistics.__isset.null_count && + chunk.meta_data.statistics.null_count == chunk.meta_data.num_values; +} + +bool variant_statistics_exclude(const VariantShreddedPredicate& predicate, + const ParquetColumnStatistics& statistics) { + if (!statistics.has_any_statistics()) { + return false; + } + if (!statistics.has_not_null) { + return true; + } + if (!statistics.has_min_max) { + return false; + } + const auto& literal = predicate.literal; + switch (predicate.op) { + case VariantComparisonOp::EQ: + return literal < statistics.min_value || statistics.max_value < literal; + case VariantComparisonOp::NE: + return statistics.min_value == literal && statistics.max_value == literal; + case VariantComparisonOp::LT: + return statistics.min_value >= literal; + case VariantComparisonOp::LE: + return statistics.min_value > literal; + case VariantComparisonOp::GT: + return statistics.max_value <= literal; + case VariantComparisonOp::GE: + return statistics.max_value < literal; + } + __builtin_unreachable(); +} + bool has_expr_zonemap_filter(const format::FileScanRequest& request, const RuntimeState*) { // FileScannerV2 metadata pruning is a fixed part of its scan pipeline and must not inherit // the legacy scanner's expression ZoneMap session gate. - // TODO: Fence metadata pruning at the first unsafe/error-preserving conjunct so a later - // ZoneMap predicate cannot bypass its row-level evaluation. - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { if (conjunct != nullptr && conjunct->root() != nullptr && conjunct->root()->can_evaluate_zonemap_filter()) { return true; } } - return false; + return has_variant_shredded_filter(request); } std::set collect_expr_zonemap_slot_indexes(const VExprContextSPtrs& conjuncts) { @@ -617,9 +941,11 @@ void collect_filtered_leaf_ids(const ParquetColumnSchema& column_schema, if (!format::is_child_projected(projection, child_schema->local_id)) { continue; } - collect_filtered_leaf_ids(*child_schema, - format::find_child_projection(projection, child_schema->local_id), - leaf_column_ids); + // The leaf set must match the physical projection. A complete Variant projection naturally + // reaches every sibling; a validated typed-leaf projection reads only retained children. + const auto* child_projection = + format::find_child_projection(projection, child_schema->local_id); + collect_filtered_leaf_ids(*child_schema, child_projection, leaf_column_ids); } } @@ -627,7 +953,21 @@ bool native_metadata_predicate_is_type_safe(const ParquetColumnSchema& column_sc DORIS_CHECK(column_schema.type != nullptr); // Raw VARBINARY file slots may feed table-side STRING casts. Footer/page metadata is still in // the pre-cast domain, so using it for a rewritten table predicate can cause false negatives. - return remove_nullable(column_schema.type)->get_primitive_type() != TYPE_VARBINARY; + if (remove_nullable(column_schema.type)->get_primitive_type() == TYPE_VARBINARY) { + return false; + } + // UUID readers render canonical text, so their physical 16-byte bounds are not STRING bounds. + return !column_schema.type_descriptor.is_uuid; +} + +bool variant_metadata_predicate_is_type_safe(const ParquetColumnSchema& column_schema) { + if (!native_metadata_predicate_is_type_safe(column_schema)) { + return false; + } + const auto& descriptor = column_schema.type_descriptor; + // An ordinary raw-binary STRING slot preserves its bytes, but Variant reconstruction renders + // the binary identity before the residual STRING cast and therefore changes the domain. + return !descriptor.is_string_like || descriptor.is_string_annotation; } bool check_native_statistics(const tparquet::FileMetaData& metadata, @@ -635,7 +975,8 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata, const std::vector>& file_schema, const format::FileScanRequest& request, ParquetPruningStats* pruning_stats, const cctz::time_zone* timezone) { - const auto slot_indexes = collect_expr_zonemap_slot_indexes(request.conjuncts); + const auto conjuncts = metadata_pruning_conjuncts(request); + const auto slot_indexes = collect_expr_zonemap_slot_indexes(conjuncts); if (slot_indexes.empty()) { return false; } @@ -670,11 +1011,54 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata, } add_slot_zonemap(&ctx, slot_index, column_schema->type, std::move(zone_map)); } - const auto result = VExprContext::evaluate_zonemap_filter(request.conjuncts, ctx); + const auto result = VExprContext::evaluate_zonemap_filter(conjuncts, ctx); accumulate_zonemap_stats(ctx, pruning_stats); return result == ZoneMapFilterResult::kNoMatch; } +bool check_shredded_variant_statistics( + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, + const std::vector>& file_schema, + const format::FileScanRequest& request, const cctz::time_zone* timezone) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { + const auto predicate = extract_variant_shredded_predicate(conjunct); + if (!predicate.has_value()) { + continue; + } + const auto shredding = resolve_variant_shredding(file_schema, request, *predicate); + if (!shredding.has_value() || shredding->typed_value->leaf_column_id < 0 || + shredding->typed_value->leaf_column_id >= static_cast(row_group.columns.size()) || + !fallback_is_all_null(row_group, *shredding->fallback_value) || + !variant_metadata_predicate_is_type_safe(*shredding->typed_value) || + !detail::has_supported_type_defined_order(metadata, + shredding->typed_value->leaf_column_id)) { + continue; + } + const auto& chunk = row_group.columns[shredding->typed_value->leaf_column_id]; + if (!chunk.__isset.meta_data) { + continue; + } + const auto& column_metadata = chunk.meta_data; + if (column_metadata.num_values != row_group.num_rows) { + continue; + } + std::optional safe_statistics; + if (column_metadata.__isset.statistics) { + safe_statistics = detail::sanitize_native_footer_statistics( + shredding->typed_value->type_descriptor, column_metadata.statistics, true); + } + const auto statistics = ParquetStatisticsUtils::TransformColumnStatistics( + *shredding->typed_value, safe_statistics.has_value() ? &*safe_statistics : nullptr, + column_metadata.num_values, timezone); + const auto normalized = + normalize_variant_statistics(*predicate, *shredding->typed_value, statistics); + if (normalized.has_value() && variant_statistics_exclude(*predicate, *normalized)) { + return true; + } + } + return false; +} + bool is_native_dictionary_data_encoding(tparquet::Encoding::type encoding) { return encoding == tparquet::Encoding::PLAIN_DICTIONARY || encoding == tparquet::Encoding::RLE_DICTIONARY; @@ -738,7 +1122,7 @@ ParquetRowGroupPruneReason native_dictionary_prune_reason( return ParquetRowGroupPruneReason::NONE; } const auto conjuncts_by_slot = collect_conjuncts_by_single_slot( - request.conjuncts, expr_zonemap::single_slot_dictionary_index); + metadata_pruning_conjuncts(request), expr_zonemap::single_slot_dictionary_index); for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { const auto file_column_id = file_column_id_by_block_position(request, slot_index); if (!file_column_id.has_value()) { @@ -807,7 +1191,7 @@ ParquetRowGroupPruneReason native_bloom_filter_prune_reason( return ParquetRowGroupPruneReason::NONE; } const auto conjuncts_by_slot = collect_conjuncts_by_single_slot( - request.conjuncts, expr_zonemap::single_slot_bloom_filter_index); + metadata_pruning_conjuncts(request), expr_zonemap::single_slot_bloom_filter_index); for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { const auto file_column_id = file_column_id_by_block_position(request, slot_index); if (!file_column_id.has_value()) { @@ -899,6 +1283,12 @@ Status select_row_groups_by_metadata( if (pruning_stats != nullptr) { pruning_stats->total_row_groups = cast_set(candidate_size); } + const bool contains_variant = + file_context != nullptr ? file_context->contains_variant + : std::ranges::any_of(file_schema, [](const auto& column) { + DORIS_CHECK(column != nullptr); + return column->contains_variant; + }); selected_row_groups->reserve(candidate_size); for (size_t candidate_idx = 0; candidate_idx < candidate_size; ++candidate_idx) { const int row_group_idx = candidate_row_groups == nullptr @@ -923,8 +1313,10 @@ Status select_row_groups_by_metadata( ParquetRowGroupPruneReason prune_reason = ParquetRowGroupPruneReason::NONE; if (probe_mode != ParquetMetadataProbeMode::EXPENSIVE_ONLY && has_expr_zonemap_filter(request, runtime_state) && - check_native_statistics(metadata, row_group, file_schema, request, pruning_stats, - timezone)) { + (check_native_statistics(metadata, row_group, file_schema, request, pruning_stats, + timezone) || + (contains_variant && check_shredded_variant_statistics( + metadata, row_group, file_schema, request, timezone)))) { prune_reason = ParquetRowGroupPruneReason::STATISTICS; } if (probe_mode != ParquetMetadataProbeMode::FOOTER_ONLY && @@ -984,6 +1376,38 @@ std::vector intersect_ranges(const std::vector& left, return result; } +std::vector union_ranges(const std::vector& left, + const std::vector& right) { + std::vector result; + result.reserve(left.size() + right.size()); + auto append = [&](const RowRange& range) { + if (range.length == 0) { + return; + } + if (!result.empty()) { + auto& previous = result.back(); + const int64_t previous_end = previous.start + previous.length; + if (range.start <= previous_end) { + previous.length = + std::max(previous_end, range.start + range.length) - previous.start; + return; + } + } + result.push_back(range); + }; + size_t left_idx = 0; + size_t right_idx = 0; + while (left_idx < left.size() || right_idx < right.size()) { + if (right_idx == right.size() || + (left_idx < left.size() && left[left_idx].start <= right[right_idx].start)) { + append(left[left_idx++]); + } else { + append(right[right_idx++]); + } + } + return result; +} + int64_t count_range_rows(const std::vector& ranges) { int64_t rows = 0; for (const auto& range : ranges) { @@ -1029,11 +1453,16 @@ void collect_leaf_schemas(const ParquetColumnSchema& column_schema, return; } for (const auto& child_schema : column_schema.children) { - if (!format::is_child_projected(projection, child_schema->local_id)) { + if (column_schema.kind != ParquetColumnSchemaKind::VARIANT && + !format::is_child_projected(projection, child_schema->local_id)) { continue; } + // A logical Variant projection materializes every physical sibling; build skip plans for + // that identical leaf set so shredded columns cannot drift to different row positions. const auto* child_projection = - format::find_child_projection(projection, child_schema->local_id); + column_schema.kind == ParquetColumnSchemaKind::VARIANT + ? nullptr + : format::find_child_projection(projection, child_schema->local_id); collect_leaf_schemas(*child_schema, child_projection, leaf_schemas); } } @@ -1235,10 +1664,158 @@ RowRange native_page_row_range(const tparquet::OffsetIndex& offset_index, size_t return {.start = start, .length = end - start}; } +class NativePageIndexPredicateEvaluator { +public: + NativePageIndexPredicateEvaluator( + const tparquet::FileMetaData& metadata, + const std::unordered_map& page_indexes, + const std::vector>& file_schema, + const format::FileScanRequest& request, int64_t row_group_rows, + ParquetPruningStats* pruning_stats, const cctz::time_zone* timezone) + : _metadata(metadata), + _page_indexes(page_indexes), + _file_schema(file_schema), + _request(request), + _row_group_rows(row_group_rows), + _pruning_stats(pruning_stats), + _timezone(timezone) {} + + std::optional> evaluate(const VExprSPtr& expr) const { + if (expr == nullptr || !expr->can_evaluate_zonemap_filter()) { + return std::nullopt; + } + if (expr->op() == TExprOpcode::COMPOUND_AND) { + return evaluate_compound(expr, true); + } + if (expr->op() == TExprOpcode::COMPOUND_OR) { + return evaluate_compound(expr, false); + } + return evaluate_leaf(expr); + } + +private: + struct SlotPageZoneMaps { + DataTypePtr data_type; + std::vector ranges; + std::vector> zone_maps; + }; + + std::optional> evaluate_compound(const VExprSPtr& expr, + bool is_and) const { + std::optional> ranges; + for (const auto& child : expr->children()) { + if (!child->can_evaluate_zonemap_filter()) { + if (!is_and) { + return std::nullopt; + } + continue; + } + auto child_ranges = evaluate(child); + if (!child_ranges.has_value()) { + // An unavailable AND child can be ignored, while an unavailable OR branch must + // retain the complete range so metadata pruning cannot create a false negative. + if (!is_and) { + return std::nullopt; + } + continue; + } + if (!ranges.has_value()) { + ranges = std::move(*child_ranges); + } else if (is_and) { + ranges = intersect_ranges(*ranges, *child_ranges); + } else { + ranges = union_ranges(*ranges, *child_ranges); + } + if (is_and && ranges->empty()) { + return ranges; + } + } + return ranges; + } + + std::optional> evaluate_leaf(const VExprSPtr& expr) const { + std::set slot_indexes; + expr->collect_slot_column_ids(slot_indexes); + if (slot_indexes.size() != 1) { + return std::nullopt; + } + const int slot_index = *slot_indexes.begin(); + const auto* pages = load_slot_pages(slot_index); + if (pages == nullptr) { + return std::nullopt; + } + + std::vector ranges; + for (size_t page_idx = 0; page_idx < pages->ranges.size(); ++page_idx) { + ZoneMapEvalContext ctx; + add_slot_zonemap(&ctx, slot_index, pages->data_type, pages->zone_maps[page_idx]); + if (expr->evaluate_zonemap_filter(ctx) != ZoneMapFilterResult::kNoMatch) { + append_row_range(pages->ranges[page_idx], &ranges); + } + accumulate_zonemap_stats(ctx, _pruning_stats); + } + return ranges; + } + + const SlotPageZoneMaps* load_slot_pages(int slot_index) const { + const auto cached = _slot_page_zone_maps.find(slot_index); + if (cached != _slot_page_zone_maps.end()) { + return cached->second.has_value() ? &*cached->second : nullptr; + } + const auto file_column_id = file_column_id_by_block_position(_request, slot_index); + if (!file_column_id.has_value()) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + const auto* column_schema = resolve_local_leaf_schema(_file_schema, *file_column_id); + if (column_schema == nullptr || column_schema->type == nullptr || + !native_metadata_predicate_is_type_safe(*column_schema) || + !detail::has_supported_type_defined_order(_metadata, column_schema->leaf_column_id)) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + const auto index_it = _page_indexes.find(column_schema->leaf_column_id); + if (index_it == _page_indexes.end()) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + + const auto& indexes = index_it->second; + SlotPageZoneMaps pages; + pages.data_type = column_schema->type; + pages.ranges.reserve(indexes.offset_index.page_locations.size()); + pages.zone_maps.reserve(indexes.offset_index.page_locations.size()); + for (size_t page_idx = 0; page_idx < indexes.offset_index.page_locations.size(); + ++page_idx) { + const auto page_range = + native_page_row_range(indexes.offset_index, page_idx, _row_group_rows); + ParquetColumnStatistics statistics; + if (!build_native_page_statistics(indexes.column_index, *column_schema, page_idx, + page_range.length, &statistics, _timezone)) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + pages.ranges.push_back(page_range); + pages.zone_maps.push_back(ParquetStatisticsUtils::MakeZoneMap(statistics)); + } + const auto inserted = _slot_page_zone_maps.emplace(slot_index, std::move(pages)); + return &*inserted.first->second; + } + + const tparquet::FileMetaData& _metadata; + const std::unordered_map& _page_indexes; + const std::vector>& _file_schema; + const format::FileScanRequest& _request; + int64_t _row_group_rows; + ParquetPruningStats* _pruning_stats; + const cctz::time_zone* _timezone; + mutable std::unordered_map> _slot_page_zone_maps; +}; + } // namespace Status select_row_group_ranges_by_native_page_index( - const tparquet::FileMetaData& metadata, + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, const std::unordered_map& page_indexes, const std::vector>& file_schema, const format::FileScanRequest& request, int64_t row_group_rows, @@ -1263,10 +1840,15 @@ Status select_row_group_ranges_by_native_page_index( } std::map conjuncts_by_slot; - for (const auto& conjunct : request.conjuncts) { + VExprContextSPtrs multi_slot_conjuncts; + // Compound predicates must honor the same metadata-pruning fence as single-slot predicates. + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto slot_index = expr_zonemap::single_slot_zonemap_index(conjunct); if (slot_index >= 0) { conjuncts_by_slot[slot_index].push_back(conjunct); + } else if (conjunct != nullptr && conjunct->root() != nullptr && + conjunct->root()->can_evaluate_zonemap_filter()) { + multi_slot_conjuncts.push_back(conjunct); } } for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { @@ -1304,11 +1886,76 @@ Status select_row_group_ranges_by_native_page_index( ZoneMapFilterResult::kNoMatch) { append_row_range(page_range, &filter_ranges); } + accumulate_zonemap_stats(ctx, pruning_stats); + } + if (!usable) { + continue; + } + *selected_ranges = intersect_ranges(*selected_ranges, filter_ranges); + if (selected_ranges->empty()) { + if (pruning_stats != nullptr) { + pruning_stats->filtered_page_rows += row_group_rows; + ++pruning_stats->filtered_row_groups_by_page_index; + } + return Status::OK(); + } + } + + NativePageIndexPredicateEvaluator evaluator(metadata, page_indexes, file_schema, request, + row_group_rows, pruning_stats, timezone); + for (const auto& conjunct : multi_slot_conjuncts) { + auto conjunct_ranges = evaluator.evaluate(conjunct->root()); + if (!conjunct_ranges.has_value()) { + continue; + } + *selected_ranges = intersect_ranges(*selected_ranges, *conjunct_ranges); + if (selected_ranges->empty()) { if (pruning_stats != nullptr) { - pruning_stats->expr_zonemap_unusable_evals += ctx.stats.unusable_zonemap_eval_count; - pruning_stats->in_zonemap_point_check_count += - ctx.stats.in_zonemap_point_check_count; - pruning_stats->in_zonemap_range_only_count += ctx.stats.in_zonemap_range_only_count; + pruning_stats->filtered_page_rows += row_group_rows; + ++pruning_stats->filtered_row_groups_by_page_index; + } + return Status::OK(); + } + } + + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { + const auto predicate = extract_variant_shredded_predicate(conjunct); + if (!predicate.has_value()) { + continue; + } + const auto shredding = resolve_variant_shredding(file_schema, request, *predicate); + if (!shredding.has_value() || shredding->typed_value->leaf_column_id < 0 || + !fallback_is_all_null(row_group, *shredding->fallback_value) || + !variant_metadata_predicate_is_type_safe(*shredding->typed_value) || + !detail::has_supported_type_defined_order(metadata, + shredding->typed_value->leaf_column_id)) { + continue; + } + const auto index_it = page_indexes.find(shredding->typed_value->leaf_column_id); + if (index_it == page_indexes.end()) { + continue; + } + const auto& indexes = index_it->second; + std::vector filter_ranges; + bool usable = true; + for (size_t page_idx = 0; page_idx < indexes.offset_index.page_locations.size(); + ++page_idx) { + const auto page_range = + native_page_row_range(indexes.offset_index, page_idx, row_group_rows); + ParquetColumnStatistics statistics; + if (!build_native_page_statistics(indexes.column_index, *shredding->typed_value, + page_idx, page_range.length, &statistics, timezone)) { + usable = false; + break; + } + const auto normalized = + normalize_variant_statistics(*predicate, *shredding->typed_value, statistics); + if (!normalized.has_value()) { + usable = false; + break; + } + if (!variant_statistics_exclude(*predicate, *normalized)) { + append_row_range(page_range, &filter_ranges); } } if (!usable) { diff --git a/be/src/format_v2/parquet/parquet_statistics.h b/be/src/format_v2/parquet/parquet_statistics.h index 72381548656f9d..611ec5abe97ac2 100644 --- a/be/src/format_v2/parquet/parquet_statistics.h +++ b/be/src/format_v2/parquet/parquet_statistics.h @@ -145,7 +145,7 @@ Status select_row_groups_by_metadata( ParquetMetadataProbeMode probe_mode = ParquetMetadataProbeMode::ALL); Status select_row_group_ranges_by_native_page_index( - const tparquet::FileMetaData& metadata, + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, const std::unordered_map& page_indexes, const std::vector>& file_schema, const format::FileScanRequest& request, int64_t row_group_rows, diff --git a/be/src/format_v2/parquet/parquet_type.h b/be/src/format_v2/parquet/parquet_type.h index 7ab3d2b3b39d8d..f06f4c25dd8ba2 100644 --- a/be/src/format_v2/parquet/parquet_type.h +++ b/be/src/format_v2/parquet/parquet_type.h @@ -67,6 +67,8 @@ struct ParquetTypeDescriptor { bool is_timestamp = false; // whether this is a timestamp type bool timestamp_is_adjusted_to_utc = false; // whether the timestamp is UTC-normalized bool is_string_like = false; // binary type that is neither decimal nor FLOAT16 + bool is_string_annotation = false; // STRING logical type, distinct from raw BINARY + bool is_uuid = false; // UUID logical type over FIXED_LEN_BYTE_ARRAY(16) std::string unsupported_reason; // non-empty when this Parquet logical type is unsupported }; diff --git a/be/src/format_v2/parquet/reader/count_column_reader.cpp b/be/src/format_v2/parquet/reader/count_column_reader.cpp index 171107107f1a38..8be684d7235957 100644 --- a/be/src/format_v2/parquet/reader/count_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/count_column_reader.cpp @@ -69,6 +69,17 @@ Status find_count_leaf(const ParquetColumnSchema& schema, // reading a potentially huge value BYTE_ARRAY for COUNT(map_col). DORIS_CHECK(!schema.children.empty()); return find_count_leaf(*schema.children.front(), nullptr, leaf); + case ParquetColumnSchemaKind::VARIANT: { + // The required metadata leaf is present exactly when the enclosing Variant group is + // present, so its levels preserve COUNT(variant_col) SQL-null semantics without decoding + // any Variant payload. + const auto metadata = std::ranges::find_if( + schema.children, [](const auto& child) { return child->name == "metadata"; }); + if (metadata == schema.children.end()) { + return Status::Corruption("Parquet Variant {} has no metadata column", schema.name); + } + return find_count_leaf(**metadata, nullptr, leaf); + } } return Status::InternalError("Unknown Parquet schema kind for column {}", schema.name); } diff --git a/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp b/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp index f3c3af5ba9944b..d1677c666762c5 100644 --- a/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp +++ b/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp @@ -681,6 +681,12 @@ void remap_nullable_conversion_failures(IColumn::Filter* conversion_failure_null DORIS_CHECK_EQ(source, 0); } +Status decode_prepared_nullable_values(IColumn& column, const DataTypeSerDe& serde, + Decoder& decoder, const ParquetDecodeContext& context, + ParquetMaterializationState& state, + const NullMap& selected_nulls, + int64_t* materialization_time); + Status decode_selected_nullable_values(IColumn& column, const DataTypeSerDe& serde, Decoder& decoder, const ParquetDecodeContext& context, ParquetMaterializationState& state, @@ -722,6 +728,16 @@ Status decode_selected_nullable_values(IColumn& column, const DataTypeSerDe& ser DORIS_CHECK_EQ(selection.total_values, select_vector.num_values() - select_vector.num_nulls()); DORIS_CHECK_EQ(selected_nulls.size(), select_vector.num_values() - select_vector.num_filtered()); + return decode_prepared_nullable_values(column, serde, decoder, context, state, selected_nulls, + materialization_time); +} + +Status decode_prepared_nullable_values(IColumn& column, const DataTypeSerDe& serde, + Decoder& decoder, const ParquetDecodeContext& context, + ParquetMaterializationState& state, + const NullMap& selected_nulls, + int64_t* materialization_time) { + auto& selection = state.selection; const size_t old_size = column.size(); SCOPED_RAW_TIMER(materialization_time); @@ -1735,6 +1751,49 @@ Status ColumnChunkReader::materialize_values( return Status::OK(); } +template +bool ColumnChunkReader::supports_fused_nullable_selection( + IColumn& column) const { + return visit_nullable_expandable_column(column, [](auto&) {}); +} + +template +Status ColumnChunkReader::materialize_fused_nullable_values( + MutableColumnPtr& doris_column, const DataTypeSerDe& serde, ParquetDecodeContext& context, + ParquetMaterializationState& state, size_t num_values, size_t num_nulls, + const NullMap& selected_nulls) { + if (num_values == 0) { + return Status::OK(); + } + SCOPED_RAW_TIMER(&_chunk_statistics.decode_value_time); + DORIS_CHECK_GT(num_nulls, 0); + const size_t physical_values = num_values - num_nulls; + DORIS_CHECK_EQ(state.selection.total_values, physical_values); + DORIS_CHECK_LE(state.selection.selected_values, selected_nulls.size()); + if (UNLIKELY(_empty_value_section && physical_values != 0)) { + return Status::Corruption( + "Parquet definition levels require {} values from an empty value section", + physical_values); + } + if (UNLIKELY((doris_column->is_column_dictionary() || context.dictionary_index_only) && + !_has_dict && physical_values != 0)) { + return Status::IOError("Not dictionary coded"); + } + if (UNLIKELY(_remaining_num_values < num_values)) { + return Status::IOError("Decode too many values in current page"); + } + RETURN_IF_ERROR(translate_value_encoding(_current_encoding, &context.encoding)); + + ++_chunk_statistics.hybrid_selection_batches; + const auto status = decode_prepared_nullable_values(*doris_column, serde, *_page_decoder, + context, state, selected_nulls, + &_chunk_statistics.materialization_time); + _chunk_statistics.hybrid_selection_ranges += state.selection.ranges.size(); + RETURN_IF_ERROR(status); + _remaining_num_values -= num_values; + return Status::OK(); +} + template bool ColumnChunkReader::can_filter_fixed_width_values( const VExprSPtrs& conjuncts, int column_id, const DataTypeSerDe* serde, diff --git a/be/src/format_v2/parquet/reader/native/column_chunk_reader.h b/be/src/format_v2/parquet/reader/native/column_chunk_reader.h index 35bf5200d7aebb..a50b7e4bc080ff 100644 --- a/be/src/format_v2/parquet/reader/native/column_chunk_reader.h +++ b/be/src/format_v2/parquet/reader/native/column_chunk_reader.h @@ -177,6 +177,14 @@ class ColumnChunkReader { ParquetDecodeContext& context, ParquetMaterializationState& state, ColumnSelectVector& select_vector); + bool supports_fused_nullable_selection(IColumn& column) const; + + Status materialize_fused_nullable_values(MutableColumnPtr& doris_column, + const DataTypeSerDe& serde, + ParquetDecodeContext& context, + ParquetMaterializationState& state, size_t num_values, + size_t num_nulls, const NullMap& selected_nulls); + static bool supports_raw_fixed_filter_encoding(tparquet::Encoding::type encoding, tparquet::Type::type physical_type) { switch (encoding) { diff --git a/be/src/format_v2/parquet/reader/native/column_reader.cpp b/be/src/format_v2/parquet/reader/native/column_reader.cpp index f762cb0f9d35f5..b5ea931b29fdeb 100644 --- a/be/src/format_v2/parquet/reader/native/column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native/column_reader.cpp @@ -799,6 +799,8 @@ void ScalarColumnReader::release_batch_scratch( max_retained_bytes); release_selection |= release_vector_if_oversized(&_materialization_state.selection.ranges, max_retained_bytes); + release_selection |= + release_filter_if_oversized(&_fused_nullable_selection_nulls, max_retained_bytes); release_selection |= release_filter_if_oversized(&_fixed_width_predicate_nulls, max_retained_bytes); release_selection |= @@ -832,6 +834,7 @@ void ScalarColumnReader::release_batch_scratch( release_selection |= release_vector_for_aggregate(&_nested_filter_map_data); release_selection |= release_vector_for_aggregate(&_materialization_state.dictionary_indices); release_selection |= release_vector_for_aggregate(&_materialization_state.selection.ranges); + release_selection |= release_filter_for_aggregate(&_fused_nullable_selection_nulls); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_nulls); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_matches); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_conversion_nulls); @@ -860,7 +863,8 @@ size_t ScalarColumnReader::retained_batch_scratch_b _def_levels.capacity() * sizeof(level_t) + _null_run_lengths.capacity() * sizeof(uint16_t) + _nested_filter_map_data.capacity() * sizeof(uint8_t) + - _fixed_width_predicate_nulls.capacity() + _fixed_width_predicate_matches.capacity() + + _fused_nullable_selection_nulls.capacity() + _fixed_width_predicate_nulls.capacity() + + _fixed_width_predicate_matches.capacity() + _fixed_width_predicate_conversion_nulls.capacity() + _materialization_state.dictionary_indices.capacity() * sizeof(uint32_t) + _materialization_state.selection.ranges.capacity() * sizeof(ParquetSelectionRange) + @@ -875,7 +879,8 @@ size_t ScalarColumnReader::active_batch_scratch_byt _serde == nullptr ? 0 : _serde->active_parquet_raw_predicate_scratch_bytes(); return decoder_bytes + serde_bytes + _rep_levels.size() * sizeof(level_t) + _def_levels.size() * sizeof(level_t) + _null_run_lengths.size() * sizeof(uint16_t) + - _nested_filter_map_data.size() * sizeof(uint8_t) + _fixed_width_predicate_nulls.size() + + _nested_filter_map_data.size() * sizeof(uint8_t) + + _fused_nullable_selection_nulls.size() + _fixed_width_predicate_nulls.size() + _fixed_width_predicate_matches.size() + _fixed_width_predicate_conversion_nulls.size() + _materialization_state.dictionary_indices.size() * sizeof(uint32_t) + _materialization_state.selection.ranges.size() * sizeof(ParquetSelectionRange) + @@ -892,6 +897,7 @@ void ScalarColumnReader::reserve_batch_scratch_for_ _nested_filter_map_data.reserve(elements); _materialization_state.dictionary_indices.reserve(elements); _materialization_state.selection.ranges.reserve(elements); + _fused_nullable_selection_nulls.reserve(elements); _ancestor_null_indices.reserve(elements); } @@ -955,6 +961,7 @@ Status ScalarColumnReader::_read_values(size_t num_ } MutableColumnPtr data_column; _null_run_lengths.clear(); + size_t num_nulls = 0; NullMap* map_data_column = nullptr; doris_column = IColumn::mutate(std::move(doris_column)); if (is_column_nullable(*doris_column)) { @@ -977,6 +984,9 @@ Status ScalarColumnReader::_read_values(size_t num_ } bool is_null = def_level < _field_schema->definition_level; + if (is_null) { + num_nulls += loop_read; + } if (!(prev_is_null ^ is_null)) { _null_run_lengths.emplace_back(0); } @@ -1006,10 +1016,26 @@ Status ScalarColumnReader::_read_values(size_t num_ } _null_run_lengths.emplace_back((u_short)remaining); } + const bool use_fused_nullable_selection = + map_data_column != nullptr && filter_map.has_filter() && num_nulls > 0 && + should_use_fused_nullable_selection(num_values, num_nulls, _null_run_lengths.size()) && + _chunk_reader->supports_fused_nullable_selection(*data_column); { SCOPED_RAW_TIMER(&_decode_null_map_time); - RETURN_IF_ERROR(_select_vector.init(_null_run_lengths, num_values, map_data_column, - &filter_map, _filter_map_index)); + if (use_fused_nullable_selection) { + size_t num_filtered = 0; + // The fused path owns both the physical ranges and selected NULL layout. Restrict it + // to fragmented, materially nullable level plans: clustered, low-NULL, and no-NULL + // pages already collapse into a few cheap legacy runs, while fusing them adds planning + // branches without removing enough work to guarantee a win. + RETURN_IF_ERROR(build_filtered_nullable_selection( + _null_run_lengths, num_values, num_nulls, map_data_column, &filter_map, + _filter_map_index, &_materialization_state.selection, + &_fused_nullable_selection_nulls, &num_filtered)); + } else { + RETURN_IF_ERROR(_select_vector.init(_null_run_lengths, num_values, map_data_column, + &filter_map, _filter_map_index)); + } _filter_map_index += num_values; } DORIS_CHECK(_serde != nullptr); @@ -1020,8 +1046,13 @@ Status ScalarColumnReader::_read_values(size_t num_ conversion_failure_map(*_field_schema, type, _materialization_state.enable_strict_mode, map_data_column, &compatibility_scratch); const size_t materialization_start_row = data_column->size(); - const auto status = _chunk_reader->materialize_values(data_column, *_serde, _decode_context, - _materialization_state, _select_vector); + const auto status = + use_fused_nullable_selection + ? _chunk_reader->materialize_fused_nullable_values( + data_column, *_serde, _decode_context, _materialization_state, + num_values, num_nulls, _fused_nullable_selection_nulls) + : _chunk_reader->materialize_values(data_column, *_serde, _decode_context, + _materialization_state, _select_vector); _materialization_state.conversion_failure_null_map = nullptr; if (status.ok()) { mark_local_timestamp_defaults(*_field_schema, type, diff --git a/be/src/format_v2/parquet/reader/native/column_reader.h b/be/src/format_v2/parquet/reader/native/column_reader.h index 18bcdd866f83d7..594f0ac0921523 100644 --- a/be/src/format_v2/parquet/reader/native/column_reader.h +++ b/be/src/format_v2/parquet/reader/native/column_reader.h @@ -432,6 +432,7 @@ class ScalarColumnReader : public ColumnReader { std::vector _null_run_lengths; std::unordered_set _ancestor_null_indices; std::vector _nested_filter_map_data; + NullMap _fused_nullable_selection_nulls; NullMap _fixed_width_predicate_nulls; IColumn::Filter _fixed_width_predicate_matches; IColumn::Filter _fixed_width_predicate_conversion_nulls; diff --git a/be/src/format_v2/parquet/reader/native/common.cpp b/be/src/format_v2/parquet/reader/native/common.cpp index c50488575e4993..020f613c5d267f 100644 --- a/be/src/format_v2/parquet/reader/native/common.cpp +++ b/be/src/format_v2/parquet/reader/native/common.cpp @@ -17,6 +17,7 @@ #include "format_v2/parquet/reader/native/common.h" +#include #include #include "core/types.h" @@ -68,6 +69,133 @@ bool FilterMap::can_filter_all(size_t remaining_num_values, size_t filter_map_in remaining_num_values) == remaining_num_values; } +bool should_use_fused_nullable_selection(size_t num_values, size_t num_nulls, + size_t num_null_runs) { + constexpr size_t MIN_BATCH_VALUES = 1024; + constexpr size_t MIN_NULL_RUNS = 32; + constexpr size_t MAX_AVERAGE_NULL_RUN = 64; + constexpr size_t MIN_NULL_RATIO_DENOMINATOR = 10; + if (num_values < MIN_BATCH_VALUES || num_nulls < num_values / MIN_NULL_RATIO_DENOMINATOR) { + return false; + } + return num_null_runs >= std::max(MIN_NULL_RUNS, num_values / MAX_AVERAGE_NULL_RUN); +} + +Status build_filtered_nullable_selection(const std::vector& run_length_null_map, + size_t num_values, size_t num_nulls, + NullMap* output_null_map, FilterMap* filter_map, + size_t filter_map_index, ParquetSelection* selection, + NullMap* selected_nulls, size_t* num_filtered) { + if (output_null_map == nullptr || filter_map == nullptr || selection == nullptr || + selected_nulls == nullptr || num_filtered == nullptr) { + return Status::InvalidArgument( + "Nullable selection planning requires non-null output state"); + } + if (!filter_map->has_filter()) { + return Status::InvalidArgument("Nullable selection planning requires a row filter"); + } + if (!filter_map->filter_all() && + (filter_map->filter_map_data() == nullptr || + filter_map_index + num_values > filter_map->filter_map_size())) { + return Status::InvalidArgument("Nullable selection filter range [{}, {}) exceeds size {}", + filter_map_index, filter_map_index + num_values, + filter_map->filter_map_size()); + } + if (num_nulls > num_values) { + return Status::InvalidArgument("Nullable selection has {} nulls for {} values", num_nulls, + num_values); + } + + selection->ranges.clear(); + selection->total_values = num_values - num_nulls; + selection->selected_values = 0; + selected_nulls->clear(); + *num_filtered = 0; + if (filter_map->filter_all()) { + *num_filtered = num_values; + return Status::OK(); + } + + selected_nulls->reserve(num_values); + const uint8_t* filter = filter_map->filter_map_data() + filter_map_index; + const auto select_physical_values = [&](size_t physical_index, size_t count) { + if (!selection->ranges.empty() && + selection->ranges.back().first + selection->ranges.back().count == physical_index) { + selection->ranges.back().count += count; + } else { + selection->ranges.push_back({.first = physical_index, .count = count}); + } + selection->selected_values += count; + }; + + if (num_nulls == 0) { + size_t row = 0; + while (row < num_values) { + const bool selected = filter[row] != 0; + const size_t run_start = row++; + while (row < num_values && (filter[row] != 0) == selected) { + ++row; + } + const size_t run_length = row - run_start; + if (selected) { + select_physical_values(run_start, run_length); + } else { + *num_filtered += run_length; + } + } + selected_nulls->resize_fill(selection->selected_values, 0); + } else { + size_t logical_index = 0; + size_t physical_index = 0; + size_t observed_nulls = 0; + bool is_null = false; + for (const size_t run_length : run_length_null_map) { + if (logical_index + run_length > num_values) { + return Status::InvalidArgument("Nullable selection run lengths exceed {} values", + num_values); + } + const size_t run_end = logical_index + run_length; + while (logical_index < run_end) { + const bool selected = filter[logical_index] != 0; + const size_t filter_run_start = logical_index++; + while (logical_index < run_end && (filter[logical_index] != 0) == selected) { + ++logical_index; + } + const size_t filter_run_length = logical_index - filter_run_start; + if (selected) { + selected_nulls->resize_fill(selected_nulls->size() + filter_run_length, + static_cast(is_null)); + if (!is_null) { + select_physical_values(physical_index, filter_run_length); + } + } else { + *num_filtered += filter_run_length; + } + if (!is_null) { + physical_index += filter_run_length; + } else { + observed_nulls += filter_run_length; + } + } + is_null = !is_null; + } + if (logical_index != num_values || observed_nulls != num_nulls || + physical_index != selection->total_values) { + return Status::InvalidArgument( + "Nullable selection level plan is inconsistent: values={}, nulls={}", + logical_index, observed_nulls); + } + } + + const size_t old_null_size = output_null_map->size(); + output_null_map->resize(old_null_size + selected_nulls->size()); + if (!selected_nulls->empty()) { + memcpy(output_null_map->data() + old_null_size, selected_nulls->data(), + selected_nulls->size()); + } + return Status::OK(); +} + Status FilterMap::generate_nested_filter_map(const std::vector& rep_levels, std::vector& nested_filter_map_data, std::unique_ptr* nested_filter_map, diff --git a/be/src/format_v2/parquet/reader/native/common.h b/be/src/format_v2/parquet/reader/native/common.h index eb6848ee299f10..bd687616f226ce 100644 --- a/be/src/format_v2/parquet/reader/native/common.h +++ b/be/src/format_v2/parquet/reader/native/common.h @@ -25,6 +25,7 @@ #include "common/status.h" #include "core/column/column_nullable.h" +#include "core/data_type_serde/parquet_decode_source.h" namespace doris::format::parquet::native { @@ -116,4 +117,14 @@ class ColumnSelectVector { size_t _read_index = 0; }; +Status build_filtered_nullable_selection(const std::vector& run_length_null_map, + size_t num_values, size_t num_nulls, + NullMap* output_null_map, FilterMap* filter_map, + size_t filter_map_index, ParquetSelection* selection, + NullMap* selected_nulls, size_t* num_filtered); + +// Fusion pays for its additional planning branches only when definition levels are materially +// nullable and fragmented. Keep compact/no-NULL batches on the run-oriented legacy path. +bool should_use_fused_nullable_selection(size_t num_values, size_t num_nulls, size_t num_null_runs); + } // namespace doris::format::parquet::native diff --git a/be/src/format_v2/parquet/reader/native_column_reader.cpp b/be/src/format_v2/parquet/reader/native_column_reader.cpp index 6b686597fa600e..273964e5d1f451 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native_column_reader.cpp @@ -42,6 +42,7 @@ #include "format_v2/column_data.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_file_context.h" +#include "format_v2/parquet/reader/variant_column_reader.h" #include "runtime/runtime_state.h" namespace doris::format::parquet { @@ -50,25 +51,57 @@ namespace { constexpr size_t MAX_RETAINED_BATCH_SCRATCH_BYTES = 4UL << 20; DataTypePtr projected_type(const ParquetColumnSchema& schema, - const format::LocalColumnIndex* projection) { - if (!format::is_partial_projection(projection)) { - return schema.type; - } + const format::LocalColumnIndex* projection, bool physical_variant) { switch (schema.kind) { case ParquetColumnSchemaKind::PRIMITIVE: return schema.type; + case ParquetColumnSchemaKind::VARIANT: + DORIS_CHECK(schema.variant_physical_type != nullptr); + if (!physical_variant || !format::is_partial_projection(projection)) { + return physical_variant ? schema.variant_physical_type : schema.type; + } + { + DataTypes child_types; + Strings child_names; + child_types.reserve(projection->children.size()); + child_names.reserve(projection->children.size()); + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + child_types.push_back(make_nullable( + projected_type(**child_it, &child_projection, physical_variant))); + child_names.push_back((*child_it)->name); + } + DataTypePtr type = std::make_shared(std::move(child_types), + std::move(child_names)); + return schema.variant_physical_type->is_nullable() ? make_nullable(std::move(type)) + : std::move(type); + } case ParquetColumnSchemaKind::STRUCT: { DataTypes child_types; Strings child_names; - child_types.reserve(projection->children.size()); - child_names.reserve(projection->children.size()); - for (const auto& child_projection : projection->children) { - const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { - return child->local_id == child_projection.local_id(); - }); - DORIS_CHECK(child_it != schema.children.end()); - child_types.push_back(make_nullable(projected_type(**child_it, &child_projection))); - child_names.push_back((*child_it)->name); + if (format::is_partial_projection(projection)) { + child_types.reserve(projection->children.size()); + child_names.reserve(projection->children.size()); + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + child_types.push_back(make_nullable( + projected_type(**child_it, &child_projection, physical_variant))); + child_names.push_back((*child_it)->name); + } + } else { + child_types.reserve(schema.children.size()); + child_names.reserve(schema.children.size()); + for (const auto& child : schema.children) { + child_types.push_back( + make_nullable(projected_type(*child, nullptr, physical_variant))); + child_names.push_back(child->name); + } } DataTypePtr type = std::make_shared(child_types, child_names); return schema.type->is_nullable() ? make_nullable(type) : type; @@ -76,20 +109,25 @@ DataTypePtr projected_type(const ParquetColumnSchema& schema, case ParquetColumnSchemaKind::LIST: { DORIS_CHECK(schema.children.size() == 1); const auto* child_projection = - format::find_child_projection(projection, schema.children[0]->local_id); - DORIS_CHECK(child_projection != nullptr); + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[0]->local_id) + : nullptr; + DORIS_CHECK(!format::is_partial_projection(projection) || child_projection != nullptr); DataTypePtr type = std::make_shared( - projected_type(*schema.children[0], child_projection)); + projected_type(*schema.children[0], child_projection, physical_variant)); return schema.type->is_nullable() ? make_nullable(type) : type; } case ParquetColumnSchemaKind::MAP: { DORIS_CHECK(schema.children.size() == 2); const auto* value_projection = - format::find_child_projection(projection, schema.children[1]->local_id); - DORIS_CHECK(value_projection != nullptr); + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[1]->local_id) + : nullptr; + DORIS_CHECK(!format::is_partial_projection(projection) || value_projection != nullptr); DataTypePtr type = std::make_shared( - make_nullable(schema.children[0]->type), - make_nullable(projected_type(*schema.children[1], value_projection))); + make_nullable(projected_type(*schema.children[0], nullptr, physical_variant)), + make_nullable( + projected_type(*schema.children[1], value_projection, physical_variant))); return schema.type->is_nullable() ? make_nullable(type) : type; } } @@ -97,6 +135,64 @@ DataTypePtr projected_type(const ParquetColumnSchema& schema, return nullptr; } +std::unique_ptr build_variant_plan( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection) { + auto plan = std::make_unique(); + plan->schema = &schema; + if (schema.kind == ParquetColumnSchemaKind::VARIANT) { + plan->contains_variant = true; + if (projection != nullptr) { + plan->variant_projection = *projection; + } + plan->variant_state_schema = create_variant_state_schema(schema, projection); + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::PRIMITIVE) { + return plan; + } + + auto append_child = [&](const ParquetColumnSchema& child, + const format::LocalColumnIndex* child_projection) { + auto child_plan = build_variant_plan(child, child_projection); + plan->contains_variant = plan->contains_variant || child_plan->contains_variant; + plan->children.push_back(std::move(child_plan)); + }; + if (schema.kind == ParquetColumnSchemaKind::STRUCT && + format::is_partial_projection(projection)) { + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + append_child(**child_it, &child_projection); + } + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::LIST) { + DORIS_CHECK(schema.children.size() == 1); + const auto* child_projection = + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[0]->local_id) + : nullptr; + append_child(*schema.children[0], child_projection); + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::MAP) { + DORIS_CHECK(schema.children.size() == 2); + append_child(*schema.children[0], nullptr); + const auto* value_projection = + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[1]->local_id) + : nullptr; + append_child(*schema.children[1], value_projection); + return plan; + } + for (const auto& child : schema.children) { + append_child(*child, nullptr); + } + return plan; +} + const NativeFieldSchema* find_child_field(const NativeFieldSchema& parent, const ParquetColumnSchema& child) { auto field_it = std::ranges::find_if(parent.children, [&](const NativeFieldSchema& field) { @@ -149,10 +245,13 @@ void collect_projected_ids(const ParquetColumnSchema& schema, } // namespace -NativeColumnReader::NativeColumnReader(const ParquetColumnSchema& schema, - DataTypePtr projected_type, +NativeColumnReader::NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr logical_type, + DataTypePtr native_type, + std::unique_ptr variant_plan, ParquetColumnReaderProfile profile) - : ParquetColumnReader(schema, std::move(projected_type), profile), + : ParquetColumnReader(schema, std::move(logical_type), profile), + _native_type(std::move(native_type)), + _variant_plan(std::move(variant_plan)), _nested(schema.kind != ParquetColumnSchemaKind::PRIMITIVE) {} NativeColumnReader::~NativeColumnReader() { @@ -192,14 +291,22 @@ Status NativeColumnReader::create( column_schema.local_id, field->name, column_schema.name); } - auto type = projected_type(column_schema, projection); + auto logical_type = projected_type(column_schema, projection, false); + auto native_type = logical_type; + std::unique_ptr variant_plan; + if (column_schema.contains_variant) { + // Native readers are instantiated per projected column and row group. Keep Variant tree + // construction out of ordinary scans instead of charging that setup cost at every split. + native_type = projected_type(column_schema, projection, true); + variant_plan = build_variant_plan(column_schema, projection); + } std::shared_ptr schema_node; - RETURN_IF_ERROR(build_native_schema_node(type, column_schema, &schema_node)); + RETURN_IF_ERROR(build_native_schema_node(native_type, column_schema, &schema_node)); std::set projected_ids; collect_projected_ids(column_schema, projection, *field, &projected_ids); - auto native_reader = std::unique_ptr( - new NativeColumnReader(column_schema, std::move(type), profile)); + new NativeColumnReader(column_schema, std::move(logical_type), std::move(native_type), + std::move(variant_plan), profile)); RETURN_IF_ERROR(native_reader->init( std::move(file), metadata, row_group_id, field, std::move(schema_node), std::move(projected_ids), selected_ranges, offset_indexes, timezone, io_ctx, @@ -261,7 +368,10 @@ Status NativeColumnReader::init( _filter_column_ids, page_cache_file_key, compat, runtime_state != nullptr && runtime_state->enable_strict_mode())); DORIS_CHECK(_native_reader != nullptr); - _skip_column = _type->create_column(); + _skip_column = _native_type->create_column(); + if (_variant_plan != nullptr) { + _variant_physical_column = _native_type->create_column(); + } return Status::OK(); } @@ -281,15 +391,22 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ native::FilterMap filter; RETURN_IF_ERROR(filter.init(filter_data, static_cast(rows), filter_all)); _native_reader->reset_filter_map_index(); - ColumnPtr native_column(std::move(column)); + const bool materialize_variant = + !dictionary_ids && _variant_plan != nullptr && output_type->equals(*_type); + if (materialize_variant) { + _variant_physical_column->clear(); + } + ColumnPtr native_column = materialize_variant ? ColumnPtr(std::move(_variant_physical_column)) + : ColumnPtr(std::move(column)); bool eof = false; int64_t native_calls = 0; int64_t consecutive_empty_calls = 0; while (*rows_read < rows && !eof) { ++native_calls; size_t loop_rows = 0; + const DataTypePtr& decoder_type = materialize_variant ? _native_type : output_type; RETURN_IF_ERROR(_native_reader->read_column_data( - native_column, output_type, _schema_node, filter, + native_column, decoder_type, _schema_node, filter, static_cast(rows - *rows_read), &loop_rows, &eof, dictionary_ids)); if (loop_rows == 0 && !eof) { // A selected RowRanges plan may reject the current data page completely. V1 advances @@ -297,7 +414,11 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ // next page. Bound consecutive empty transitions by the Row Group row count to retain // a deterministic corruption exit if a decoder ever stops advancing. if (++consecutive_empty_calls > _row_group_rows + 1) { - column = IColumn::mutate(std::move(native_column)); + if (materialize_variant) { + _variant_physical_column = IColumn::mutate(std::move(native_column)); + } else { + column = IColumn::mutate(std::move(native_column)); + } return Status::Corruption("Native parquet reader made no progress for column {}", _name); } @@ -306,7 +427,20 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ consecutive_empty_calls = 0; *rows_read += static_cast(loop_rows); } - column = IColumn::mutate(std::move(native_column)); + if (materialize_variant) { + if (*rows_read != rows) { + _variant_physical_column = IColumn::mutate(std::move(native_column)); + return Status::Corruption("Native parquet reader returned {} rows, expected {} for {}", + *rows_read, rows, _name); + } + // The shredded state owns this decoded batch. Replace scanner scratch before handing the + // pointer off so typed path expressions can retain its physical leaves without a copy. + _variant_physical_column = _native_type->create_column(); + RETURN_IF_ERROR(materialize_variant_columns(*_variant_plan, std::move(native_column), + column, _profile)); + } else { + column = IColumn::mutate(std::move(native_column)); + } if (_profile.native_read_calls != nullptr) { COUNTER_UPDATE(_profile.native_read_calls, native_calls); } @@ -545,7 +679,7 @@ Status NativeColumnReader::skip(int64_t rows) { _filter_scratch.assign(static_cast(selected_rows), 0); int64_t rows_read = 0; RETURN_IF_ERROR(read_with_filter(selected_rows, _filter_scratch.data(), true, _skip_column, - _type, false, &rows_read)); + _native_type, false, &rows_read)); DORIS_CHECK(_skip_column->empty()); DORIS_CHECK(rows_read == selected_rows); _logical_row_position += rows_read; @@ -586,6 +720,11 @@ Status NativeColumnReader::select_with_dictionary_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(survivor_count != nullptr); DORIS_CHECK(used_filter != nullptr); + if (_variant_plan != nullptr) { + row_filter->clear(); + *used_filter = false; + return Status::OK(); + } RETURN_IF_ERROR(validate_selected_span(batch_rows)); *used_filter = false; *survivor_count = 0; @@ -737,6 +876,13 @@ Status NativeColumnReader::select_with_fixed_width_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(used_filter != nullptr); DORIS_CHECK(execution_kind != nullptr); + if (_variant_plan != nullptr) { + // Direct fixed-width evaluation cannot preserve a Variant physical subtree's row shape. + row_filter->clear(); + *used_filter = false; + *execution_kind = DirectPredicateExecutionKind::NONE; + return Status::OK(); + } RETURN_IF_ERROR(validate_selected_span(batch_rows)); const uint8_t* filter_data = nullptr; RETURN_IF_ERROR(selection.materialize_filter(selected_rows, batch_rows, &filter_data)); @@ -867,6 +1013,10 @@ bool NativeColumnReader::crossed_page_since_last_batch() { Result NativeColumnReader::dictionary_values() { DORIS_CHECK(_native_reader != nullptr); + if (_variant_plan != nullptr) { + return ResultError( + Status::NotSupported("Parquet Variant columns do not expose dictionary values")); + } return _native_reader->dictionary_values(_type); } diff --git a/be/src/format_v2/parquet/reader/native_column_reader.h b/be/src/format_v2/parquet/reader/native_column_reader.h index f7dfcb3e696be8..150ab4352d00c6 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.h +++ b/be/src/format_v2/parquet/reader/native_column_reader.h @@ -43,6 +43,7 @@ struct IOContext; namespace doris::format::parquet { class NativeParquetMetadata; +struct VariantMaterializationNode; namespace detail { inline constexpr int64_t MAX_NATIVE_LAZY_SKIP_ROWS = std::numeric_limits::max(); @@ -102,7 +103,9 @@ class NativeColumnReader final : public ParquetColumnReader { Result dictionary_values() override; private: - NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr projected_type, + NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr logical_type, + DataTypePtr native_type, + std::unique_ptr variant_plan, ParquetColumnReaderProfile profile); Status init(io::FileReaderSPtr file, const NativeParquetMetadata* metadata, int row_group_id, @@ -142,6 +145,9 @@ class NativeColumnReader final : public ParquetColumnReader { const std::unordered_map* _offset_indexes = nullptr; std::shared_ptr _schema_node; std::unique_ptr _native_reader; + DataTypePtr _native_type; + std::unique_ptr _variant_plan; + MutableColumnPtr _variant_physical_column; std::unique_ptr _page_cache_runtime_state; std::vector _selected_ranges; size_t _selected_range_idx = 0; diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp new file mode 100644 index 00000000000000..da5ecbb8310a20 --- /dev/null +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -0,0 +1,1170 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/parquet/reader/variant_column_reader.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common/exception.h" +#include "core/assert_cast.h" +#include "core/column/column_array.h" +#include "core/column/column_decimal.h" +#include "core/column/column_map.h" +#include "core/column/column_nullable.h" +#include "core/column/column_struct.h" +#include "core/column/column_vector.h" +#include "core/column/variant_v2/column_variant_v2.h" +#include "core/column/variant_v2/column_variant_v2_typed_column.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_variant_v2.h" +#include "core/value/variant/variant_batch_builder.h" +#include "core/value/variant/variant_metadata.h" +#include "format_v2/parquet/parquet_column_schema.h" + +namespace doris::format::parquet { +namespace { + +struct Cell { + const IColumn* column = nullptr; + bool is_null = false; +}; + +Cell cell_at(const IColumn& column, size_t row) { + if (row >= column.size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant row {} exceeds column size {}", row, + column.size()); + } + if (const auto* nullable = check_and_get_column(column)) { + return {.column = &nullable->get_nested_column(), + .is_null = nullable->get_null_map_data()[row] != 0}; + } + return {.column = &column, .is_null = false}; +} + +const ParquetColumnSchema* find_child(const ParquetColumnSchema& schema, std::string_view name, + size_t* index) { + for (size_t i = 0; i < schema.children.size(); ++i) { + if (schema.children[i]->name == name) { + if (index != nullptr) { + *index = i; + } + return schema.children[i].get(); + } + } + return nullptr; +} + +Cell struct_child_at(const ParquetColumnSchema& schema, const IColumn& physical, size_t row, + std::string_view name, const ParquetColumnSchema** child_schema) { + const auto& structure = assert_cast(physical); + size_t index = 0; + const auto* child = find_child(schema, name, &index); + if (child == nullptr || index >= structure.tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant {} has no physical child {}", + schema.name, name); + } + if (child_schema != nullptr) { + *child_schema = child; + } + return cell_at(structure.get_column(index), row); +} + +uint8_t decimal_width(int precision) { + if (precision <= 0 || precision > 38) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant decimal precision {} is outside [1, 38]", precision); + } + return precision <= 9 ? 4 : (precision <= 18 ? 8 : 16); +} + +uint8_t integer_width(const ParquetColumnSchema& schema, PrimitiveType type) { + if (schema.type_descriptor.is_unsigned_integer) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Unsigned integers are not valid Parquet Variant typed values"); + } + if (schema.type_descriptor.integer_bit_width > 0) { + switch (schema.type_descriptor.integer_bit_width) { + case 8: + return 1; + case 16: + return 2; + case 32: + return 4; + case 64: + return 8; + default: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant integer width {}", + schema.type_descriptor.integer_bit_width); + } + } + switch (type) { + case TYPE_TINYINT: + return 1; + case TYPE_SMALLINT: + return 2; + case TYPE_INT: + return 4; + case TYPE_BIGINT: + return 8; + default: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant integer type {}", type); + } +} + +void append_typed_scalar(const ParquetColumnSchema& schema, const IColumn& column, size_t row, + VariantBatchBuilder::Row& builder) { + const PrimitiveType type = remove_nullable(schema.type)->get_primitive_type(); + switch (type) { + case TYPE_BOOLEAN: + builder.add_bool(assert_cast(column).get_data()[row] != 0); + return; + case TYPE_TINYINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_SMALLINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_INT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_BIGINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_FLOAT: + builder.add_float(assert_cast(column).get_data()[row]); + return; + case TYPE_DOUBLE: + builder.add_double(assert_cast(column).get_data()[row]); + return; + case TYPE_DECIMAL128I: { + const auto value = assert_cast(column).get_data()[row].value; + builder.add_decimal(value, static_cast(schema.type_descriptor.decimal_scale), + decimal_width(schema.type_descriptor.decimal_precision)); + return; + } + case TYPE_TIMEV2: { + const double seconds = assert_cast(column).get_data()[row]; + if (!std::isfinite(seconds) || + std::abs(seconds) > static_cast(std::numeric_limits::max()) / 1e6) { + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant TIME value"); + } + builder.add_time_ntz_micros(static_cast(std::llround(seconds * 1e6))); + return; + } + case TYPE_DATETIMEV2: { + if (schema.type_descriptor.time_unit == ParquetTimeUnit::NANOS) { + // Native DATETIMEV2 is microsecond based. Reject before returning a silently truncated + // value; a raw INT64 nanos decoder can be added independently. + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + const auto& value = assert_cast(column).get_data()[row]; + builder.add_timestamp_micros( + variant_timestamp_micros(value, row, "Parquet Variant TIMESTAMP"), + schema.type_descriptor.timestamp_is_adjusted_to_utc); + return; + } + case TYPE_TIMESTAMPTZ: { + if (schema.type_descriptor.time_unit == ParquetTimeUnit::NANOS) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + const auto& value = assert_cast(column).get_data()[row]; + builder.add_timestamp_micros( + variant_timestamp_micros(value, row, "Parquet Variant TIMESTAMP"), true); + return; + } + case TYPE_VARBINARY: { + const StringRef value = column.get_data_at(row); + if (!schema.type_descriptor.is_uuid) { + builder.add_binary(value); + return; + } + if (value.size != 16) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant UUID has {} bytes instead of 16", value.size); + } + std::array uuid {}; + std::memcpy(uuid.data(), value.data, uuid.size()); + builder.add_uuid(uuid); + return; + } + case TYPE_STRING: { + const StringRef value = column.get_data_at(row); + if (schema.type_descriptor.is_uuid) { + if (value.size != 16) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant UUID has {} bytes instead of 16", value.size); + } + std::array uuid {}; + std::memcpy(uuid.data(), value.data, uuid.size()); + builder.add_uuid(uuid); + } else if (schema.type_descriptor.is_string_annotation) { + builder.add_string(value); + } else { + builder.add_binary(value); + } + return; + } + default: + if (!is_supported_variant_typed_identity(type)) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant typed value {} is not supported", + remove_nullable(schema.type)->get_name()); + } + dispatch_variant_typed_column( + column, type, [&](const auto& typed_column) { + with_variant_typed_scalar( + typed_column, row, + static_cast(remove_nullable(schema.type)->get_scale()), + [&](const VariantScalarRef& scalar) { builder.add_scalar(scalar); }); + }); + } +} + +enum class WrapperContext { ROOT, ARRAY_ELEMENT, OBJECT_FIELD }; + +bool append_wrapper(const ParquetColumnSchema& schema, const IColumn& wrapper, size_t row, + VariantMetadataRef metadata, VariantBatchBuilder::Row& builder, + WrapperContext context); + +void append_typed_value(const ParquetColumnSchema& schema, const IColumn& column, size_t row, + VariantMetadataRef metadata, const VariantRef* residual, + VariantBatchBuilder::Row& builder) { + switch (schema.kind) { + case ParquetColumnSchemaKind::PRIMITIVE: + if (static_cast(residual)) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant scalar typed_value cannot have residual value bytes"); + } + append_typed_scalar(schema, column, row, builder); + return; + case ParquetColumnSchemaKind::STRUCT: { + if (static_cast(residual) && residual->basic_type() != VariantBasicType::OBJECT) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant object typed_value has non-object residual value"); + } + const auto& structure = assert_cast(column); + if (structure.tuple_size() != schema.children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant object {} physical field count mismatch", schema.name); + } + auto object = builder.start_object(); + if (static_cast(residual)) { + for (uint32_t i = 0; i < residual->num_elements(); ++i) { + uint32_t field_id = 0; + const VariantRef child = residual->object_value_at(i, &field_id); + object.add_key(residual->metadata.key_at(field_id)); + builder.add_value(child); + } + } + for (size_t i = 0; i < schema.children.size(); ++i) { + const auto& child_schema = *schema.children[i]; + const Cell child = cell_at(structure.get_column(i), row); + if (child.is_null) { + // Shredded object fields are optional wrapper groups. A missing group means the + // key is absent, which differs from a present wrapper encoding a Variant null. + continue; + } + // A null/null wrapper means this object field is absent. Delay add_key until its + // presence is known so absent shredded fields do not turn into Variant nulls. + size_t value_index = 0; + const auto* value_schema = find_child(child_schema, "value", &value_index); + const auto& child_struct = assert_cast(*child.column); + const bool value_present = value_schema != nullptr && + !cell_at(child_struct.get_column(value_index), row).is_null; + size_t typed_index = 0; + const auto* typed_schema = find_child(child_schema, "typed_value", &typed_index); + const bool typed_present = typed_schema != nullptr && + !cell_at(child_struct.get_column(typed_index), row).is_null; + if (!value_present && !typed_present) { + continue; + } + object.add_key(StringRef(child_schema.name)); + (void)append_wrapper(child_schema, *child.column, row, metadata, builder, + WrapperContext::OBJECT_FIELD); + } + object.finish(); + return; + } + case ParquetColumnSchemaKind::LIST: { + if (static_cast(residual)) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant array typed_value cannot have residual value bytes"); + } + if (schema.children.size() != 1) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant array {} has invalid element schema", schema.name); + } + const auto& array = assert_cast(column); + const size_t begin = array.offset_at(static_cast(row)); + const size_t end = array.get_offsets()[row]; + auto scope = builder.start_array(); + for (size_t element = begin; element < end; ++element) { + const Cell cell = cell_at(array.get_data(), element); + if (cell.is_null) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant shredded array element wrapper is null"); + } + (void)append_wrapper(*schema.children[0], *cell.column, element, metadata, builder, + WrapperContext::ARRAY_ELEMENT); + } + scope.finish(); + return; + } + case ParquetColumnSchemaKind::MAP: + case ParquetColumnSchemaKind::VARIANT: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant typed_value schema {}", + schema.name); + } +} + +bool append_wrapper(const ParquetColumnSchema& schema, const IColumn& wrapper, size_t row, + VariantMetadataRef metadata, VariantBatchBuilder::Row& builder, + WrapperContext context) { + Cell value; + if (find_child(schema, "value", nullptr) != nullptr) { + value = struct_child_at(schema, wrapper, row, "value", nullptr); + } else { + value.is_null = true; + } + const ParquetColumnSchema* typed_schema = nullptr; + Cell typed; + if (find_child(schema, "typed_value", nullptr) != nullptr) { + typed = struct_child_at(schema, wrapper, row, "typed_value", &typed_schema); + } else { + typed.is_null = true; + } + + if (find_child(schema, "value", nullptr) == nullptr && typed_schema == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant wrapper {} has neither value nor typed_value", + schema.name); + } + if (value.is_null && typed.is_null) { + if (context == WrapperContext::OBJECT_FIELD) { + return false; + } + if (context == WrapperContext::ARRAY_ELEMENT) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant array element is missing"); + } + builder.add_null(); + return true; + } + + VariantRef residual {.metadata = metadata, .value = {}}; + if (!value.is_null) { + residual.value = value.column->get_data_at(row); + } + if (typed.is_null) { + builder.add_value(residual); + return true; + } + append_typed_value(*typed_schema, *typed.column, row, metadata, + value.is_null ? nullptr : &residual, builder); + return true; +} + +void encode_variant_range(const ParquetColumnSchema& schema, const IColumn& wrapper, + const ColumnNullable* outer_nullable, size_t begin, size_t end, + ColumnVariantV2& variants) { + try { + VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = end - begin}); + for (size_t row = begin; row < end; ++row) { + auto output_row = builder.begin_row(); + if (outer_nullable != nullptr && outer_nullable->get_null_map_data()[row] != 0) { + output_row.add_null(); + output_row.finish(); + continue; + } + const Cell metadata_cell = struct_child_at(schema, wrapper, row, "metadata", nullptr); + if (metadata_cell.is_null) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant {} has null metadata at row {}", schema.name, row); + } + const StringRef metadata_bytes = metadata_cell.column->get_data_at(row); + const VariantMetadataRef metadata {metadata_bytes.data, metadata_bytes.size}; + metadata.validate(); + (void)append_wrapper(schema, wrapper, row, metadata, output_row, WrapperContext::ROOT); + output_row.finish(); + } + VariantBatchBuilder batch = builder.finish_batch(); + variants.insert_encoded_batch(batch); + } catch (...) { + if (end - begin <= 1) { + throw; + } + // A single builder has one metadata dictionary. If heterogeneous file rows cannot fit in + // that dictionary, split without changing the destination column's already-valid batches. + // Corrupt input still reaches a one-row range and propagates its original exception. + const size_t middle = begin + (end - begin) / 2; + encode_variant_range(schema, wrapper, outer_nullable, begin, middle, variants); + encode_variant_range(schema, wrapper, outer_nullable, middle, end, variants); + } +} + +ColumnVariantV2::MutablePtr encode_variant_column(const ParquetColumnSchema& schema, + const IColumn& physical) { + if (schema.kind != ParquetColumnSchemaKind::VARIANT) { + throw Exception(ErrorCode::INVALID_ARGUMENT, "Parquet column {} is not Variant", + schema.name); + } + const auto* outer_nullable = check_and_get_column(physical); + const IColumn& wrapper = + outer_nullable == nullptr ? physical : outer_nullable->get_nested_column(); + const auto& structure = assert_cast(wrapper); + if (structure.tuple_size() != schema.children.size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant {} physical field count mismatch", + schema.name); + } + + auto variants = ColumnVariantV2::create(); + constexpr size_t MAX_RECONSTRUCTION_BATCH_ROWS = 4096; + for (size_t begin = 0; begin < physical.size(); begin += MAX_RECONSTRUCTION_BATCH_ROWS) { + encode_variant_range(schema, wrapper, outer_nullable, begin, + std::min(physical.size(), begin + MAX_RECONSTRUCTION_BATCH_ROWS), + *variants); + } + return variants; +} + +std::unique_ptr clone_schema( + const ParquetColumnSchema& source, const format::LocalColumnIndex* projection = nullptr) { + auto result = std::make_unique(); + result->local_id = source.local_id; + result->parquet_field_id = source.parquet_field_id; + result->name = source.name; + result->type = source.type; + result->variant_physical_type = source.variant_physical_type; + result->leaf_column_id = source.leaf_column_id; + result->type_descriptor = source.type_descriptor; + result->kind = source.kind; + result->contains_variant = source.contains_variant; + result->max_definition_level = source.max_definition_level; + result->max_repetition_level = source.max_repetition_level; + result->nullable_definition_level = source.nullable_definition_level; + result->definition_level = source.definition_level; + result->repetition_level = source.repetition_level; + result->repeated_ancestor_definition_level = source.repeated_ancestor_definition_level; + result->repeated_repetition_level = source.repeated_repetition_level; + const bool partial = format::is_partial_projection(projection); + result->children.reserve(partial ? projection->children.size() : source.children.size()); + if (partial) { + // NativeColumnReader emits a partial STRUCT in projection order, so the retained schema + // must use that same order or field names will address the wrong physical tuple element. + for (const auto& child_projection : projection->children) { + const auto child = std::ranges::find_if(source.children, [&](const auto& candidate) { + return candidate->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child != source.children.end()); + result->children.push_back(clone_schema(**child, &child_projection)); + } + } else { + for (const auto& child : source.children) { + result->children.push_back(clone_schema(*child)); + } + } + return result; +} + +ColumnPtr unwrap_nullable(ColumnPtr column) { + if (const auto* nullable = check_and_get_column(*column)) { + return nullable->get_nested_column_ptr(); + } + return column; +} + +ColumnPtr struct_child(const ParquetColumnSchema& schema, ColumnPtr column, std::string_view name, + const ParquetColumnSchema** child_schema) { + column = unwrap_nullable(std::move(column)); + const auto* structure = check_and_get_column(*column); + if (structure == nullptr) { + return nullptr; + } + size_t index = 0; + const auto* child = find_child(schema, name, &index); + if (child == nullptr || index >= structure->tuple_size()) { + return nullptr; + } + if (child_schema != nullptr) { + *child_schema = child; + } + return structure->get_column_ptr(index); +} + +bool has_present_value(const ColumnPtr& column) { + if (const auto* nullable = check_and_get_column(*column)) { + return std::ranges::any_of(nullable->get_null_map_data(), + [](uint8_t is_null) { return is_null == 0; }); + } + return !column->empty(); +} + +bool supports_direct_typed_variant_state(const ParquetColumnSchema& schema) { + if (schema.type == nullptr || schema.kind != ParquetColumnSchemaKind::PRIMITIVE) { + return false; + } + // ColumnVariantV2 typed state carries only a Doris type. Binary/UUID annotations, temporal + // units, and other Parquet-only identity must therefore reconstruct canonical Variant bytes. + switch (remove_nullable(schema.type)->get_primitive_type()) { + case TYPE_BOOLEAN: + case TYPE_TINYINT: + case TYPE_SMALLINT: + case TYPE_INT: + case TYPE_BIGINT: + case TYPE_FLOAT: + case TYPE_DOUBLE: + case TYPE_DECIMAL128I: + case TYPE_DATEV2: + return true; + default: + return false; + } +} + +bool same_data_type(const DataTypePtr& left, const DataTypePtr& right) { + return (!left && !right) || (left && right && left->equals(*right)); +} + +bool same_type_descriptor(const ParquetTypeDescriptor& left, const ParquetTypeDescriptor& right) { + return same_data_type(left.doris_type, right.doris_type) && + same_data_type(left.physical_doris_type, right.physical_doris_type) && + left.extra_type_info == right.extra_type_info && left.time_unit == right.time_unit && + left.physical_type == right.physical_type && + left.integer_bit_width == right.integer_bit_width && + left.decimal_precision == right.decimal_precision && + left.decimal_scale == right.decimal_scale && left.fixed_length == right.fixed_length && + left.is_unsigned_integer == right.is_unsigned_integer && + left.is_decimal == right.is_decimal && left.is_timestamp == right.is_timestamp && + left.timestamp_is_adjusted_to_utc == right.timestamp_is_adjusted_to_utc && + left.is_string_like == right.is_string_like && + left.is_string_annotation == right.is_string_annotation && + left.is_uuid == right.is_uuid && left.unsupported_reason == right.unsupported_reason; +} + +bool same_shredded_schema(const ParquetColumnSchema& left, const ParquetColumnSchema& right) { + if (left.name != right.name || left.kind != right.kind || + !same_data_type(left.type, right.type) || + !same_type_descriptor(left.type_descriptor, right.type_descriptor) || + left.children.size() != right.children.size()) { + return false; + } + for (size_t i = 0; i < left.children.size(); ++i) { + if (!same_shredded_schema(*left.children[i], *right.children[i])) { + return false; + } + } + return true; +} + +void append_compatible_column(IColumn& output, const IColumn& converted); +void validate_compatible_column(const IColumn& output, const IColumn& converted); + +class ParquetVariantShreddedState final : public VariantShreddedState { +public: + ParquetVariantShreddedState(std::shared_ptr schema, + ColumnPtr physical, bool complete, + ParquetColumnReaderProfile profile = {}) + : _schema(std::move(schema)), + _physical(std::move(physical)), + _complete(complete), + _profile(profile) { + DORIS_CHECK(_schema != nullptr && static_cast(_physical)); + const ColumnPtr wrapper = unwrap_nullable(_physical); + const auto* structure = check_and_get_column(*wrapper); + if (structure == nullptr || structure->tuple_size() != _schema->children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant {} physical field count mismatch", _schema->name); + } + } + + size_t size() const override { return _physical->size(); } + size_t byte_size() const override { + std::lock_guard lock(_materialization_lock); + return _physical->byte_size() + (_materialized ? _materialized->byte_size() : 0); + } + size_t allocated_bytes() const override { + std::lock_guard lock(_materialization_lock); + return _physical->allocated_bytes() + + (_materialized ? _materialized->allocated_bytes() : 0); + } + void sanity_check() const override { _physical->sanity_check(); } + + void for_each_subcolumn(IColumn::ColumnCallback callback) const override { + callback(*_physical); + } + + std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t result_size_hint) const override { + // Compact the decoded physical tree directly. In particular, a leaf-only projection has + // no metadata/value columns from which a canonical Variant could be reconstructed. + // The projection schema is immutable and reader-scoped, so derived selections share it + // instead of cloning the whole shredded tree for every filter operation. + return std::make_shared( + _schema, _physical->filter(filter, result_size_hint), _complete, _profile); + } + + std::shared_ptr select_range(size_t start, size_t length) const override { + return std::make_shared(_schema, _physical->cut(start, length), + _complete, _profile); + } + + std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const override { + MutableColumnPtr selected = _physical->clone_empty(); + selected->insert_indices_from(*_physical, indices_begin, indices_end); + return std::make_shared(_schema, std::move(selected), + _complete, _profile); + } + + bool try_append(const VariantShreddedState& source) override { + const auto* parquet_source = dynamic_cast(&source); + if (parquet_source == nullptr || _complete != parquet_source->_complete || + !same_shredded_schema(*_schema, *parquet_source->_schema)) { + return false; + } + validate_compatible_column(*_physical, *parquet_source->_physical); + auto mutable_physical = IColumn::mutate(std::move(_physical)); + append_compatible_column(*mutable_physical, *parquet_source->_physical); + _physical = std::move(mutable_physical); + std::lock_guard lock(_materialization_lock); + _materialized.reset(); + return true; + } + + std::optional find_typed_value( + std::span path) const override { + auto path_miss = [&]() -> std::optional { + update_counter(_profile.variant_direct_leaf_path_misses, 1); + return std::nullopt; + }; + if (path.empty()) { + return path_miss(); + } + + const ParquetColumnSchema* typed_schema = nullptr; + ColumnPtr typed = struct_child(*_schema, _physical, "typed_value", &typed_schema); + if (!typed || typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { + return path_miss(); + } + + for (size_t position = 0; position < path.size(); ++position) { + if (path[position].kind != VariantShreddedPathSegment::Kind::OBJECT_KEY) { + return path_miss(); + } + + const std::string_view key(path[position].key.data, path[position].key.size); + const ParquetColumnSchema* wrapper_schema = nullptr; + ColumnPtr wrapper = struct_child(*typed_schema, typed, key, &wrapper_schema); + if (!wrapper) { + return path_miss(); + } + + if (ColumnPtr residual = struct_child(*wrapper_schema, wrapper, "value", nullptr); + static_cast(residual) && has_present_value(residual)) { + // A residual value can contribute data to the same logical object. Reconstructing + // is required in that case; returning only the typed leaf would drop information. + update_counter(_profile.variant_direct_leaf_residual_fallbacks, 1); + return std::nullopt; + } + + typed = struct_child(*wrapper_schema, wrapper, "typed_value", &typed_schema); + if (!typed) { + return path_miss(); + } + if (position + 1 == path.size()) { + if (typed_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || + check_and_get_column(*typed) == nullptr || + !supports_direct_typed_variant_state(*typed_schema)) { + update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); + return std::nullopt; + } + update_counter(_profile.variant_direct_leaf_rows, + static_cast(typed->size())); + return VariantShreddedTypedValue {.column = std::move(typed), + .type = remove_nullable(typed_schema->type)}; + } + if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { + return path_miss(); + } + } + return std::nullopt; + } + + const ColumnVariantV2& materialized_column() const override { + std::lock_guard lock(_materialization_lock); + if (!_complete) { + throw Exception( + ErrorCode::INTERNAL_ERROR, + "A projected Parquet Variant can only serve its validated shredded leaves"); + } + if (!_materialized) { + SCOPED_TIMER(_profile.variant_reconstruction_time); + _materialized = encode_variant_column(*_schema, *_physical); + update_counter(_profile.variant_reconstructed_rows, + static_cast(_physical->size())); + } + return *_materialized; + } + +private: + static void update_counter(RuntimeProfile::Counter* counter, int64_t value) { + if (counter != nullptr) { + COUNTER_UPDATE(counter, value); + } + } + + std::shared_ptr _schema; + ColumnPtr _physical; + bool _complete = true; + ParquetColumnReaderProfile _profile; + mutable std::mutex _materialization_lock; + mutable ColumnVariantV2::MutablePtr _materialized; +}; + +MutableColumnPtr build_variant_column(std::shared_ptr schema, + ColumnPtr physical, bool complete, + const ParquetColumnReaderProfile& profile) { + DORIS_CHECK(schema != nullptr); + if (schema->kind != ParquetColumnSchemaKind::VARIANT) { + throw Exception(ErrorCode::INVALID_ARGUMENT, "Parquet column {} is not Variant", + schema->name); + } + + const auto* outer_nullable = check_and_get_column(*physical); + MutableColumnPtr variants = + ColumnVariantV2::create_shredded(std::make_shared( + std::move(schema), physical, complete, profile)); + if (outer_nullable == nullptr) { + return variants; + } + auto nulls = outer_nullable->get_null_map_column().clone_resized(physical->size()); + return ColumnNullable::create(std::move(variants), std::move(nulls)); +} + +ColumnPtr transform_node(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile); + +ColumnPtr transform_non_nullable(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile) { + const auto& schema = *plan.schema; + switch (schema.kind) { + case ParquetColumnSchemaKind::PRIMITIVE: + return physical; + case ParquetColumnSchemaKind::VARIANT: + return build_variant_column( + plan.variant_state_schema + ? plan.variant_state_schema + : create_variant_state_schema(schema, plan.variant_projection + ? &*plan.variant_projection + : nullptr), + std::move(physical), + !format::is_partial_projection(plan.variant_projection ? &*plan.variant_projection + : nullptr), + profile); + case ParquetColumnSchemaKind::STRUCT: { + const auto& structure = assert_cast(*physical); + if (structure.tuple_size() != plan.children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Projected Parquet STRUCT {} field count mismatch", schema.name); + } + Columns fields; + fields.reserve(plan.children.size()); + for (size_t i = 0; i < plan.children.size(); ++i) { + fields.push_back( + transform_node(*plan.children[i], structure.get_column_ptr(i), profile)); + } + return ColumnStruct::create(std::move(fields)); + } + case ParquetColumnSchemaKind::LIST: { + const auto& array = assert_cast(*physical); + if (plan.children.size() != 1) { + throw Exception(ErrorCode::CORRUPTION, "Projected Parquet ARRAY plan is invalid"); + } + auto values = transform_node(*plan.children[0], array.get_data_ptr(), profile); + return ColumnArray::create(std::move(values), array.get_offsets_ptr()); + } + case ParquetColumnSchemaKind::MAP: { + const auto& map = assert_cast(*physical); + if (plan.children.size() != 2) { + throw Exception(ErrorCode::CORRUPTION, "Projected Parquet MAP plan is invalid"); + } + auto keys = transform_node(*plan.children[0], map.get_keys_ptr(), profile); + auto values = transform_node(*plan.children[1], map.get_values_ptr(), profile); + return ColumnMap::create(std::move(keys), std::move(values), map.get_offsets_ptr()); + } + } + throw Exception(ErrorCode::INTERNAL_ERROR, "Unknown Parquet schema kind"); +} + +ColumnPtr transform_node(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile) { + if (plan.schema == nullptr) { + throw Exception(ErrorCode::INTERNAL_ERROR, "Parquet Variant materialization plan is null"); + } + if (plan.schema->kind == ParquetColumnSchemaKind::VARIANT) { + return build_variant_column( + plan.variant_state_schema + ? plan.variant_state_schema + : create_variant_state_schema( + *plan.schema, + plan.variant_projection ? &*plan.variant_projection : nullptr), + std::move(physical), + !format::is_partial_projection(plan.variant_projection ? &*plan.variant_projection + : nullptr), + profile); + } + if (const auto* nullable = check_and_get_column(*physical)) { + auto nested = transform_non_nullable(plan, nullable->get_nested_column_ptr(), profile); + return ColumnNullable::create(std::move(nested), nullable->get_null_map_column_ptr()); + } + return transform_non_nullable(plan, std::move(physical), profile); +} + +void append_compatible_column(IColumn& output, const IColumn& converted) { + if (auto* output_nullable = check_and_get_column(output)) { + auto& nested = output_nullable->get_nested_column(); + auto& null_map = output_nullable->get_null_map_column(); + const size_t nested_size = nested.size(); + const size_t null_map_size = null_map.size(); + try { + if (const auto* converted_nullable = check_and_get_column(converted)) { + append_compatible_column(nested, converted_nullable->get_nested_column()); + null_map.insert_range_from(converted_nullable->get_null_map_column(), 0, + converted.size()); + } else { + append_compatible_column(nested, converted); + // External slots and nested Iceberg fields may remain nullable even when one + // file's physical node is required. Preserve that destination invariant with + // non-null bits. + output_nullable->push_false_to_nullmap(converted.size()); + } + } catch (...) { + if (nested.size() > nested_size) { + nested.pop_back(nested.size() - nested_size); + } + if (null_map.size() > null_map_size) { + null_map.pop_back(null_map.size() - null_map_size); + } + throw; + } + return; + } + + if (const auto* converted_nullable = check_and_get_column(converted)) { + // Parquet writers may encode an Iceberg required field as optional. It can populate a + // non-nullable destination only when this batch proves that every value is present. + if (converted_nullable->has_null()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced null data for a " + "non-nullable destination"); + } + append_compatible_column(output, converted_nullable->get_nested_column()); + return; + } + + if (auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible STRUCT"); + } + std::vector original_sizes(output_struct->tuple_size()); + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + original_sizes[i] = output_struct->get_column(i).size(); + } + try { + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + append_compatible_column(output_struct->get_column(i), + converted_struct->get_column(i)); + } + } catch (...) { + // Variant corruption can surface only during lazy fallback after earlier siblings + // were appended. Roll every child back to preserve the failed-append invariant. + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + auto& child = output_struct->get_column(i); + if (child.size() > original_sizes[i]) { + child.pop_back(child.size() - original_sizes[i]); + } + } + throw; + } + return; + } + + if (auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + if (converted_array == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible ARRAY"); + } + auto& output_data = output_array->get_data(); + auto& output_offsets = output_array->get_offsets(); + const size_t element_base = output_data.size(); + const size_t offsets_size = output_offsets.size(); + try { + append_compatible_column(output_data, converted_array->get_data()); + output_offsets.reserve(output_offsets.size() + converted_array->size()); + for (const auto offset : converted_array->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + } catch (...) { + if (output_data.size() > element_base) { + output_data.pop_back(output_data.size() - element_base); + } + output_offsets.resize(offsets_size); + throw; + } + return; + } + + if (auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + if (converted_map == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible MAP"); + } + auto& output_keys = output_map->get_keys(); + auto& output_values = output_map->get_values(); + auto& output_offsets = output_map->get_offsets(); + const size_t element_base = output_keys.size(); + const size_t values_size = output_values.size(); + const size_t offsets_size = output_offsets.size(); + try { + append_compatible_column(output_keys, converted_map->get_keys()); + append_compatible_column(output_values, converted_map->get_values()); + output_offsets.reserve(output_offsets.size() + converted_map->size()); + for (const auto offset : converted_map->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + } catch (...) { + if (output_keys.size() > element_base) { + output_keys.pop_back(output_keys.size() - element_base); + } + if (output_values.size() > values_size) { + output_values.pop_back(output_values.size() - values_size); + } + output_offsets.resize(offsets_size); + throw; + } + return; + } + + if (auto* output_variant = check_and_get_column(output)) { + const auto* converted_variant = check_and_get_column(converted); + if (converted_variant == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible column"); + } + output_variant->insert_range_from(*converted_variant, 0, converted_variant->size()); + return; + } + + if (output.get_name() != converted.get_name()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced {} for {} destination", + converted.get_name(), output.get_name()); + } + output.insert_range_from(converted, 0, converted.size()); +} + +void validate_compatible_column(const IColumn& output, const IColumn& converted) { + if (const auto* output_nullable = check_and_get_column(output)) { + if (const auto* converted_nullable = check_and_get_column(converted)) { + validate_compatible_column(output_nullable->get_nested_column(), + converted_nullable->get_nested_column()); + } else { + validate_compatible_column(output_nullable->get_nested_column(), converted); + } + return; + } + if (const auto* converted_nullable = check_and_get_column(converted)) { + if (converted_nullable->has_null()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced null data for a " + "non-nullable destination"); + } + validate_compatible_column(output, converted_nullable->get_nested_column()); + return; + } + if (const auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible STRUCT"); + } + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + validate_compatible_column(output_struct->get_column(i), + converted_struct->get_column(i)); + } + return; + } + if (const auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + if (converted_array == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible ARRAY"); + } + validate_compatible_column(output_array->get_data(), converted_array->get_data()); + return; + } + if (const auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + if (converted_map == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible MAP"); + } + validate_compatible_column(output_map->get_keys(), converted_map->get_keys()); + validate_compatible_column(output_map->get_values(), converted_map->get_values()); + return; + } + if (check_and_get_column(output) != nullptr) { + if (check_and_get_column(converted) == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible column"); + } + return; + } + if (output.get_name() != converted.get_name()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced {} for {} destination", + converted.get_name(), output.get_name()); + } +} + +bool has_exact_column_shape(const IColumn& output, const IColumn& converted) { + const auto* output_nullable = check_and_get_column(output); + const auto* converted_nullable = check_and_get_column(converted); + if (output_nullable != nullptr || converted_nullable != nullptr) { + return output_nullable != nullptr && converted_nullable != nullptr && + has_exact_column_shape(output_nullable->get_nested_column(), + converted_nullable->get_nested_column()); + } + if (const auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + return false; + } + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + if (!has_exact_column_shape(output_struct->get_column(i), + converted_struct->get_column(i))) { + return false; + } + } + return true; + } + if (const auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + return converted_array != nullptr && + has_exact_column_shape(output_array->get_data(), converted_array->get_data()); + } + if (const auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + return converted_map != nullptr && + has_exact_column_shape(output_map->get_keys(), converted_map->get_keys()) && + has_exact_column_shape(output_map->get_values(), converted_map->get_values()); + } + if (check_and_get_column(output) != nullptr) { + return check_and_get_column(converted) != nullptr; + } + return output.get_name() == converted.get_name(); +} + +void append_materialized_column(MutableColumnPtr& output, ColumnPtr converted) { + // Validate the complete destination shape before mutation. This preserves atomic failures + // without copying a full scratch batch, while an empty exact-shape output can adopt the tree. + validate_compatible_column(*output, *converted); + if (output->empty() && has_exact_column_shape(*output, *converted)) { + if (converted->is_exclusive()) { + // The transformed tree consumed the decoder tree and is recursively exclusive. Keep + // primitive siblings and their buffers intact instead of recursively COW-cloning them. + output = converted->assert_mutable(); + return; + } + output = IColumn::mutate(std::move(converted)); + return; + } + append_compatible_column(*output, *converted); +} + +} // namespace + +std::shared_ptr create_variant_state_schema( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection) { + return std::shared_ptr(clone_schema(schema, projection)); +} + +Status materialize_variant_rows(const ParquetColumnSchema& schema, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + return materialize_variant_rows(schema, physical.get_ptr(), output, profile); +} + +Status materialize_variant_rows(const ParquetColumnSchema& schema, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + if (!output) { + return Status::InvalidArgument("Parquet Variant output column is null"); + } + RETURN_IF_CATCH_EXCEPTION({ + auto converted = build_variant_column(create_variant_state_schema(schema), + std::move(physical), true, profile); + append_materialized_column(output, std::move(converted)); + }); + return Status::OK(); +} + +Status materialize_variant_columns(const VariantMaterializationNode& plan, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + return materialize_variant_columns(plan, physical.get_ptr(), output, profile); +} + +Status materialize_variant_columns(const VariantMaterializationNode& plan, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + if (!output) { + return Status::InvalidArgument("Parquet Variant output column is null"); + } + RETURN_IF_CATCH_EXCEPTION({ + auto converted = transform_node(plan, std::move(physical), profile); + append_materialized_column(output, std::move(converted)); + }); + return Status::OK(); +} + +} // namespace doris::format::parquet diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.h b/be/src/format_v2/parquet/reader/variant_column_reader.h new file mode 100644 index 00000000000000..53cc53983181c8 --- /dev/null +++ b/be/src/format_v2/parquet/reader/variant_column_reader.h @@ -0,0 +1,67 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include + +#include "common/status.h" +#include "core/column/column.h" +#include "format_v2/column_data.h" +#include "format_v2/parquet/parquet_profile.h" + +namespace doris::format::parquet { + +struct ParquetColumnSchema; + +// Projection-aligned view of the file schema. Complex nodes contain only the children decoded by +// NativeColumnReader. A VARIANT node may own a validated fully-shredded physical leaf projection. +struct VariantMaterializationNode { + const ParquetColumnSchema* schema = nullptr; + std::vector> children; + bool contains_variant = false; + std::optional variant_projection; + std::shared_ptr variant_state_schema; +}; + +// Builds the immutable schema retained by a shredded state in the exact order of the decoded +// physical projection. +std::shared_ptr create_variant_state_schema( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection = nullptr); + +// Converts one physical Parquet Variant wrapper column to ColumnVariantV2 and appends it to output. +// SQL NULL is represented by the wrapper's outer null map; a present wrapper with neither value nor +// typed_value is the Variant null value. +Status materialize_variant_rows(const ParquetColumnSchema& schema, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); +Status materialize_variant_rows(const ParquetColumnSchema& schema, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); + +// Recursively replaces projected VARIANT nodes inside STRUCT/LIST/MAP columns while preserving the +// surrounding column shape, offsets, and null maps. The destination is unchanged on decode errors. +Status materialize_variant_columns(const VariantMaterializationNode& plan, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); +Status materialize_variant_columns(const VariantMaterializationNode& plan, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); + +} // namespace doris::format::parquet diff --git a/be/src/format_v2/schema_projection.cpp b/be/src/format_v2/schema_projection.cpp index 342f4c91898c92..d7b2a359131ab7 100644 --- a/be/src/format_v2/schema_projection.cpp +++ b/be/src/format_v2/schema_projection.cpp @@ -88,6 +88,11 @@ Status rebuild_semantic_projected_type(const DataTypePtr& original_type, nested_projected_type = std::make_shared(key_type, value_type); break; } + case TYPE_VARIANT: + // Variant children describe a format-specific physical shredding carrier, not the public + // logical type. Pruning those children must keep the file block exposed as Variant. + *projected_type = original_type; + return Status::OK(); default: return Status::InvalidArgument("Cannot project children from non-complex type {}", original_type->get_name()); diff --git a/be/src/format_v2/table/iceberg_reader.cpp b/be/src/format_v2/table/iceberg_reader.cpp index 097e6bab111091..1fc36a57c1ab69 100644 --- a/be/src/format_v2/table/iceberg_reader.cpp +++ b/be/src/format_v2/table/iceberg_reader.cpp @@ -63,6 +63,94 @@ namespace doris::format::iceberg { static constexpr const char* ROW_LINEAGE_ROW_ID = "_row_id"; static constexpr int32_t ROW_LINEAGE_ROW_ID_FIELD_ID = 2147483540; +namespace { + +bool contains_variant_type(const DataTypePtr& input) { + if (input == nullptr) { + return false; + } + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + +bool mapping_reads_variant(const format::ColumnMapping& mapping) { + if (!mapping.file_local_id.has_value()) { + return false; + } + if (contains_variant_type(mapping.original_file_type)) { + return true; + } + if (mapping.table_type != nullptr && + remove_nullable(mapping.table_type)->get_primitive_type() == TYPE_VARIANT) { + return true; + } + return std::ranges::any_of(mapping.child_mappings, mapping_reads_variant); +} + +const char* file_format_name(FileFormat format) { + switch (format) { + case FileFormat::PARQUET: + return "PARQUET"; + case FileFormat::ORC: + return "ORC"; + case FileFormat::CSV: + return "CSV"; + case FileFormat::JSON: + return "JSON"; + case FileFormat::TEXT: + return "TEXT"; + case FileFormat::JNI: + return "JNI"; + case FileFormat::NATIVE: + return "NATIVE"; + case FileFormat::ARROW: + return "ARROW"; + case FileFormat::WAL: + return "WAL"; + } + return "UNKNOWN"; +} + +} // namespace + +Status IcebergTableReader::validate_variant_file_mappings( + FileFormat format, const std::vector& mappings) { + if (format == FileFormat::PARQUET || !std::ranges::any_of(mappings, mapping_reads_variant)) { + return Status::OK(); + } + // Gate on a physical mapping, not the table schema: an older ORC/Avro file may legitimately + // omit a Variant field added by schema evolution, in which case the mapper synthesizes NULL. + return Status::NotSupported( + "Iceberg Variant is supported only for Parquet files in FileScannerV2; file format {} " + "(including ORC/Avro readers) is not supported", + file_format_name(format)); +} + +Status IcebergTableReader::validate_file_mapping(const format::TableColumnMapper& mapper) const { + if (_push_down_agg_type == TPushAggOp::type::COUNT && _push_down_count_columns.has_value() && + _push_down_count_columns->empty()) { + // COUNT(*) may retain an arbitrary minimum-width slot, but that carrier is never a + // semantic physical read and must not trigger the Variant file-format capability gate. + return Status::OK(); + } + return validate_variant_file_mappings(_format, mapper.mappings()); +} + template static std::string join_values_for_debug(const std::vector& values) { std::ostringstream out; diff --git a/be/src/format_v2/table/iceberg_reader.h b/be/src/format_v2/table/iceberg_reader.h index 2768e4cd3e8c73..4118655dca46f0 100644 --- a/be/src/format_v2/table/iceberg_reader.h +++ b/be/src/format_v2/table/iceberg_reader.h @@ -49,6 +49,8 @@ namespace doris::format::iceberg { class IcebergTableReader : public format::TableReader { public: ~IcebergTableReader() override = default; + static Status validate_variant_file_mappings( + FileFormat format, const std::vector& mappings); Status init(format::TableReadOptions&& options) override { RETURN_IF_ERROR(format::TableReader::init(std::move(options))); _mapper_options.mode = format::TableColumnMappingMode::BY_FIELD_ID; @@ -73,6 +75,8 @@ class IcebergTableReader : public format::TableReader { } protected: + Status validate_file_mapping(const format::TableColumnMapper& mapper) const override; + void configure_mapper_options(format::TableColumnMapperOptions* options) const override { options->enable_row_lineage_virtual_columns = true; options->allow_idless_complex_wrapper_projection = diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index 2691e32f8bdbf8..c82d27999a913c 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -1057,8 +1057,15 @@ Status TableReader::_build_table_filters_from_conjuncts() { if (in_safe_prefix && !_is_safe_to_pre_execute(conjunct)) { in_safe_prefix = false; } + const size_t first_new_filter = _table_filters.size(); RETURN_IF_ERROR( build_table_filters_from_conjunct(conjunct, _runtime_state, &_table_filters)); + for (size_t filter_idx = first_new_filter; filter_idx < _table_filters.size(); + ++filter_idx) { + // Preserve the original conjunct-order fence even when the unsafe expression itself + // had no slot and therefore produced no TableFilter entry. + _table_filters[filter_idx].metadata_pruning_safe = in_safe_prefix; + } if (in_safe_prefix) { _constant_pruning_safe_filter_count = _table_filters.size(); } @@ -1068,47 +1075,29 @@ Status TableReader::_build_table_filters_from_conjuncts() { namespace { -bool same_scan_projection(const LocalColumnIndex& lhs, const LocalColumnIndex& rhs) { - if (lhs.index != rhs.index || lhs.project_all_children != rhs.project_all_children || - lhs.children.size() != rhs.children.size()) { +bool same_scan_projections(const std::vector& lhs, + const std::vector& rhs) { + if (lhs.size() != rhs.size()) { return false; } - for (size_t index = 0; index < lhs.children.size(); ++index) { - if (!same_scan_projection(lhs.children[index], rhs.children[index])) { + for (const auto& lhs_projection : lhs) { + const auto rhs_it = std::ranges::find_if(rhs, [&](const LocalColumnIndex& rhs_projection) { + return rhs_projection.column_id() == lhs_projection.column_id(); + }); + if (rhs_it == rhs.end() || !same_local_column_index(lhs_projection, *rhs_it)) { return false; } } return true; } -const LocalColumnIndex* find_scan_projection(const FileScanRequest& request, - LocalColumnId column_id) { - const auto find_by_id = [column_id](const std::vector& projections) { - return std::ranges::find_if(projections, [column_id](const LocalColumnIndex& projection) { - return projection.column_id() == column_id; - }); - }; - auto it = find_by_id(request.predicate_columns); - if (it != request.predicate_columns.end()) { - return &*it; - } - it = find_by_id(request.non_predicate_columns); - return it == request.non_predicate_columns.end() ? nullptr : &*it; -} - bool same_physical_scan_layout(const FileScanRequest& lhs, const FileScanRequest& rhs) { - if (lhs.local_positions != rhs.local_positions) { - return false; - } - for (const auto& [column_id, _] : lhs.local_positions) { - const auto* lhs_projection = find_scan_projection(lhs, column_id); - const auto* rhs_projection = find_scan_projection(rhs, column_id); - if (lhs_projection == nullptr || rhs_projection == nullptr || - !same_scan_projection(*lhs_projection, *rhs_projection)) { - return false; - } - } - return true; + // Deferred complex roots occupy independent output slots. Comparing only eager positions can + // accept a refresh whose second Variant root now aliases or overruns the active block layout. + return lhs.local_positions == rhs.local_positions && + lhs.non_predicate_positions == rhs.non_predicate_positions && + same_scan_projections(lhs.predicate_columns, rhs.predicate_columns) && + same_scan_projections(lhs.non_predicate_columns, rhs.non_predicate_columns); } } // namespace @@ -1137,7 +1126,9 @@ Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { auto refreshed_request = std::make_shared(); RETURN_IF_ERROR(refreshed_mapper->create_scan_request( _table_filters, _projected_columns, refreshed_request.get(), _runtime_state, - _file_scan_request == nullptr ? nullptr : &_file_scan_request->local_positions)); + _file_scan_request == nullptr ? nullptr : &_file_scan_request->local_positions, + _file_scan_request == nullptr ? nullptr + : &_file_scan_request->non_predicate_positions)); // A refresh does not prove that every future runtime filter has arrived. Keep carrier values // available whenever the split started with pending filters. if (_push_down_agg_type == TPushAggOp::type::COUNT && _push_down_count_columns.has_value() && diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index 07d2ed652a4306..4d0bd7474f3808 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -82,6 +82,7 @@ using DeleteRows = std::vector; struct TableFilter { VExprContextSPtr conjunct; std::vector global_indices; + bool metadata_pruning_safe = true; }; struct ScanTask { @@ -414,6 +415,7 @@ class TableReader { DORIS_CHECK(file_schema != nullptr); return Status::OK(); } + virtual Status validate_file_mapping(const TableColumnMapper&) const { return Status::OK(); } // Open the concrete reader for the current split/task and build the file-local scan request. virtual Status open_reader() { @@ -457,6 +459,7 @@ class TableReader { RETURN_IF_ERROR(close_current_reader()); return Status::OK(); } + RETURN_IF_ERROR(validate_file_mapping(*_data_reader.column_mapper)); // COUNT(*) has no semantic column argument, but Nereids retains a minimum-width scan slot // so the scan node still has an output tuple. Record only the current non-predicate file // columns before table-format hooks add row-position or equality-delete dependencies. This @@ -479,40 +482,34 @@ class TableReader { _data_reader.file_block_layout.clear(); _data_reader.block_template.clear(); _file_scan_request.reset(); - _data_reader.file_block_layout.resize(file_request->local_positions.size()); + _data_reader.file_block_layout.resize(file_request->block_column_count()); // 4. Build file block layout from file schema and column mapping. The layout describes // the block returned by file reader before table-column materialization. - for (const auto& [file_column_id, block_position] : file_request->local_positions) { + auto add_file_block_column = [&](const LocalColumnIndex& projection, + LocalIndex block_position) -> Status { DORIS_CHECK(block_position.value() < _data_reader.file_block_layout.size()); + const auto file_column_id = projection.column_id(); const auto* field = _find_column_definition(_data_reader.file_schema, file_column_id); DORIS_CHECK(field != nullptr); ColumnDefinition projected_field; - { - auto it = std::find_if( - file_request->non_predicate_columns.begin(), - file_request->non_predicate_columns.end(), - [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); - if (it != file_request->non_predicate_columns.end()) { - RETURN_IF_ERROR(project_column_definition(*field, *it, &projected_field)); - } - } - { - auto it = std::find_if( - file_request->predicate_columns.begin(), - file_request->predicate_columns.end(), - [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); - if (it != file_request->predicate_columns.end()) { - RETURN_IF_ERROR(project_column_definition(*field, *it, &projected_field)); - } - } + RETURN_IF_ERROR(project_column_definition(*field, projection, &projected_field)); _data_reader.file_block_layout[block_position.value()] = { .file_column_id = file_column_id, .name = projected_field.name, .type = projected_field.type, }; DORIS_CHECK(_data_reader.file_block_layout[block_position.value()].type != nullptr); + return Status::OK(); + }; + for (const auto& projection : file_request->predicate_columns) { + RETURN_IF_ERROR(add_file_block_column( + projection, file_request->local_positions.at(projection.column_id()))); + } + for (const auto& projection : file_request->non_predicate_columns) { + RETURN_IF_ERROR(add_file_block_column( + projection, file_request->non_predicate_position(projection.column_id()))); } // 5. Prepare block template from file block layout. The block template stores the block diff --git a/be/test/core/column/column_self_check.cpp b/be/test/core/column/column_self_check.cpp index 4a906cc62a7417..4f9a6f9b47b29a 100644 --- a/be/test/core/column/column_self_check.cpp +++ b/be/test/core/column/column_self_check.cpp @@ -17,9 +17,14 @@ #include +#include "common/config.h" +#include "common/exception.h" #include "core/column/column.h" #include "core/column/column_array.h" #include "core/column/column_const.h" +#include "core/column/column_map.h" +#include "core/column/column_nullable.h" +#include "core/column/column_string.h" #include "core/data_type/data_type_number.h" #include "core/data_type/primitive_type.h" #include "exprs/function/function.h" @@ -28,6 +33,29 @@ namespace doris { +namespace { + +class ScopedMemAllocFaultInjection { +public: + ScopedMemAllocFaultInjection() : _old_probability(config::mem_alloc_fault_probability) { + config::mem_alloc_fault_probability = 1.0; + ++enable_thread_catch_bad_alloc; + } + + ~ScopedMemAllocFaultInjection() { + --enable_thread_catch_bad_alloc; + config::mem_alloc_fault_probability = _old_probability; + } + + ScopedMemAllocFaultInjection(const ScopedMemAllocFaultInjection&) = delete; + ScopedMemAllocFaultInjection& operator=(const ScopedMemAllocFaultInjection&) = delete; + +private: + double _old_probability; +}; + +} // namespace + TEST(ColumnSelfCheckTest, const_check_test) { { ColumnPtr col = ColumnHelper::create_column({1, 2, 3}); @@ -147,4 +175,44 @@ TEST(ColumnSelfCheckTest, boolean_check) { EXPECT_EQ(column_nullable_bool->column_boolean_check(), true); } } -} // namespace doris \ No newline at end of file + +TEST(ColumnSelfCheckTest, non_boolean_complex_payload_does_not_allocate_during_boolean_check) { + auto keys = ColumnString::create(); + keys->insert_data("key", 3); + auto values = ColumnString::create(); + values->insert_data("value", 5); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + auto map = ColumnMap::create(std::move(keys), std::move(values), std::move(offsets)); + auto nullable_map = ColumnNullable::create(std::move(map), ColumnUInt8::create(1, 0)); + + bool is_valid = false; + { + ScopedMemAllocFaultInjection inject_allocation_failure; + EXPECT_NO_THROW(is_valid = nullable_map->column_boolean_check()); + } + EXPECT_TRUE(is_valid); +} + +TEST(ColumnSelfCheckTest, nested_boolean_check_respects_parent_null_map) { + auto create_nullable_map = [](UInt8 null_row_value, UInt8 non_null_row_value) { + auto keys = ColumnString::create(); + keys->insert_data("first", 5); + keys->insert_data("second", 6); + auto values = ColumnUInt8::create(); + values->insert_value(null_row_value); + values->insert_value(non_null_row_value); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + offsets->insert_value(2); + auto map = ColumnMap::create(std::move(keys), std::move(values), std::move(offsets)); + auto null_map = ColumnUInt8::create(); + null_map->insert_value(1); + null_map->insert_value(0); + return ColumnNullable::create(std::move(map), std::move(null_map)); + }; + + EXPECT_TRUE(create_nullable_map(2, 1)->column_boolean_check()); + EXPECT_FALSE(create_nullable_map(1, 2)->column_boolean_check()); +} +} // namespace doris diff --git a/be/test/core/column/column_variant_v2_test.cpp b/be/test/core/column/column_variant_v2_test.cpp index 2665304cfa0601..9937555b96d7c8 100644 --- a/be/test/core/column/column_variant_v2_test.cpp +++ b/be/test/core/column/column_variant_v2_test.cpp @@ -34,6 +34,7 @@ #include "common/exception.h" #include "core/arena.h" #include "core/assert_cast.h" +#include "core/block/block.h" #include "core/column/column_const.h" #include "core/column/column_decimal.h" #include "core/column/column_nullable.h" @@ -51,6 +52,7 @@ #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_time.h" #include "core/data_type/data_type_timestamptz.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/value/decimalv2_value.h" #include "core/value/ipv4_value.h" #include "core/value/ipv6_value.h" @@ -1788,6 +1790,31 @@ TEST(ColumnVariantV2Test, CowDetachAndClear) { EXPECT_EQ(a.num_elements(), 2); } +TEST(ColumnVariantV2Test, BlockClearDetachesSharedEncodedSubcolumns) { + for (bool clear_selected_only : {false, true}) { + SCOPED_TRACE(clear_selected_only); + auto column = ColumnVariantV2::create(); + insert_encoded_field(*column, encode_json(R"({"a":[1,2]})")); + const std::vector shared_subcolumns = subcolumns(*column); + + Block block; + block.insert({std::move(column), std::make_shared(), "v"}); + ASSERT_TRUE(block.get_by_position(0).column->is_exclusive()); + + if (clear_selected_only) { + block.clear_column_data(std::vector {0}); + } else { + block.clear_column_data(); + } + + EXPECT_EQ(block.get_by_position(0).column->size(), 0); + ASSERT_EQ(shared_subcolumns.size(), 3); + EXPECT_EQ(shared_subcolumns[0]->size(), 1); + EXPECT_EQ(shared_subcolumns[1]->size(), 1); + EXPECT_EQ(shared_subcolumns[2]->size(), 1); + } +} + TEST(ColumnVariantV2Test, EncodedRowCountInvariant) { EXPECT_DEATH( { diff --git a/be/test/exec/scan/access_path_parser_test.cpp b/be/test/exec/scan/access_path_parser_test.cpp index f72da2a69b0b56..23b95d54c0d7b6 100644 --- a/be/test/exec/scan/access_path_parser_test.cpp +++ b/be/test/exec/scan/access_path_parser_test.cpp @@ -32,6 +32,7 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/vexpr_context.h" #include "exprs/vliteral.h" @@ -130,6 +131,92 @@ TEST(AccessPathParserTest, IgnoresPrimitiveColumnsAndScannerVirtualColumns) { EXPECT_TRUE(rowid.children.empty()); } +TEST(AccessPathParserTest, PreservesVariantObjectKeysForPhysicalShreddingProjection) { + auto variant = root_column(100, "v", std::make_shared()); + auto status = AccessPathParser::build_nested_children( + &variant, + std::vector {data_access_path({"100", "typed_col"}), + data_access_path({"100", "nested", "leaf"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_EQ(variant.variant_access_paths, + (std::vector> {{"nested", "leaf"}, {"typed_col"}})); + + status = AccessPathParser::build_nested_children( + &variant, std::vector {data_access_path({"100"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); + + status = AccessPathParser::build_nested_children( + &variant, + std::vector {data_access_path({"100", "typed_col"}), + data_access_path({})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); +} + +TEST(AccessPathParserTest, SeparatesFinalAndPredicateComplexAccessPaths) { + auto variant = root_column(100, "v", std::make_shared()); + auto status = AccessPathParser::build_nested_children( + &variant, std::vector {data_access_path({"v"})}, + std::vector {data_access_path({"v", "n"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); + ASSERT_TRUE(variant.has_predicate_access_paths); + EXPECT_EQ(variant.predicate_variant_access_paths, + (std::vector> {{"n"}})); + + auto int_type = std::make_shared(); + auto struct_type = + std::make_shared(DataTypes {int_type, int_type}, Strings {"a", "b"}); + auto structure = root_column(101, "s", struct_type); + status = AccessPathParser::build_nested_children( + &structure, std::vector {data_access_path({"s"})}, + std::vector {data_access_path({"s", "b"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(structure.children.size(), 2); + ASSERT_TRUE(structure.has_predicate_access_paths); + ASSERT_EQ(structure.predicate_children.size(), 1); + EXPECT_EQ(structure.predicate_children[0].name, "b"); +} + +TEST(AccessPathParserTest, PreservesVariantPathsNestedInComplexColumns) { + auto variant_type = std::make_shared(); + + auto struct_type = + std::make_shared(DataTypes {variant_type}, Strings {"payload"}); + auto structure = root_column(100, "s", struct_type); + auto status = AccessPathParser::build_nested_children( + &structure, + std::vector {data_access_path({"s", "payload", "typed_col"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(structure.children.size(), 1); + EXPECT_EQ(structure.children[0].variant_access_paths, + (std::vector> {{"typed_col"}})); + + auto array = root_column(101, "items", std::make_shared(variant_type)); + status = AccessPathParser::build_nested_children( + &array, std::vector {data_access_path({"items", "*", "kind"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(array.children.size(), 1); + EXPECT_EQ(array.children[0].variant_access_paths, + (std::vector> {{"kind"}})); + + auto map = root_column( + 102, "attrs", + std::make_shared(std::make_shared(), variant_type)); + status = AccessPathParser::build_nested_children( + &map, std::vector {data_access_path({"attrs", "*", "enabled"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + const auto* value = find_child_by_name(map, "value"); + ASSERT_NE(value, nullptr); + EXPECT_EQ(value->variant_access_paths, (std::vector> {{"enabled"}})); +} + // Scenario: reject unsupported top-level inputs before recursive type parsing, including META // paths, missing DATA payloads, and access paths whose root does not match the projected slot. TEST(AccessPathParserTest, RejectsUnsupportedTopLevelAccessPathInputs) { diff --git a/be/test/exec/scan/file_scanner_v2_test.cpp b/be/test/exec/scan/file_scanner_v2_test.cpp index 3506e5db27ae8a..fa167d2c7dce4a 100644 --- a/be/test/exec/scan/file_scanner_v2_test.cpp +++ b/be/test/exec/scan/file_scanner_v2_test.cpp @@ -117,6 +117,16 @@ TEST(FileScannerTest, V1CountPushdownRequiresExplicitCountStarArguments) { TPushAggOp::type::MINMAX, std::nullopt)); } +TEST(FileScannerTest, CountStarPlaceholderIsNotASemanticProjection) { + EXPECT_TRUE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, + std::vector {})); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, + std::vector {7})); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, std::nullopt)); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::MINMAX, + std::vector {})); +} + TEST(FileScannerV2Test, AdaptiveBatchSizeRunsForCountFallbackOnly) { EXPECT_TRUE(FileScannerV2::TEST_should_run_adaptive_batch_size(true, false)); EXPECT_FALSE(FileScannerV2::TEST_should_run_adaptive_batch_size(true, true)); @@ -456,6 +466,30 @@ TEST(FileScannerV2Test, FileScanLocalStateSelectsV2ForSupportedQueriesOnly) { EXPECT_FALSE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); } +TEST(FileScannerV2Test, LegacyCountExemptionRequiresMetadataCountOnEveryRange) { + auto scan_range = [](std::optional row_count) { + TScanRangeParams params; + auto& file_range = params.scan_range.ext_scan_range.file_scan_range; + TFileRangeDesc range; + if (row_count.has_value()) { + TTableFormatFileDesc table_format; + table_format.__set_table_level_row_count(*row_count); + range.__set_table_format_params(table_format); + } + file_range.ranges.push_back(std::move(range)); + return params; + }; + + LocalSplitSourceConnector proven({scan_range(4), scan_range(0)}, 2); + EXPECT_TRUE(proven.all_ranges_have_table_level_row_count()); + + LocalSplitSourceConnector missing({scan_range(4), scan_range(std::nullopt)}, 2); + EXPECT_FALSE(missing.all_ranges_have_table_level_row_count()); + + LocalSplitSourceConnector invalid({scan_range(4), scan_range(-1)}, 2); + EXPECT_FALSE(invalid.all_ranges_have_table_level_row_count()); +} + TEST(FileScannerV2Test, JniCompatibilityShapesUseV2Scanner) { TQueryOptions query_options; query_options.__set_enable_file_scanner_v2(true); diff --git a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp index 960eba0f9c1bd9..23b7cd815c5f54 100644 --- a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp +++ b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp @@ -44,12 +44,16 @@ namespace doris { class TestScanner final : public Scanner { public: TestScanner(RuntimeState* state, ScanLocalStateBase* local_state, int64_t limit, - RuntimeProfile* profile) - : Scanner(state, local_state, limit, profile) {} + RuntimeProfile* profile, bool allow_padding = true) + : Scanner(state, local_state, limit, profile), _allow_padding(allow_padding) {} void add_block(Block block) { _blocks.push_back(std::move(block)); } protected: + bool _can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const override { + return _allow_padding; + } + Status _get_block_impl(RuntimeState* /*state*/, Block* block, bool* eof) override { if (_blocks.empty()) { *eof = true; @@ -62,6 +66,7 @@ class TestScanner final : public Scanner { } private: + bool _allow_padding = true; std::list _blocks; }; @@ -210,4 +215,44 @@ TEST(ScannerProjectionTest, publishes_shared_column_and_reuses_output_block) { EXPECT_EQ(output.get_by_position(0).column->get_int(1), 4); } +TEST(ScannerProjectionTest, projects_incompatible_blocks_before_reading_the_next_block) { + ObjectPool pool; + auto data_type = std::make_shared(); + auto row_descriptor = MockRowDescriptor({data_type}, &pool); + + MockRuntimeState state; + state._batch_size = 8; + + auto op = std::make_shared(); + op->_row_descriptor = row_descriptor; + op->_output_row_descriptor = + std::make_unique(std::vector {data_type}, &pool); + op->_output_tuple_desc = op->_output_row_descriptor->tuple_descriptors()[0]; + + auto local_state = std::make_shared(&state, op.get()); + local_state->_projections = MockSlotRef::create_mock_contexts(0, data_type); + + RuntimeProfile profile("scanner"); + TestScanner scanner(&state, local_state.get(), -1, &profile, false); + ASSERT_TRUE(scanner.init(&state, {}).ok()); + scanner.add_block(ColumnHelper::create_block({0, 1})); + scanner.add_block(ColumnHelper::create_block({2, 3, 4})); + + Block first_output; + bool eos = false; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &first_output, &eos).ok()); + EXPECT_FALSE(eos); + EXPECT_EQ(first_output.rows(), 2); + + Block second_output; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &second_output, &eos).ok()); + EXPECT_FALSE(eos); + EXPECT_EQ(second_output.rows(), 3); + + Block final_output; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &final_output, &eos).ok()); + EXPECT_TRUE(eos); + EXPECT_EQ(final_output.rows(), 0); +} + } // namespace doris diff --git a/be/test/exec/sink/viceberg_merge_sink_test.cpp b/be/test/exec/sink/viceberg_merge_sink_test.cpp index eb7c0159d5fd38..e965fab05e1305 100644 --- a/be/test/exec/sink/viceberg_merge_sink_test.cpp +++ b/be/test/exec/sink/viceberg_merge_sink_test.cpp @@ -33,6 +33,7 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "exec/sink/sink_common.h" #include "exec/sink/viceberg_delete_sink.h" #include "exec/sink/writer/iceberg/viceberg_table_writer.h" #include "exprs/vexpr_context.h" @@ -200,6 +201,43 @@ TEST_F(VIcebergMergeSinkTest, TestUpdateProducesDeleteAndInsert) { ASSERT_TRUE(sink->close(Status::OK()).ok()); } +TEST_F(VIcebergMergeSinkTest, TestDeleteOnlySkipsVariantDataWriter) { + ObjectPool pool; + MockRuntimeState state; + + DataTypes types {std::make_shared(), + std::make_shared(DataTypes {std::make_shared(), + std::make_shared()}, + Strings {"file_path", "row_position"}), + std::make_shared(), std::make_shared()}; + MockRowDescriptor row_desc(types, &pool); + + auto output_exprs = build_output_exprs(&pool, &state, row_desc); + TDataSink t_sink = build_sink(); + t_sink.iceberg_merge_sink.__set_writes_data_files(false); + t_sink.iceberg_merge_sink.__set_schema_json( + "{\"type\":\"struct\",\"schema-id\":0,\"fields\":[" + "{\"id\":1,\"name\":\"payload\",\"required\":false,\"type\":\"variant\"}" + "]}"); + + auto sink = std::make_shared(t_sink, output_exprs, nullptr, nullptr); + sink->set_skip_io(true); + + ASSERT_TRUE(sink->init_properties(&pool, row_desc).ok()); + EXPECT_EQ(nullptr, sink->_table_writer); + RuntimeProfile profile("iceberg_merge_sink"); + ASSERT_TRUE(sink->open(&state, &profile).ok()); + + // Delete-only plans must never use the insert opcode, which intentionally requires a data writer. + Block block = build_block_with_ops({kDeleteOperation}); + Status status = sink->write(&state, block); + ASSERT_TRUE(status.ok()) << status; + EXPECT_EQ(1, sink->_delete_row_count); + EXPECT_EQ(0, sink->_insert_row_count); + + ASSERT_TRUE(sink->close(Status::OK()).ok()); +} + TEST_F(VIcebergMergeSinkTest, TestMissingOperationColumn) { ObjectPool pool; MockRuntimeState state; @@ -435,6 +473,28 @@ TEST_F(VIcebergMergeSinkTest, TestRollingUpgradeSkipsCardinalityState) { EXPECT_TRUE(sink->_matched_row_positions.empty()); } +TEST_F(VIcebergMergeSinkTest, TestRollingUpgradeRejectsDeleteOnlyWriterOmission) { + ObjectPool pool; + MockRuntimeState state; + state.set_be_exec_version(SUPPORT_ICEBERG_VARIANT_VERSION - 1); + + DataTypes types {std::make_shared(), + std::make_shared(DataTypes {std::make_shared(), + std::make_shared()}, + Strings {"file_path", "row_position"}), + std::make_shared(), std::make_shared()}; + MockRowDescriptor row_desc(types, &pool); + auto output_exprs = build_output_exprs(&pool, &state, row_desc); + auto t_sink = build_sink(); + t_sink.iceberg_merge_sink.__set_writes_data_files(false); + auto sink = std::make_shared(t_sink, output_exprs, nullptr, nullptr); + + ASSERT_TRUE(sink->init_properties(&pool, row_desc).ok()); + RuntimeProfile profile("rolling_upgrade_delete_only_iceberg_merge_sink"); + const Status status = sink->open(&state, &profile); + EXPECT_TRUE(status.is()) << status; +} + TEST_F(VIcebergMergeSinkTest, TestErrorCloseRemovesRolledDataFiles) { ObjectPool pool; MockRuntimeState state; diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index dd669f84a061f2..b53164bac4ae28 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -39,6 +39,7 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/data_type_timestamptz.h" #include "core/data_type/data_type_varbinary.h" +#include "core/data_type/data_type_variant_v2.h" #include "exprs/vectorized_fn_call.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" @@ -65,18 +66,10 @@ DataTypePtr i64() { return std::make_shared(); } -DataTypePtr f32() { - return std::make_shared(); -} - DataTypePtr f64() { return std::make_shared(); } -DataTypePtr dec32(uint32_t precision, uint32_t scale) { - return std::make_shared(precision, scale); -} - DataTypePtr str() { return std::make_shared(); } @@ -93,6 +86,10 @@ DataTypePtr u8() { return std::make_shared(); } +DataTypePtr variant_v2() { + return std::make_shared(); +} + ColumnDefinition field_id_col(const std::string& name, int32_t field_id, DataTypePtr type, int32_t local_id = -1) { ColumnDefinition column; @@ -518,14 +515,6 @@ VExprSPtr like_expr(const VExprSPtr& left, const std::string& pattern) { return expr; } -VExprSPtr struct_element_by_selector(const VExprSPtr& parent, DataTypePtr child_type, - const VExprSPtr& selector) { - auto expr = std::make_shared("struct_element", std::move(child_type)); - expr->add_child(parent); - expr->add_child(selector); - return expr; -} - VExprSPtr int_gt(const VExprSPtr& left, int32_t value) { auto expr = std::make_shared("gt", u8(), TExprNodeType::BINARY_PRED, TExprOpcode::GT); @@ -543,58 +532,12 @@ VExprSPtr binary_predicate(TExprOpcode::type opcode, const VExprSPtr& left, return expr; } -VExprSPtr in_predicate(const VExprSPtr& probe, const DataTypePtr& literal_type, - const std::vector& values) { - auto expr = std::make_shared("in", u8(), TExprNodeType::IN_PRED); - expr->add_child(probe); - for (const auto& value : values) { - expr->add_child(literal(literal_type, value)); - } - return expr; -} - -VExprSPtr null_predicate(const VExprSPtr& child, bool is_null) { - auto expr = - std::make_shared(is_null ? "is_null_pred" : "is_not_null_pred", u8()); - expr->add_child(child); - return expr; -} - VExprSPtr cast_expr(const VExprSPtr& child, DataTypePtr target_type) { auto expr = Cast::create_shared(std::move(target_type)); expr->add_child(child); return expr; } -VExprSPtr compound_predicate(TExprOpcode::type opcode, const VExprSPtr& left, - const VExprSPtr& right) { - auto expr = std::make_shared("compound", u8(), TExprNodeType::COMPOUND_PRED, - opcode); - expr->add_child(left); - expr->add_child(right); - return expr; -} - -std::vector collect_paths(const VExprSPtr& expr) { - std::vector paths; - collect_nested_struct_paths(expr, &paths); - return paths; -} - -void expect_name_selector(const StructChildSelector& selector, const std::string& name) { - EXPECT_TRUE(selector.by_name); - EXPECT_EQ(selector.name, name); -} - -void expect_ordinal_selector(const StructChildSelector& selector, size_t ordinal) { - EXPECT_FALSE(selector.by_name); - EXPECT_EQ(selector.ordinal, ordinal); -} - -void expect_path_root(const NestedStructPath& path, size_t global_index) { - EXPECT_EQ(path.root_global_index, GlobalIndex(global_index)); -} - class ColumnMapperCastTest : public testing::Test { protected: void SetUp() override { state.set_enable_strict_cast(true); } @@ -890,225 +833,6 @@ TEST(ColumnMapperNestedHelperTest, BuildsProjectionByNameAndOrdinalSelectors) { EXPECT_EQ(ordinal_projection.children[0].local_id(), 0); } -// ---------------------------------------------------------------------- -// collect_nested_struct_paths() helper tests. -// These tests assert the entry helper for nested scan projection: it only discovers -// table-side struct paths. Later localization decides how to add scan projections. -// ---------------------------------------------------------------------- - -TEST(ColumnMapperCollectNestedStructPathsTest, CollectsNameOrdinalAndBooleanSelectors) { - const auto leaf_type = i32(); - const auto inner_type = - std::make_shared(DataTypes {leaf_type, leaf_type}, Strings {"x", "y"}); - const auto root_type = std::make_shared(DataTypes {inner_type, leaf_type}, - Strings {"nested", "missing"}); - const auto root = table_slot(0, 3, root_type, "s"); - - const auto nested_by_ordinal = struct_element_by_selector( - struct_element_by_selector(root, inner_type, - literal(i32(), Field::create_field(1))), - leaf_type, literal(i32(), Field::create_field(2))); - auto paths = collect_paths(nested_by_ordinal); - ASSERT_EQ(paths.size(), 1); - expect_path_root(paths[0], 3); - ASSERT_EQ(paths[0].selectors.size(), 2); - expect_ordinal_selector(paths[0].selectors[0], 1); - expect_ordinal_selector(paths[0].selectors[1], 2); - - const std::vector positive_ordinal_selectors = { - literal(std::make_shared(), - Field::create_field(static_cast(1))), - literal(std::make_shared(), - Field::create_field(static_cast(2))), - literal(i32(), Field::create_field(3)), - literal(i64(), Field::create_field(4)), - literal(u8(), Field::create_field(true)), - }; - for (size_t idx = 0; idx < positive_ordinal_selectors.size(); ++idx) { - const auto selected = - struct_element_by_selector(root, leaf_type, positive_ordinal_selectors[idx]); - paths = collect_paths(selected); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 1); - expect_ordinal_selector(paths[0].selectors[0], idx == 4 ? 1 : idx + 1); - } - - paths = collect_paths(struct_element(root, leaf_type, "missing")); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 1); - expect_name_selector(paths[0].selectors[0], "missing"); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, IgnoresInvalidSelectorsAndNonPathRoots) { - const auto leaf_type = i32(); - const auto root_type = std::make_shared(DataTypes {leaf_type}, Strings {"a"}); - const auto root = table_slot(0, 0, root_type, "s"); - - const std::vector invalid_selectors = { - literal(i32(), Field::create_field(0)), - literal(i32(), Field::create_field(-1)), - literal(u8(), Field::create_field(false)), - literal(f32(), Field::create_field(1.0F)), - literal(f64(), Field::create_field(1.0)), - table_slot(1, 1, i32(), "selector"), - }; - for (const auto& selector : invalid_selectors) { - EXPECT_TRUE(collect_paths(struct_element_by_selector(root, leaf_type, selector)).empty()); - } - - auto wrong_arity = std::make_shared("struct_element", leaf_type); - wrong_arity->add_child(root); - EXPECT_TRUE(collect_paths(wrong_arity).empty()); - - auto not_struct_element = std::make_shared("other_function", leaf_type); - not_struct_element->add_child(root); - not_struct_element->add_child(literal(str(), Field::create_field("a"))); - EXPECT_TRUE(collect_paths(not_struct_element).empty()); - - EXPECT_TRUE(collect_paths(struct_element(literal(str(), Field::create_field("x")), - leaf_type, "a")) - .empty()); - EXPECT_TRUE(collect_paths(nullptr).empty()); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, RecursesThroughExpressionsAndKeepsCompletePath) { - const auto leaf_type = i32(); - const auto inner_type = std::make_shared(DataTypes {leaf_type}, Strings {"b"}); - const auto root_type = - std::make_shared(DataTypes {inner_type, leaf_type}, Strings {"a", "c"}); - const auto root = table_slot(0, 2, root_type, "s"); - const auto path_a = struct_element_by_selector( - root, inner_type, literal(str(), Field::create_field("a"))); - const auto path_ab = struct_element_by_selector( - path_a, leaf_type, literal(str(), Field::create_field("b"))); - const auto path_c = struct_element_by_selector( - root, leaf_type, literal(str(), Field::create_field("c"))); - - auto paths = collect_paths(binary_predicate( - TExprOpcode::GT, path_ab, literal(leaf_type, Field::create_field(1)))); - ASSERT_EQ(paths.size(), 1); - expect_path_root(paths[0], 2); - ASSERT_EQ(paths[0].selectors.size(), 2); - expect_name_selector(paths[0].selectors[0], "a"); - expect_name_selector(paths[0].selectors[1], "b"); - - paths = collect_paths(compound_predicate( - TExprOpcode::COMPOUND_OR, - binary_predicate(TExprOpcode::GT, path_ab, - literal(leaf_type, Field::create_field(1))), - binary_predicate(TExprOpcode::LT, path_c, - literal(leaf_type, Field::create_field(2))))); - ASSERT_EQ(paths.size(), 2); - ASSERT_EQ(paths[0].selectors.size(), 2); - ASSERT_EQ(paths[1].selectors.size(), 1); - expect_name_selector(paths[0].selectors[0], "a"); - expect_name_selector(paths[0].selectors[1], "b"); - expect_name_selector(paths[1].selectors[0], "c"); - - auto fn = std::make_shared("fn", leaf_type); - fn->add_child(path_ab); - fn->add_child(table_slot(3, 4, leaf_type, "other")); - paths = collect_paths(fn); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 2); - - auto if_expr = std::make_shared("if", leaf_type); - if_expr->add_child(literal(u8(), Field::create_field(true))); - if_expr->add_child(path_ab); - if_expr->add_child(path_c); - paths = collect_paths(if_expr); - ASSERT_EQ(paths.size(), 2); - - paths = collect_paths(compound_predicate(TExprOpcode::COMPOUND_AND, path_ab, path_ab)); - ASSERT_EQ(paths.size(), 2); - - paths = collect_paths(path_ab); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 2); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, CastBehaviorSeparatesProjectionAndPruningRules) { - const auto int_type = i32(); - const auto bigint_type = i64(); - const auto float_type = f32(); - const auto double_type = f64(); - const auto decimal_small = dec32(8, 2); - const auto decimal_wide = dec32(9, 2); - const auto decimal_changed_scale = dec32(9, 3); - - const auto root_type = std::make_shared( - DataTypes {int_type, float_type, decimal_small}, Strings {"i", "f", "d"}); - const auto root = table_slot(0, 0, root_type, "s"); - const auto int_path = struct_element(root, int_type, "i"); - const auto float_path = struct_element(root, float_type, "f"); - const auto decimal_path = struct_element(root, decimal_small, "d"); - - auto paths = collect_paths(cast_expr(int_path, bigint_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - paths = collect_paths(cast_expr(float_path, double_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "f"); - - paths = collect_paths(cast_expr(decimal_path, decimal_wide)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "d"); - - paths = collect_paths( - cast_expr(struct_element(root, make_nullable(int_type), "i"), make_nullable(int_type))); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - // Unsafe casts are not accepted as pruning paths, but collect_nested_struct_paths() still - // recurses into children so scan projection can read the column needed by row-level filters. - paths = collect_paths(cast_expr(struct_element(root, bigint_type, "i"), int_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - paths = collect_paths(cast_expr(decimal_path, decimal_changed_scale)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "d"); - - EXPECT_TRUE(collect_paths(cast_expr(table_slot(1, 1, int_type, "plain"), bigint_type)).empty()); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, ProjectionMergeKeepsFilterOnlyPathAndDeduplicates) { - const auto int_type = i32(); - const auto string_type = str(); - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", int_type); - auto table_output = struct_name_col("s", {table_a}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", int_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b, name_col("c", string_type, 2)}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_output}, {}, {file_struct}).ok()); - - const auto path_b = - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "b"); - auto filter_expr = compound_predicate( - TExprOpcode::COMPOUND_AND, - binary_predicate(TExprOpcode::GT, path_b, - literal(int_type, Field::create_field(1))), - binary_predicate(TExprOpcode::LT, path_b, - literal(int_type, Field::create_field(10)))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_output}, &request).ok()); - - EXPECT_TRUE(request.non_predicate_columns.empty()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - ASSERT_FALSE(request.predicate_columns[0].project_all_children); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); -} - // Scenario: row-oriented readers such as CSV/Text cannot lazy-read predicate columns separately. // For a complex root that is both projected and referenced by a filter, the materialized mapper // keeps one non-predicate scan entry and asks the reader to read the full top-level struct. @@ -2409,41 +2133,6 @@ TEST(ColumnMapperLocalizeFiltersTest, ConstantFilterBuildsEntryWithoutFileScanCo mapper.mappings()[0].constant_index); } -TEST(ColumnMapperLocalizeFiltersTest, NestedFilterOnlyChildMergesIntoPredicateProjection) { - const auto int_type = i32(); - const auto string_type = str(); - - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"), 10); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.localize_filters({filter}, &request).ok()); - - EXPECT_TRUE(request.non_predicate_columns.empty()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - ASSERT_FALSE(request.predicate_columns[0].project_all_children); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); - ASSERT_EQ(request.local_positions.size(), 1); - EXPECT_EQ(request.local_positions.at(LocalColumnId(5)), LocalIndex(0)); - ASSERT_TRUE(mapper.filter_entries().at(GlobalIndex(0)).is_local()); - EXPECT_EQ(mapper.filter_entries().at(GlobalIndex(0)).local_index(), LocalIndex(0)); -} - TEST(ColumnMapperLocalizeFiltersTest, PreservesExistingScanStateWhenAddingPredicateColumn) { const auto int_type = i32(); const std::vector table_schema = { @@ -2542,24 +2231,24 @@ TEST(ColumnMapperScanRequestTest, OrdinaryPredicateSlotRetainsPayloadForScannerB EXPECT_TRUE(request.predicate_only_columns.empty()); } -TEST(ColumnMapperScanRequestTest, StructOutputAndFilterOnlyChildAreMerged) { +TEST(ColumnMapperScanRequestTest, StructAllAccessPathsAreEagerWithoutPredicateMapping) { const auto int_type = i32(); const auto string_type = str(); auto table_a = name_col("a", int_type); auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); + auto table_struct = struct_name_col("s", {table_a, table_b}); auto file_a = name_col("a", int_type, 0); auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); + auto file_c = name_col("c", int_type, 2); + auto file_struct = struct_name_col("s", {file_a, file_b, file_c}, 5); TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"), 10); + auto filter_expr = + int_gt(struct_element(table_slot(0, 0, table_struct.type, "s"), int_type, "a"), 10); TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), .global_indices = {GlobalIndex(0)}}; @@ -2573,159 +2262,6 @@ TEST(ColumnMapperScanRequestTest, StructOutputAndFilterOnlyChildAreMerged) { EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); } -TEST(ColumnMapperScanRequestTest, RenamedNestedPredicateTargetsMappedFileChild) { - const auto int_type = i32(); - - auto table_a = field_id_col("a", 1, int_type); - auto table_renamed_b = field_id_col("renamed_b", 2, int_type); - auto table_struct = struct_col("s", 10, {table_a, table_renamed_b}); - auto file_a = field_id_col("a", 1, int_type, 0); - auto file_b = field_id_col("b", 2, int_type, 1); - auto file_struct = struct_col("s", 10, {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, table_struct.type, "s"), int_type, "renamed_b"), 10); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, NestedInNullAndReverseComparisonFiltersAreMerged) { - const auto int_type = i32(); - const auto string_type = str(); - - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"); - auto in_filter = - in_predicate(nested_a, int_type, - {Field::create_field(5), Field::create_field(7)}); - auto reverse_filter = binary_predicate( - TExprOpcode::LT, literal(int_type, Field::create_field(3)), nested_a); - auto null_filter = null_predicate(nested_a, true); - auto not_null_filter = null_predicate(nested_a, false); - auto filter_expr = compound_predicate( - TExprOpcode::COMPOUND_AND, - compound_predicate(TExprOpcode::COMPOUND_AND, in_filter, reverse_filter), - compound_predicate(TExprOpcode::COMPOUND_AND, null_filter, not_null_filter)); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, NestedPredicateFilterThroughSafeCast) { - const auto file_int_type = i32(); - const auto table_bigint_type = i64(); - const auto string_type = str(); - - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = std::make_shared( - DataTypes {table_bigint_type, string_type}, Strings {"a", "b"}); - - auto file_a = name_col("a", file_int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct, "s"), file_int_type, "a"); - auto filter_expr = - binary_predicate(TExprOpcode::GT, cast_expr(nested_a, table_bigint_type), - literal(table_bigint_type, Field::create_field(5))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, UnsafeCastDoesNotBuildNestedPredicateFilter) { - const auto file_bigint_type = i64(); - const auto table_int_type = i32(); - const auto string_type = str(); - - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = std::make_shared( - DataTypes {table_int_type, string_type}, Strings {"a", "b"}); - - auto file_a = name_col("a", file_bigint_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct, "s"), file_bigint_type, "a"); - auto filter_expr = binary_predicate(TExprOpcode::GT, cast_expr(nested_a, table_int_type), - literal(table_int_type, Field::create_field(5))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); -} - -TEST(ColumnMapperScanRequestTest, DeepNestedPredicateTargetsLeafPath) { - const auto id_type = i32(); - const auto name_type = str(); - const auto string_type = str(); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - - auto full_table_inner_type = - std::make_shared(DataTypes {id_type, name_type}, Strings {"id", "n"}); - auto full_table_struct_type = std::make_shared( - DataTypes {full_table_inner_type, string_type}, Strings {"a", "b"}); - - auto file_id = name_col("id", id_type, 0); - auto file_name = name_col("n", name_type, 1); - auto file_a = struct_name_col("a", {file_id, file_name}, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_id = - struct_element(struct_element(table_slot(0, 0, full_table_struct_type, "s"), - full_table_inner_type, "a"), - id_type, "id"); - auto filter_expr = - in_predicate(nested_id, id_type, - {Field::create_field(5), Field::create_field(7)}); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - TEST(ColumnMapperScanRequestTest, ArrayStructProjectionPrunesElementChildren) { const auto int_type = i32(); const auto string_type = str(); @@ -2878,51 +2414,6 @@ TEST(ColumnMapperScanRequestTest, ArrayWrapperDoesNotBuildNestedPredicateFilter) EXPECT_TRUE(request.non_predicate_columns[0].children.empty()); } -// Scenario: a map value struct projects child `b`, while a row filter reads value child `a`. -// The filter is too complex to become a file-local nested predicate. Lazy demotion must move the -// merged projection to the non-predicate set without dropping either physical value child. -TEST(ColumnMapperScanRequestTest, MapFilterOnlyValueChildMergesWithOutputProjection) { - const auto key_type = i32(); - const auto int_type = i32(); - const auto string_type = str(); - - auto table_value_b = name_col("b", string_type); - auto table_value = struct_name_col("value", {table_value_b}); - auto table_map = map_col("m", -1, {table_value}, key_type, table_value.type); - set_name_identifiers(&table_map, 0); - - auto file_key = name_col("key", key_type, 0); - auto file_value_a = name_col("a", int_type, 0); - auto file_value_b = name_col("b", string_type, 1); - auto file_value = struct_name_col("value", {file_value_a, file_value_b}, 1); - auto file_map = map_col("m", -1, {file_key, file_value}, key_type, file_value.type, 0); - set_name_identifiers(&file_map, 0); - - auto full_value_type = - std::make_shared(DataTypes {int_type, string_type}, Strings {"a", "b"}); - auto full_map_type = std::make_shared(key_type, full_value_type); - auto value_expr = - struct_element(table_slot(0, 0, full_map_type, "m"), full_value_type, "value"); - auto filter_expr = int_gt(struct_element(value_expr, int_type, "a"), 5); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_map}, {}, {file_map}).ok()); - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_map}, &request).ok()); - - EXPECT_TRUE(request.predicate_columns.empty()); - ASSERT_EQ(request.non_predicate_columns.size(), 1); - const auto& projection = request.non_predicate_columns[0]; - EXPECT_EQ(projection.column_id(), LocalColumnId(0)); - ASSERT_FALSE(projection.project_all_children); - ASSERT_EQ(projection.children.size(), 1); - EXPECT_EQ(projection.children[0].local_id(), 1); - EXPECT_EQ(projection_ids(projection.children[0].children), std::vector({0, 1})); -} - // Scenario: when projected struct children are an in-order prefix of the file struct, the mapper can // read those physical children directly without rebuilding the file-side complex type. TEST(ColumnMapperScanRequestTest, MatchingProjectedStructDoesNotNeedComplexRematerialize) { @@ -2987,9 +2478,10 @@ TEST(ColumnMapperScanRequestTest, RenameOnlyProjectedStructDoesNotRebuildFilePro EXPECT_TRUE(mapper.mappings()[0].is_trivial); } -// Scenario: a row filter references an unprojected struct child, so the predicate projection is -// merged with the output projection and the mapper rebuilds the projected file struct type. -TEST(ColumnMapperScanRequestTest, PredicateProjectionRebuildsProjectedStructFileType) { +// Scenario: FE access paths are the sole contract for nested predicate projection. If a filter +// references a Struct child absent from all_access_paths and no predicate_access_paths were sent, +// File Scanner V2 must not infer and append that child from the expression. +TEST(ColumnMapperScanRequestTest, MissingPredicateAccessPathsDoNotInferStructProjection) { const auto int_type = i32(); const auto string_type = str(); @@ -3015,19 +2507,18 @@ TEST(ColumnMapperScanRequestTest, PredicateProjectionRebuildsProjectedStructFile FileScanRequest request; ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_TRUE(request.non_predicate_columns.empty()); - const auto& projection = request.predicate_columns[0]; + EXPECT_TRUE(request.predicate_columns.empty()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& projection = request.non_predicate_columns[0]; EXPECT_FALSE(projection.project_all_children); - EXPECT_EQ(projection_ids(projection.children), std::vector({0, 1, 2})); + EXPECT_EQ(projection_ids(projection.children), std::vector({0, 1})); const auto* mapped_type = assert_cast( remove_nullable(mapper.mappings()[0].file_type).get()); - ASSERT_EQ(mapped_type->get_elements().size(), 3); + ASSERT_EQ(mapped_type->get_elements().size(), 2); EXPECT_EQ(mapped_type->get_element_name(0), "a"); EXPECT_EQ(mapped_type->get_element_name(1), "b"); - EXPECT_EQ(mapped_type->get_element_name(2), "c"); - EXPECT_FALSE(mapper.mappings()[0].is_trivial); + EXPECT_TRUE(request.conjuncts.empty()); } // Scenario: a filter references a top-level column that is not projected by the query; the mapper @@ -3435,63 +2926,6 @@ TEST_F(ColumnMapperCastTest, NestedElementAtInPredicateUsesAllOrNothingLiteralRe EXPECT_TRUE(fallback_root->children()[2]->data_type()->equals(*table_bigint_type)); } -// Scenario: output projection reads one struct child while the row filter reads a different nested -// struct child. File-local conjunct rewrite must use the merged scan projection type. In the SQL -// shape below, `SELECT element_at(s, 'c') WHERE element_at(element_at(s, 'b'), 'cc') LIKE ...` -// reads file children `b.cc` and `c`; the localized inner `element_at(s, 'b')` returns -// `Struct(cc)`, not the full old file child `Struct(cc, new_dd)`. -TEST(ColumnMapperScanRequestTest, NestedElementAtConjunctUsesMergedScanProjectionChildType) { - const auto string_type = str(); - const auto int_type = i32(); - - auto table_cc = field_id_col("cc", 23, string_type); - auto table_new_dd = field_id_col("new_dd", 24, int_type); - auto table_b = struct_col("b", 20, {table_cc, table_new_dd}); - auto table_c = field_id_col("c", 25, string_type); - auto full_table_struct = struct_col("struct_column2", 19, {table_b, table_c}); - auto projected_table_struct = struct_col("struct_column2", 19, {table_c}); - - auto file_cc = field_id_col("cc", 23, string_type, 0); - auto file_new_dd = field_id_col("new_dd", 24, int_type, 1); - auto file_b = struct_col("b", 20, {file_cc, file_new_dd}, 0); - auto file_c = field_id_col("c", 25, string_type, 1); - auto file_struct = struct_col("new_struct_column", 19, {file_b, file_c}, 10); - - const auto table_slot_expr = table_slot(0, 0, full_table_struct.type, "struct_column2"); - const auto table_parent_expr = element_at(table_slot_expr, table_b.type, "b"); - const auto table_leaf_expr = element_at(table_parent_expr, string_type, "cc"); - auto filter_expr = like_expr(table_leaf_expr, "NestedC%"); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); - ASSERT_TRUE(mapper.create_mapping({projected_table_struct}, {}, {file_struct}).ok()); - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {projected_table_struct}, &request).ok()); - ASSERT_EQ(request.conjuncts.size(), 1); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(10)); - - const auto& localized_leaf = request.conjuncts[0]->root()->children()[0]; - ASSERT_EQ(localized_leaf->expr_name(), "element_at"); - const auto& localized_parent = localized_leaf->children()[0]; - ASSERT_EQ(localized_parent->expr_name(), "element_at"); - - const auto* localized_slot = - assert_cast(localized_parent->children()[0].get()); - EXPECT_EQ(localized_slot->column_name(), "new_struct_column"); - // The scan projection keeps the top-level file column id above, while the localized conjunct - // executes on the file-reader Block. The VSlotRef column id is therefore the block position of - // `new_struct_column` in this request, not the file schema id 10. - EXPECT_EQ(localized_slot->column_id(), 0); - - const auto* localized_parent_type = assert_cast( - remove_nullable(localized_parent->data_type()).get()); - ASSERT_EQ(localized_parent_type->get_elements().size(), 1); - EXPECT_EQ(localized_parent_type->get_element_name(0), "cc"); -} - // Scenario: struct child access through a computed map/array parent is not localized as a file // conjunct, because the projected value struct can have a different physical child order. TEST(ColumnMapperScanRequestTest, MapValuesStructChildConjunctStaysTableLevel) { @@ -4591,5 +4025,362 @@ TEST_F(ColumnMapperCastTest, ColumnMapperKeepsTableSlotIdWhenFileBlockPositionCh conjunct->close(); } +TEST(ColumnMapperTest, VariantAccessPathProjectsOnlyPhysicalTypedLeaf) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& root = request.non_predicate_columns[0]; + ASSERT_FALSE(root.project_all_children); + ASSERT_EQ(root.children.size(), 1); + EXPECT_EQ(root.children[0].local_id(), 2); + ASSERT_EQ(root.children[0].children.size(), 1); + EXPECT_EQ(root.children[0].children[0].local_id(), 0); + ASSERT_EQ(root.children[0].children[0].children.size(), 1); + EXPECT_EQ(root.children[0].children[0].children[0].local_id(), 1); + EXPECT_TRUE(root.children[0].children[0].children[0].project_all_children); +} + +TEST(ColumnMapperTest, PredicateAccessPathsCreateDeferredStructOutputProjection) { + auto table_a = field_id_col("a", 2, i64()); + auto table_b = field_id_col("b", 3, i64()); + auto table_struct = struct_col("s", 1, {table_a, table_b}); + table_struct.has_predicate_access_paths = true; + table_struct.predicate_children = {table_b}; + + auto file_a = field_id_col("a", 2, i64(), 0); + auto file_b = field_id_col("b", 3, i64(), 1); + auto file_struct = struct_col("s", 1, {file_a, file_b}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + auto b = struct_element(table_slot(0, 0, table_struct.type, "s"), i64(), "b"); + auto predicate = binary_predicate(TExprOpcode::GT, b, + literal(i64(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns.size(), 1) << request.debug_string(); + ASSERT_EQ(request.predicate_columns[0].children.size(), 1); + EXPECT_EQ(request.predicate_columns[0].children[0].local_id(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); + EXPECT_EQ(request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0))); +} + +TEST(ColumnMapperTest, PredicateAccessPathsCreateDeferredVariantRootProjection) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.has_predicate_access_paths = true; + table_variant.predicate_variant_access_paths = {{"typed_col"}}; + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + + auto typed_col = + element_at(table_slot(0, 0, table_variant.type, "v"), variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i64()), + literal(i64(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_variant}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_FALSE(request.predicate_columns[0].project_all_children); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); + EXPECT_EQ(request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0))); +} + +TEST(ColumnMapperTest, RowGroupRefreshPreservesTwoDeferredVariantRootLayouts) { + auto make_file_variant = [](const std::string& name, int32_t field_id, int32_t local_id) { + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, + 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto variant = field_id_col(name, field_id, variant_v2(), local_id); + variant.children = {name_col("metadata", varbinary(), 0), name_col("value", varbinary(), 1), + std::move(typed_value)}; + return variant; + }; + auto first = field_id_col("v1", 10, variant_v2()); + first.has_predicate_access_paths = true; + first.predicate_variant_access_paths = {{"typed_col"}}; + auto second = field_id_col("v2", 11, variant_v2()); + second.has_predicate_access_paths = true; + second.predicate_variant_access_paths = {{"typed_col"}}; + const std::vector table_columns {first, second}; + const std::vector file_columns {make_file_variant("v1", 10, 0), + make_file_variant("v2", 11, 1)}; + + std::vector filters; + for (int32_t index = 0; index < 2; ++index) { + auto typed_col = element_at( + table_slot(index, index, table_columns[index].type, table_columns[index].name), + variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i64()), + literal(i64(), Field::create_field(0))); + filters.push_back({.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(index)}}); + } + + ParquetColumnMapper initial_mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(initial_mapper.create_mapping(table_columns, {}, file_columns).ok()); + FileScanRequest initial_request; + ASSERT_TRUE(initial_mapper.create_scan_request(filters, table_columns, &initial_request).ok()); + EXPECT_EQ(initial_request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(initial_request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_EQ(initial_request.local_positions.at(LocalColumnId(1)), LocalIndex(2)); + EXPECT_EQ(initial_request.non_predicate_position(LocalColumnId(1)), LocalIndex(3)); + + ParquetColumnMapper refreshed_mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(refreshed_mapper.create_mapping(table_columns, {}, file_columns).ok()); + FileScanRequest refreshed_request; + ASSERT_TRUE(refreshed_mapper + .create_scan_request(filters, table_columns, &refreshed_request, nullptr, + &initial_request.local_positions, + &initial_request.non_predicate_positions) + .ok()); + EXPECT_EQ(refreshed_request.local_positions, initial_request.local_positions); + EXPECT_EQ(refreshed_request.non_predicate_positions, initial_request.non_predicate_positions); + ASSERT_EQ(refreshed_request.predicate_columns.size(), 2); + ASSERT_EQ(refreshed_request.non_predicate_columns.size(), 2); + for (size_t index = 0; index < 2; ++index) { + EXPECT_TRUE(same_local_column_index(refreshed_request.predicate_columns[index], + initial_request.predicate_columns[index])); + EXPECT_TRUE(same_local_column_index(refreshed_request.non_predicate_columns[index], + initial_request.non_predicate_columns[index])); + } +} + +TEST(ColumnMapperTest, NestedVariantAccessPathProjectsPhysicalTypedLeaf) { + auto table_variant = field_id_col("payload", 2, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + auto table_struct = struct_col("info", 1, {table_variant}); + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("payload", 2, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + auto file_struct = struct_col("info", 1, {std::move(file_variant)}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_struct}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& root = request.non_predicate_columns[0]; + ASSERT_FALSE(root.project_all_children); + ASSERT_EQ(root.children.size(), 1); + const auto& variant = root.children[0]; + EXPECT_EQ(variant.local_id(), 0); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].local_id(), 0); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); +} + +TEST(ColumnMapperTest, NestedVariantAllAccessPathKeepsPhysicalTypedLeaf) { + auto table_variant = field_id_col("payload", 2, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + auto table_struct = struct_col("info", 1, {table_variant}); + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("payload", 2, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + auto file_struct = struct_col("info", 1, {std::move(file_variant)}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + auto payload = + struct_element(table_slot(0, 0, table_struct.type, "info"), variant_v2(), "payload"); + auto typed_col = element_at(payload, variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i32()), + literal(i32(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + const auto& root = request.predicate_columns[0]; + ASSERT_EQ(root.children.size(), 1); + const auto& variant = root.children[0]; + ASSERT_FALSE(variant.project_all_children); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); +} + +TEST(ColumnMapperTest, ArrayAndMapNestedVariantPathsReachPhysicalTypedLeaf) { + auto make_file_variant = [](std::string name, int32_t field_id, int32_t local_id) { + auto wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, + 0); + auto typed = struct_name_col("typed_value", {std::move(wrapper)}, 2); + auto variant = field_id_col(name, field_id, variant_v2(), local_id); + variant.children = {name_col("metadata", varbinary(), 0), name_col("value", varbinary(), 1), + std::move(typed)}; + return variant; + }; + auto assert_variant_leaf = [](const LocalColumnIndex& variant) { + ASSERT_FALSE(variant.project_all_children); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); + }; + + { + auto table_element = field_id_col("element", 2, variant_v2()); + table_element.variant_access_paths = {{"typed_col"}}; + auto table_array = array_col("items", 1, table_element); + auto file_array = array_col("items", 1, make_file_variant("element", 2, 0), 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_array}, {}, {file_array}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_array}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns[0].children.size(), 1); + assert_variant_leaf(request.non_predicate_columns[0].children[0]); + } + + { + auto table_key = field_id_col("key", 2, str()); + auto table_value = field_id_col("value", 3, variant_v2()); + table_value.variant_access_paths = {{"typed_col"}}; + auto table_map = map_col("attributes", 1, {table_key, table_value}, str(), variant_v2()); + auto file_key = field_id_col("key", 2, str(), 0); + auto file_value = make_file_variant("value", 3, 1); + auto file_map = map_col("attributes", 1, {file_key, file_value}, str(), variant_v2(), 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_map}, {}, {file_map}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_map}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& children = request.non_predicate_columns[0].children; + const auto value_it = std::ranges::find_if( + children, [](const LocalColumnIndex& child) { return child.local_id() == 1; }); + ASSERT_NE(value_it, children.end()); + assert_variant_leaf(*value_it); + } +} + +TEST(ColumnMapperTest, VariantLeafProjectionRequiresLosslessObjectPath) { + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto dotted_wrapper = struct_name_col( + "a.b", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 1); + auto numeric_wrapper = struct_name_col( + "1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 2); + auto negative_numeric_wrapper = struct_name_col( + "-1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 3); + auto positive_numeric_wrapper = struct_name_col( + "+1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 4); + auto null_wrapper = struct_name_col( + "NULL", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 5); + auto typed_value = + struct_name_col("typed_value", + {std::move(field_wrapper), std::move(dotted_wrapper), + std::move(numeric_wrapper), std::move(negative_numeric_wrapper), + std::move(positive_numeric_wrapper), std::move(null_wrapper)}, + 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + for (const std::vector& path : + {std::vector {"a.b"}, std::vector {"a", "b"}, + std::vector {"1"}, std::vector {"-1"}, + std::vector {"+1"}, std::vector {"NULL"}}) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {path}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children) + << "unsafe Variant path must fall back to the complete physical subtree"; + } +} + +TEST(ColumnMapperTest, VariantLeafProjectionRequiresObjectTypedValue) { + auto element_wrapper = struct_name_col( + "element", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto array_typed_value = array_col("typed_value", -1, std::move(element_wrapper), 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(array_typed_value)}; + + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"element"}}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); +} + +TEST(ColumnMapperTest, VariantLeafProjectionDeclinesAmbiguousPrimitiveIdentity) { + auto field_wrapper = struct_name_col( + "binary_col", + {name_col("value", varbinary(), 0), name_col("typed_value", varbinary(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"binary_col"}}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); +} + } // namespace } // namespace doris::format diff --git a/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp b/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp index 2145b6ab60da43..fe1d871816ed26 100644 --- a/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp +++ b/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp @@ -147,6 +147,32 @@ TEST(ParquetBenchmarkScenariosTest, SelectionMatrixCoversIdentityAndSuccessiveCo } } +TEST(ParquetBenchmarkScenariosTest, NullableSelectionPairsLegacyAndFusedAcrossRowShapes) { + const auto scenarios = nullable_selection_scenarios(); + EXPECT_EQ(scenarios.size(), size_t {200}); + for (const int selectivity : {1, 10, 50, 90, 99}) { + for (const int null_percent : {0, 1, 10, 50, 90}) { + for (const auto selection_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto null_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto implementation : {NullableSelectionImplementation::LEGACY, + NullableSelectionImplementation::FUSED}) { + EXPECT_TRUE(std::ranges::any_of( + scenarios, + [&](const NullableSelectionScenario& scenario) { + return scenario.selectivity_percent == selectivity && + scenario.null_percent == null_percent && + scenario.selection_pattern == selection_pattern && + scenario.null_pattern == null_pattern && + scenario.implementation == implementation; + })) + << "missing nullable selection comparison shape"; + } + } + } + } + } +} + TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversNullableSparseAndProjectionAxes) { const auto scenarios = reader_scenarios(); // Keep the exact count aligned with the upstream complex-residual scenario retained by rebase. diff --git a/be/test/format_v2/parquet/parquet_reader_control_test.cpp b/be/test/format_v2/parquet/parquet_reader_control_test.cpp index e21439e885570d..37deb32d5bf37b 100644 --- a/be/test/format_v2/parquet/parquet_reader_control_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_control_test.cpp @@ -27,6 +27,7 @@ #include "core/column/column_string.h" #include "core/column/column_vector.h" #include "core/data_type/data_type_number.h" +#include "core/data_type_serde/parquet_decode_source.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_scan.h" #include "format_v2/parquet/reader/column_reader.h" @@ -186,6 +187,79 @@ TEST(SelectionVectorTest, IdentitySelectionDoesNotMaterializeFilter) { EXPECT_EQ(filter, nullptr); } +TEST(NativeNullableSelectionTest, BuildsPhysicalRangesAndSelectedNullsInOnePass) { + using native::FilterMap; + + const std::vector null_runs {2, 1, 3, 2, 2}; + const std::vector filter_data {1, 0, 1, 1, 0, 1, 1, 1, 0, 1}; + FilterMap filter; + ASSERT_TRUE(filter.init(filter_data.data(), filter_data.size(), false).ok()); + ParquetSelection selection; + NullMap output_nulls {1}; + NullMap selected_nulls; + size_t num_filtered = 0; + + ASSERT_TRUE(native::build_filtered_nullable_selection(null_runs, filter_data.size(), 3, + &output_nulls, &filter, 0, &selection, + &selected_nulls, &num_filtered) + .ok()); + + EXPECT_EQ(selection.total_values, 7); + EXPECT_EQ(selection.selected_values, 4); + ASSERT_EQ(selection.ranges.size(), 4); + EXPECT_EQ(selection.ranges[0].first, 0); + EXPECT_EQ(selection.ranges[0].count, 1); + EXPECT_EQ(selection.ranges[1].first, 2); + EXPECT_EQ(selection.ranges[1].count, 1); + EXPECT_EQ(selection.ranges[2].first, 4); + EXPECT_EQ(selection.ranges[2].count, 1); + EXPECT_EQ(selection.ranges[3].first, 6); + EXPECT_EQ(selection.ranges[3].count, 1); + EXPECT_EQ(selected_nulls, (NullMap {0, 1, 0, 0, 1, 1, 0})); + EXPECT_EQ(output_nulls, (NullMap {1, 0, 1, 0, 0, 1, 1, 0})); + EXPECT_EQ(num_filtered, 3); +} + +TEST(NativeNullableSelectionTest, UsesDirectPhysicalCoordinatesWithoutNulls) { + using native::FilterMap; + + const std::vector no_nulls {10}; + const std::vector filter_data {1, 1, 0, 1, 0, 0, 1, 1, 1, 0}; + FilterMap filter; + ASSERT_TRUE(filter.init(filter_data.data(), filter_data.size(), false).ok()); + ParquetSelection selection; + NullMap output_nulls; + NullMap selected_nulls; + size_t num_filtered = 0; + + ASSERT_TRUE(native::build_filtered_nullable_selection(no_nulls, filter_data.size(), 0, + &output_nulls, &filter, 0, &selection, + &selected_nulls, &num_filtered) + .ok()); + + EXPECT_EQ(selection.total_values, 10); + EXPECT_EQ(selection.selected_values, 6); + ASSERT_EQ(selection.ranges.size(), 3); + EXPECT_EQ(selection.ranges[0].first, 0); + EXPECT_EQ(selection.ranges[0].count, 2); + EXPECT_EQ(selection.ranges[1].first, 3); + EXPECT_EQ(selection.ranges[1].count, 1); + EXPECT_EQ(selection.ranges[2].first, 6); + EXPECT_EQ(selection.ranges[2].count, 3); + EXPECT_EQ(selected_nulls, (NullMap {0, 0, 0, 0, 0, 0})); + EXPECT_EQ(output_nulls, selected_nulls); + EXPECT_EQ(num_filtered, 4); +} + +TEST(NativeNullableSelectionTest, EnablesFusionOnlyForMateriallyFragmentedNullableBatches) { + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 0, 3)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 655, 1311)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 32768, 3)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(512, 256, 512)); + EXPECT_TRUE(native::should_use_fused_nullable_selection(65536, 6553, 13107)); + EXPECT_TRUE(native::should_use_fused_nullable_selection(65536, 32768, 65536)); +} + TEST(NativeNestedSelectionTest, BuildsSelectionAndCompactsSurvivingParentLevels) { using native::ColumnSelectVector; using native::FilterMap; diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp b/be/test/format_v2/parquet/parquet_reader_test.cpp index bc73055ba88c5a..abc67839d2dc14 100644 --- a/be/test/format_v2/parquet/parquet_reader_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_test.cpp @@ -48,17 +48,20 @@ #include "core/column/column_string.h" #include "core/column/column_struct.h" #include "core/column/column_vector.h" +#include "core/column/variant_v2/column_variant_v2.h" #include "core/data_type/data_type_array.h" #include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/data_type/primitive_type.h" #include "core/field.h" #include "exprs/vcompound_pred.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" #include "exprs/vslot_ref.h" #include "format_v2/column_mapper.h" #include "format_v2/expr/delete_predicate.h" @@ -152,6 +155,132 @@ class Int32GreaterThanExpr final : public VExpr { const std::string _expr_name = "Int32GreaterThanExpr"; }; +class VariantPathMetadataExpr : public VExpr { +public: + VariantPathMetadataExpr(std::string name, DataTypePtr type, + TExprNodeType::type node_type = TExprNodeType::FUNCTION_CALL) + : VExpr(std::move(type), false), _name(std::move(name)) { + set_node_type(node_type); + } + + const std::string& expr_name() const override { return _name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("VariantPathMetadataExpr is not executable"); + } + +private: + std::string _name; +}; + +class VariantInt32PathGreaterThanExpr final : public VariantPathMetadataExpr { +public: + VariantInt32PathGreaterThanExpr(int column_id, std::string key, int32_t value) + : VariantPathMetadataExpr("gt", std::make_shared(), + TExprNodeType::BINARY_PRED), + _column_id(column_id), + _key(std::move(key)), + _value(value) {} + + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, + size_t count, ColumnPtr& result_column) const override { + const auto& nullable = + assert_cast(*block->get_by_position(_column_id).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef(_key)}}; + const auto typed = variants.find_shredded_typed_value(path); + if (!typed.has_value()) { + return Status::InternalError("Expected the projected Variant typed leaf"); + } + const auto& typed_nullable = assert_cast(*typed->column); + const auto& values = + assert_cast(typed_nullable.get_nested_column()).get_data(); + auto result = ColumnUInt8::create(); + auto& output = result->get_data(); + output.resize(count); + for (size_t row = 0; row < count; ++row) { + const size_t input_row = selector == nullptr ? row : (*selector)[row]; + output[row] = !nullable.is_null_at(input_row) && + !typed_nullable.is_null_at(input_row) && values[input_row] > _value; + } + result_column = std::move(result); + return Status::OK(); + } + +private: + int _column_id; + std::string _key; + int32_t _value; +}; + +VExprContextSPtr create_variant_int32_path_greater_than_conjunct(int column_id, std::string key, + int32_t value) { + auto slot = VSlotRef::create_shared(0, column_id, -1, + make_nullable(std::make_shared()), "v"); + auto key_literal = VLiteral::create_shared(std::make_shared(), + Field::create_field(key)); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key_literal); + auto cast = std::make_shared( + "CAST", make_nullable(std::make_shared()), TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared(std::make_shared(), + Field::create_field(value)); + auto gt = std::make_shared(column_id, std::move(key), value); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +class StructInt32ChildGreaterThanExpr final : public VExpr { +public: + StructInt32ChildGreaterThanExpr(int column_id, int32_t value) + : VExpr(std::make_shared(), false), + _column_id(column_id), + _value(value) {} + + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, + size_t count, ColumnPtr& result_column) const override { + const auto& nullable = + assert_cast(*block->get_by_position(_column_id).column); + const auto& structure = assert_cast(nullable.get_nested_column()); + const auto& child = assert_cast(structure.get_column(0)); + const auto& values = assert_cast(child.get_nested_column()).get_data(); + auto result = ColumnUInt8::create(); + auto& output = result->get_data(); + output.resize(count); + for (size_t row = 0; row < count; ++row) { + const size_t input_row = selector == nullptr ? row : (*selector)[row]; + output[row] = !nullable.is_null_at(input_row) && !child.is_null_at(input_row) && + values[input_row] > _value; + } + result_column = std::move(result); + return Status::OK(); + } + + const std::string& expr_name() const override { return _expr_name; } + + void collect_slot_column_ids(std::set& column_ids) const override { + column_ids.insert(_column_id); + } + +private: + int _column_id; + int32_t _value; + const std::string _expr_name = "StructInt32ChildGreaterThanExpr"; +}; + +VExprContextSPtr create_struct_int32_child_greater_than_conjunct(int column_id, int32_t value) { + auto context = VExprContext::create_shared( + std::make_shared(column_id, value)); + context->_prepared = true; + context->_opened = true; + return context; +} + class Int32DictionaryEqualsExpr final : public VExpr { public: Int32DictionaryEqualsExpr(int column_id, int32_t value) @@ -1578,6 +1707,447 @@ TEST_F(NewParquetReaderTest, CreatesParquetColumnMapper) { ASSERT_NE(dynamic_cast(mapper.get()), nullptr); } +TEST(ParquetVariantProjectionTest, ResidualStatisticsGuardPhysicalLeafProjection) { + using format::parquet::ParquetColumnSchema; + using format::parquet::ParquetColumnSchemaKind; + auto node = [](std::string name, int32_t local_id, ParquetColumnSchemaKind kind, + int leaf_id = -1) { + auto result = std::make_unique(); + result->name = std::move(name); + result->local_id = local_id; + result->kind = kind; + result->leaf_column_id = leaf_id; + return result; + }; + auto root = node("v", 0, ParquetColumnSchemaKind::VARIANT); + root->children.push_back(node("metadata", 0, ParquetColumnSchemaKind::PRIMITIVE, 0)); + root->children.push_back(node("value", 1, ParquetColumnSchemaKind::PRIMITIVE, 1)); + auto root_typed = node("typed_value", 2, ParquetColumnSchemaKind::STRUCT); + auto wrapper = node("n", 0, ParquetColumnSchemaKind::STRUCT); + wrapper->children.push_back(node("value", 0, ParquetColumnSchemaKind::PRIMITIVE, 2)); + wrapper->children.push_back(node("typed_value", 1, ParquetColumnSchemaKind::PRIMITIVE, 3)); + root_typed->children.push_back(std::move(wrapper)); + root->children.push_back(std::move(root_typed)); + + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(1)); + + tparquet::RowGroup row_group; + row_group.__set_num_rows(10); + for (int leaf = 0; leaf < 4; ++leaf) { + tparquet::Statistics statistics; + statistics.__set_null_count(leaf == 1 || leaf == 2 ? 10 : 0); + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(std::move(column_metadata)); + row_group.columns.push_back(std::move(chunk)); + } + tparquet::FileMetaData metadata; + metadata.row_groups.push_back(row_group); + EXPECT_TRUE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); + + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(9); + EXPECT_FALSE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); + metadata.row_groups[0].columns[2].meta_data.__isset.statistics = false; + EXPECT_FALSE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); +} + +TEST(ParquetVariantProjectionTest, FinalizesNestedVariantProjectionRecursively) { + using format::parquet::ParquetColumnSchema; + using format::parquet::ParquetColumnSchemaKind; + auto node = [](std::string name, int32_t local_id, ParquetColumnSchemaKind kind, + int leaf_id = -1) { + auto result = std::make_unique(); + result->name = std::move(name); + result->local_id = local_id; + result->kind = kind; + result->leaf_column_id = leaf_id; + return result; + }; + auto root = node("info", 0, ParquetColumnSchemaKind::STRUCT); + auto variant = node("payload", 0, ParquetColumnSchemaKind::VARIANT); + variant->children.push_back(node("metadata", 0, ParquetColumnSchemaKind::PRIMITIVE, 0)); + variant->children.push_back(node("value", 1, ParquetColumnSchemaKind::PRIMITIVE, 1)); + auto typed_object = node("typed_value", 2, ParquetColumnSchemaKind::STRUCT); + auto wrapper = node("n", 0, ParquetColumnSchemaKind::STRUCT); + wrapper->children.push_back(node("value", 0, ParquetColumnSchemaKind::PRIMITIVE, 2)); + wrapper->children.push_back(node("typed_value", 1, ParquetColumnSchemaKind::PRIMITIVE, 3)); + typed_object->children.push_back(std::move(wrapper)); + variant->children.push_back(std::move(typed_object)); + root->children.push_back(std::move(variant)); + + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.back().children.push_back( + format::LocalColumnIndex::local(1)); + + tparquet::RowGroup row_group; + row_group.__set_num_rows(10); + for (int leaf = 0; leaf < 4; ++leaf) { + tparquet::Statistics statistics; + statistics.__set_null_count(leaf == 1 || leaf == 2 ? 10 : 0); + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(std::move(column_metadata)); + row_group.columns.push_back(std::move(chunk)); + } + tparquet::FileMetaData metadata; + metadata.row_groups.push_back(row_group); + + EXPECT_EQ( + format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &projection), + 1); + EXPECT_FALSE(projection.children[0].project_all_children); + + auto fallback = projection; + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(9); + EXPECT_EQ(format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &fallback), + 0); + EXPECT_TRUE(fallback.children[0].project_all_children); + + auto repeated = projection; + root->children[0]->max_repetition_level = 1; + EXPECT_EQ(format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &repeated), + 0); + EXPECT_TRUE(repeated.children[0].project_all_children); +} + +TEST_F(NewParquetReaderTest, ReadsFullyShreddedVariantTypedLeafProjection) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_typed_leaf_projection"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + ASSERT_EQ(remove_nullable(schema[1].type)->get_primitive_type(), TYPE_VARIANT); + + auto find_child = [](const std::vector& children, + std::string_view name) -> const format::ColumnDefinition* { + const auto it = std::ranges::find_if( + children, [name](const auto& child) { return child.name == name; }); + return it == children.end() ? nullptr : &*it; + }; + const auto* root_typed = find_child(schema[1].children, "typed_value"); + ASSERT_NE(root_typed, nullptr); + const auto* n_wrapper = find_child(root_typed->children, "n"); + ASSERT_NE(n_wrapper, nullptr); + const auto* n_typed = find_child(n_wrapper->children, "typed_value"); + ASSERT_NE(n_typed, nullptr); + + auto projection = format::LocalColumnIndex::partial_local(schema[1].local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(root_typed->local_id)); + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(n_wrapper->local_id)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(n_typed->local_id)); + auto request = std::make_shared(); + request->non_predicate_columns.push_back(std::move(projection)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(0)); + ASSERT_TRUE(reader->open(request).ok()); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 1); + + Block block; + block.insert({schema[1].type->create_column(), schema[1].type, "v"}); + size_t rows = 0; + bool eof = false; + while (!eof) { + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + } + ASSERT_EQ(rows, 4096); + const auto& nullable = assert_cast(*block.get_by_position(0).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("n")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->type->get_primitive_type(), TYPE_INT); + EXPECT_EQ(match->column->size(), rows); + ASSERT_NE(profile.get_counter("VariantDirectLeafRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantDirectLeafRows")->value(), rows); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), 0); + const std::array missing_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("missing")}}; + EXPECT_FALSE(variants.find_shredded_typed_value(missing_path).has_value()); + ASSERT_NE(profile.get_counter("VariantDirectLeafPathMisses"), nullptr); + EXPECT_EQ(profile.get_counter("VariantDirectLeafPathMisses")->value(), 1); + const auto first_value = + assert_cast( + assert_cast(*match->column).get_nested_column()) + .get_data()[0]; + + IColumn::Filter keep(rows, 0); + keep[0] = 1; + const ColumnPtr filtered = variants.filter(keep, 1); + const auto& filtered_variants = assert_cast(*filtered); + ASSERT_TRUE(filtered_variants.is_shredded()); + const auto filtered_match = filtered_variants.find_shredded_typed_value(path); + ASSERT_TRUE(filtered_match.has_value()); + EXPECT_EQ(filtered_match->column->size(), 1); + EXPECT_EQ( + assert_cast( + assert_cast(*filtered_match->column).get_nested_column()) + .get_data()[0], + first_value); + EXPECT_TRUE(variants.clone_resized(0)->empty()); + auto mutable_filtered = variants.clone_resized(variants.size()); + EXPECT_EQ(mutable_filtered->filter(keep), 1); + EXPECT_TRUE(assert_cast(*mutable_filtered).is_shredded()); + + // TableReader detaches mapped output columns before upper expressions run. Detachment must + // preserve an incomplete leaf projection because it has no canonical Variant to materialize. + auto detached = IColumn::mutate(block.get_by_position(0).column); + const auto& detached_variants = assert_cast( + assert_cast(*detached).get_nested_column()); + ASSERT_TRUE(detached_variants.is_shredded()); + ASSERT_TRUE(detached_variants.find_shredded_typed_value(path).has_value()); + + // Adaptive predicate probing cuts retained output columns into proper subsets. Keep that row + // selection in the physical shredded state as well. + const ColumnPtr sliced = variants.cut(1, 2); + const auto& sliced_variants = assert_cast(*sliced); + ASSERT_TRUE(sliced_variants.is_shredded()); + const auto sliced_match = sliced_variants.find_shredded_typed_value(path); + ASSERT_TRUE(sliced_match.has_value()); + ASSERT_EQ(sliced_match->column->size(), 2); + EXPECT_EQ(assert_cast( + assert_cast(*sliced_match->column).get_nested_column()) + .get_data()[0], + first_value + 1); + + const std::array indices {2, 0}; + MutableColumnPtr gathered = variants.clone_empty(); + gathered->insert_indices_from(variants, indices.data(), indices.data() + indices.size()); + const auto& gathered_variants = assert_cast(*gathered); + ASSERT_TRUE(gathered_variants.is_shredded()); + const auto gathered_match = gathered_variants.find_shredded_typed_value(path); + ASSERT_TRUE(gathered_match.has_value()); + ASSERT_EQ(gathered_match->column->size(), indices.size()); + EXPECT_EQ( + assert_cast( + assert_cast(*gathered_match->column).get_nested_column()) + .get_data()[0], + first_value + 2); +} + +TEST_F(NewParquetReaderTest, ShreddedVariantPredicateUsesTypedLeafPageIndexWithRootOutput) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_page_pruning_with_root_output"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto request = std::make_shared(); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[0].local_id))); + // The root output deliberately retains the complete wrapper; its predicate may still use the + // typed leaf's page index without converting the output to a leaf-only Variant projection. + request->predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[1].local_id))); + request->local_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(0)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_variant_int32_path_greater_than_conjunct(1, "n", 3000)); + ASSERT_TRUE(reader->open(request).ok()); + + size_t rows = 0; + bool eof = false; + while (!eof) { + Block block = build_file_block(schema); + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + if (batch_rows > 0) { + const auto& nullable = + assert_cast(*block.get_by_position(1).column); + auto canonical = IColumn::mutate(nullable.get_nested_column_ptr()); + assert_cast(*canonical).ensure_encoded(); + } + } + EXPECT_EQ(rows, 1095); + ASSERT_NE(profile.get_counter("FilteredRowsByPage"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByPage")->value(), 0); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 0); + ASSERT_NE(profile.get_counter("VariantDirectLeafRows"), nullptr); + EXPECT_GT(profile.get_counter("VariantDirectLeafRows")->value(), 0); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), rows); + ASSERT_NE(profile.get_counter("VariantReconstructionTime"), nullptr); + EXPECT_GT(profile.get_counter("VariantReconstructionTime")->value(), 0); +} + +TEST_F(NewParquetReaderTest, ReadsVariantPredicateLeafBeforeDeferredRootOutput) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_predicate_leaf_deferred_root"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto find_child = [](const std::vector& children, + std::string_view name) -> const format::ColumnDefinition* { + const auto it = std::ranges::find_if( + children, [name](const auto& child) { return child.name == name; }); + return it == children.end() ? nullptr : &*it; + }; + const auto* root_typed = find_child(schema[1].children, "typed_value"); + ASSERT_NE(root_typed, nullptr); + const auto* n_wrapper = find_child(root_typed->children, "n"); + ASSERT_NE(n_wrapper, nullptr); + const auto* n_typed = find_child(n_wrapper->children, "typed_value"); + ASSERT_NE(n_typed, nullptr); + + auto predicate_projection = format::LocalColumnIndex::partial_local(schema[1].local_id); + predicate_projection.children.push_back( + format::LocalColumnIndex::partial_local(root_typed->local_id)); + predicate_projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(n_wrapper->local_id)); + predicate_projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(n_typed->local_id)); + + auto request = std::make_shared(); + request->predicate_columns.push_back(std::move(predicate_projection)); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[1].local_id))); + request->predicate_only_columns.push_back(format::LocalColumnId(schema[1].local_id)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(0)); + request->non_predicate_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_variant_int32_path_greater_than_conjunct(0, "n", 3000)); + ASSERT_TRUE(reader->open(request).ok()); + + size_t rows = 0; + bool eof = false; + while (!eof) { + Block block; + block.insert({schema[1].type->create_column(), schema[1].type, "v_predicate"}); + block.insert({schema[1].type->create_column(), schema[1].type, "v_output"}); + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + ASSERT_EQ(block.get_by_position(0).column->size(), batch_rows); + ASSERT_EQ(block.get_by_position(1).column->size(), batch_rows); + if (batch_rows > 0) { + const auto& nullable = + assert_cast(*block.get_by_position(1).column); + auto canonical = IColumn::mutate(nullable.get_nested_column_ptr()); + assert_cast(*canonical).ensure_encoded(); + } + } + EXPECT_EQ(rows, 1095); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 1); + ASSERT_NE(profile.get_counter("FilteredRowsByLazyRead"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByLazyRead")->value(), 0); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), rows); +} + +TEST_F(NewParquetReaderTest, ReadsStructPredicateChildBeforeDeferredRootOutput) { + write_struct_filter_parquet_file(_file_path); + RuntimeProfile profile("struct_predicate_child_deferred_root"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 1); + ASSERT_EQ(schema[0].children.size(), 2); + + auto predicate_projection = format::LocalColumnIndex::partial_local(schema[0].local_id); + predicate_projection.children.push_back( + format::LocalColumnIndex::local(schema[0].children[0].local_id)); + auto request = std::make_shared(); + request->predicate_columns.push_back(predicate_projection); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[0].local_id))); + request->predicate_only_columns.push_back(format::LocalColumnId(schema[0].local_id)); + request->local_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(0)); + request->non_predicate_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_struct_int32_child_greater_than_conjunct(0, 2)); + ASSERT_TRUE(reader->open(request).ok()); + + format::ColumnDefinition predicate_field; + ASSERT_TRUE(format::project_column_definition(schema[0], predicate_projection, &predicate_field) + .ok()); + size_t total_rows = 0; + std::vector names; + bool eof = false; + while (!eof) { + Block block; + block.insert({predicate_field.type->create_column(), predicate_field.type, "s_predicate"}); + block.insert({schema[0].type->create_column(), schema[0].type, "s_output"}); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + total_rows += rows; + ASSERT_EQ(block.get_by_position(0).column->size(), rows); + ASSERT_EQ(block.get_by_position(1).column->size(), rows); + const auto& output_nullable = + assert_cast(*block.get_by_position(1).column); + const auto& output_struct = + assert_cast(output_nullable.get_nested_column()); + ASSERT_EQ(output_struct.tuple_size(), 2); + const auto& name_nullable = assert_cast(output_struct.get_column(1)); + const auto& name_values = + assert_cast(name_nullable.get_nested_column()); + for (size_t row = 0; row < rows; ++row) { + names.push_back(name_values.get_data_at(row).to_string()); + } + } + EXPECT_EQ(total_rows, 2); + EXPECT_EQ(names, (std::vector {"ten", "eleven"})); + ASSERT_NE(profile.get_counter("FilteredRowsByLazyRead"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByLazyRead")->value(), 0); +} + TEST_F(NewParquetReaderTest, CountComplexColumnUsesShapeOnlyPath) { write_nullable_map_parquet_file(_file_path); RuntimeProfile profile("count_map_shape_only_path"); diff --git a/be/test/format_v2/parquet/parquet_scan_test.cpp b/be/test/format_v2/parquet/parquet_scan_test.cpp index cc95b51cebf751..d63970c7bbf675 100644 --- a/be/test/format_v2/parquet/parquet_scan_test.cpp +++ b/be/test/format_v2/parquet/parquet_scan_test.cpp @@ -1678,6 +1678,17 @@ void write_page_index_parquet_file(const std::string& file_path) { write_table(file_path, table, ids.size(), false, true); } +void write_multi_column_page_index_parquet_file(const std::string& file_path) { + std::vector ascending(128); + std::iota(ascending.begin(), ascending.end(), 0); + std::vector descending(ascending.rbegin(), ascending.rend()); + auto schema = arrow::schema({arrow::field("ascending", arrow::int32(), false), + arrow::field("descending", arrow::int32(), false)}); + auto table = arrow::Table::Make(schema, + {build_int32_array(ascending), build_int32_array(descending)}); + write_table(file_path, table, ascending.size(), false, true); +} + void write_multi_row_group_page_index_parquet_file(const std::string& file_path) { std::vector ids(384); std::iota(ids.begin(), ids.end(), 0); @@ -4195,6 +4206,52 @@ TEST_F(ParquetScanTest, ProfileCountersReflectPageIndexAndRangeGapPruning) { EXPECT_GT(profile.get_counter("RangeGapSkippedRows")->value(), 0); } +TEST_F(ParquetScanTest, MultiColumnOrUsesPageIndexAndResidualExpression) { + write_multi_column_page_index_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(1)).ok()); + const auto low_ascending = create_int32_function_conjunct(0, "lt", TExprOpcode::LT, 13); + const auto low_descending = create_int32_function_conjunct(1, "lt", TExprOpcode::LT, 13); + auto disjunction = create_compound_conjunct(TExprOpcode::COMPOUND_OR, low_ascending->root(), + low_descending->root()); + ASSERT_TRUE(disjunction->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(disjunction->open(&state).ok()); + request->conjuncts.push_back(disjunction); + ASSERT_TRUE(reader->open(request).ok()); + + std::vector selected; + bool eof = false; + while (!eof) { + Block block = build_file_block(schema); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + const auto& values = int32_data_column(*block.get_by_position(0).column); + for (size_t row = 0; row < rows; ++row) { + selected.push_back(values.get_element(row)); + } + } + + std::vector expected(13); + std::iota(expected.begin(), expected.end(), 0); + for (int32_t value = 115; value < 128; ++value) { + expected.push_back(value); + } + EXPECT_EQ(selected, expected); + EXPECT_GT(counter_value(profile, "FilteredRowsByPage"), 0); + EXPECT_LT(counter_value(profile, "RawRowsRead"), 128); + EXPECT_GT(counter_value(profile, "RowsFilteredByConjunct"), 0); + disjunction->close(); +} + TEST_F(ParquetScanTest, OpenDefersPageIndexProbeToCurrentRowGroup) { write_multi_row_group_page_index_parquet_file(_file_path); RuntimeProfile profile("lazy_page_index_profile"); diff --git a/be/test/format_v2/parquet/parquet_schema_test.cpp b/be/test/format_v2/parquet/parquet_schema_test.cpp index 74781c0e532b31..acd5e8600789ec 100644 --- a/be/test/format_v2/parquet/parquet_schema_test.cpp +++ b/be/test/format_v2/parquet/parquet_schema_test.cpp @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -28,6 +29,7 @@ #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/data_type/primitive_type.h" #include "format_v2/parquet/native_schema_desc.h" #include "format_v2/parquet/native_schema_node.h" @@ -35,6 +37,406 @@ #include "format_v2/parquet/parquet_file_context.h" namespace doris::format::parquet { +namespace { + +std::vector unshredded_variant_schema( + std::optional specification_version = 1) { + tparquet::SchemaElement root; + root.__set_name("schema"); + root.__set_num_children(1); + + tparquet::SchemaElement variant; + variant.__set_name("payload"); + variant.__set_num_children(2); + variant.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + variant.__set_logicalType(tparquet::LogicalType()); + variant.logicalType.__set_VARIANT(tparquet::VariantType()); + if (specification_version.has_value()) { + variant.logicalType.VARIANT.__set_specification_version(*specification_version); + } + + tparquet::SchemaElement metadata; + metadata.__set_name("metadata"); + metadata.__set_type(tparquet::Type::BYTE_ARRAY); + metadata.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + + tparquet::SchemaElement value; + value.__set_name("value"); + value.__set_type(tparquet::Type::BYTE_ARRAY); + value.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + return {root, variant, metadata, value}; +} + +std::vector struct_with_variant_schema() { + auto variant_fields = unshredded_variant_schema(); + variant_fields[0].__set_name("info"); + variant_fields[0].__set_num_children(2); + variant_fields[0].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + variant_fields[1].__set_name("payload"); + + tparquet::SchemaElement root; + root.__set_name("schema"); + root.__set_num_children(1); + tparquet::SchemaElement label; + label.__set_name("label"); + label.__set_type(tparquet::Type::BYTE_ARRAY); + label.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + return {root, + variant_fields[0], + label, + variant_fields[1], + variant_fields[2], + variant_fields[3]}; +} + +std::vector shredded_object_variant_schema(bool signed_integer = true, + bool required_wrapper = true) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_object; + typed_object.__set_name("typed_value"); + typed_object.__set_num_children(1); + typed_object.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement field_wrapper; + field_wrapper.__set_name("field"); + field_wrapper.__set_num_children(2); + field_wrapper.__set_repetition_type(required_wrapper ? tparquet::FieldRepetitionType::REQUIRED + : tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement fallback; + fallback.__set_name("value"); + fallback.__set_type(tparquet::Type::BYTE_ARRAY); + fallback.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_integer; + typed_integer.__set_name("typed_value"); + typed_integer.__set_type(tparquet::Type::INT32); + typed_integer.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + if (!signed_integer) { + typed_integer.__set_logicalType(tparquet::LogicalType()); + typed_integer.logicalType.__set_INTEGER(tparquet::IntType()); + typed_integer.logicalType.INTEGER.__set_bitWidth(32); + typed_integer.logicalType.INTEGER.__set_isSigned(false); + } + schema.insert(schema.end(), {typed_object, field_wrapper, fallback, typed_integer}); + return schema; +} + +std::vector shredded_primitive_variant_schema( + tparquet::SchemaElement typed_value) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_name("typed_value"); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(typed_value)); + return schema; +} + +std::vector shredded_time_variant_schema(bool adjusted_to_utc, + bool millis) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(millis ? tparquet::Type::INT32 : tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIME(tparquet::TimeType()); + typed_value.logicalType.TIME.__set_isAdjustedToUTC(adjusted_to_utc); + typed_value.logicalType.TIME.__set_unit(tparquet::TimeUnit()); + if (millis) { + typed_value.logicalType.TIME.unit.__set_MILLIS(tparquet::MilliSeconds()); + } else { + typed_value.logicalType.TIME.unit.__set_MICROS(tparquet::MicroSeconds()); + } + schema.push_back(std::move(typed_value)); + return schema; +} + +std::vector shredded_array_variant_schema(bool include_value, + bool include_typed_value) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_array; + typed_array.__set_name("typed_value"); + typed_array.__set_num_children(1); + typed_array.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_array.__set_converted_type(tparquet::ConvertedType::LIST); + tparquet::SchemaElement list; + list.__set_name("list"); + list.__set_num_children(1); + list.__set_repetition_type(tparquet::FieldRepetitionType::REPEATED); + tparquet::SchemaElement element; + element.__set_name("element"); + element.__set_num_children(static_cast(include_value) + + static_cast(include_typed_value)); + element.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + schema.insert(schema.end(), {typed_array, list, element}); + if (include_value) { + tparquet::SchemaElement value; + value.__set_name("value"); + value.__set_type(tparquet::Type::BYTE_ARRAY); + value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(value)); + } + if (include_typed_value) { + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT32); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(typed_value)); + } + return schema; +} + +} // namespace + +TEST(ParquetSchemaTest, NativeSchemaRecognizesVariantLogicalGroup) { + for (const auto version : {std::optional {}, std::optional {1}}) { + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(unshredded_variant_schema(version)).ok()); + descriptor.assign_ids(); + + const auto* native_variant = descriptor.get_column(0); + ASSERT_NE(native_variant, nullptr); + // Native readers must keep seeing the physical group. The logical Variant mapping belongs + // to ParquetColumnSchema and must not make this group look like an unindexed scalar leaf. + EXPECT_EQ(remove_nullable(native_variant->data_type)->get_primitive_type(), TYPE_STRUCT); + ASSERT_EQ(native_variant->children.size(), 2); + EXPECT_EQ(native_variant->children[0].physical_column_index, 0); + EXPECT_EQ(native_variant->children[1].physical_column_index, 1); + EXPECT_EQ(descriptor.physical_fields_size(), 2); + + std::vector> fields; + const auto status = build_parquet_column_schema(descriptor, &fields); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(fields.size(), 1); + EXPECT_EQ(fields[0]->kind, ParquetColumnSchemaKind::VARIANT); + EXPECT_TRUE(fields[0]->contains_variant); + EXPECT_EQ(remove_nullable(fields[0]->type)->get_primitive_type(), TYPE_VARIANT); + EXPECT_NE(typeid_cast(remove_nullable(fields[0]->type).get()), + nullptr); + ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_EQ(fields[0]->children[0]->name, "metadata"); + EXPECT_EQ(fields[0]->children[1]->name, "value"); + } +} + +TEST(ParquetSchemaTest, NativeSchemaAcceptsRequiredAndOptionalVariantGroups) { + for (const auto repetition : + {tparquet::FieldRepetitionType::REQUIRED, tparquet::FieldRepetitionType::OPTIONAL}) { + auto schema = unshredded_variant_schema(); + schema[1].__set_repetition_type(repetition); + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + ASSERT_TRUE(status.ok()) << status; + ASSERT_NE(descriptor.get_column(0), nullptr); + } +} + +TEST(ParquetSchemaTest, NestedVariantPropagatesIntoParentLogicalType) { + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(struct_with_variant_schema()).ok()); + descriptor.assign_ids(); + + std::vector> fields; + const auto status = build_parquet_column_schema(descriptor, &fields); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(fields.size(), 1); + EXPECT_TRUE(fields[0]->contains_variant); + ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_FALSE(fields[0]->children[0]->contains_variant); + EXPECT_TRUE(fields[0]->children[1]->contains_variant); + const auto* info_type = + assert_cast(remove_nullable(fields[0]->type).get()); + ASSERT_EQ(info_type->get_elements().size(), 2); + EXPECT_EQ(remove_nullable(info_type->get_elements()[1])->get_primitive_type(), TYPE_VARIANT); +} + +TEST(ParquetSchemaTest, NativeSchemaRejectsUnsupportedVariantVersionAndMalformedLayout) { + NativeFieldDescriptor descriptor; + const auto version_status = descriptor.parse_from_thrift(unshredded_variant_schema(2)); + EXPECT_TRUE(version_status.is()) << version_status; + EXPECT_NE(version_status.to_string().find("Variant specification version 2"), + std::string::npos); + + auto missing_metadata = unshredded_variant_schema(); + missing_metadata[2].__set_name("not_metadata"); + const auto layout_status = descriptor.parse_from_thrift(missing_metadata); + EXPECT_TRUE(layout_status.is()) << layout_status; + EXPECT_NE(layout_status.to_string().find("metadata"), std::string::npos); + + auto optional_unshredded_value = unshredded_variant_schema(); + optional_unshredded_value[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + const auto repetition_status = descriptor.parse_from_thrift(optional_unshredded_value); + EXPECT_TRUE(repetition_status.is()) << repetition_status; + EXPECT_NE(repetition_status.to_string().find("required BYTE_ARRAY"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantPreservesUtcTimestampInstant) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIMESTAMP(tparquet::TimestampType()); + typed_value.logicalType.TIMESTAMP.__set_isAdjustedToUTC(true); + typed_value.logicalType.TIMESTAMP.__set_unit(tparquet::TimeUnit()); + typed_value.logicalType.TIMESTAMP.unit.__set_MICROS(tparquet::MicroSeconds()); + schema.push_back(std::move(typed_value)); + + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(schema).ok()); + const auto* variant = descriptor.get_column(0); + ASSERT_EQ(variant->children.size(), 3); + EXPECT_EQ(remove_nullable(variant->children[2].data_type)->get_primitive_type(), + TYPE_TIMESTAMPTZ); +} + +TEST(ParquetSchemaTest, NativeVariantValidatesEveryShreddedWrapperAndScalar) { + NativeFieldDescriptor descriptor; + const auto unsigned_status = + descriptor.parse_from_thrift(shredded_object_variant_schema(false, true)); + EXPECT_TRUE(unsigned_status.is()) << unsigned_status; + EXPECT_NE(unsigned_status.to_string().find("unsigned"), std::string::npos); + + const auto optional_wrapper_status = + descriptor.parse_from_thrift(shredded_object_variant_schema(true, false)); + EXPECT_TRUE(optional_wrapper_status.is()) << optional_wrapper_status; + EXPECT_NE(optional_wrapper_status.to_string().find("wrapper"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsDuplicateObjectFieldNames) { + auto schema = shredded_object_variant_schema(); + schema[4].__set_num_children(2); + schema.insert(schema.end(), {schema[5], schema[6], schema[7]}); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("duplicate"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsUnsupportedPrimitiveTypePairs) { + std::vector invalid_typed_values; + + tparquet::SchemaElement int96; + int96.__set_type(tparquet::Type::INT96); + invalid_typed_values.push_back(int96); + + tparquet::SchemaElement fixed_binary; + fixed_binary.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + fixed_binary.__set_type_length(16); + invalid_typed_values.push_back(fixed_binary); + + tparquet::SchemaElement json; + json.__set_type(tparquet::Type::BYTE_ARRAY); + json.__set_logicalType(tparquet::LogicalType()); + json.logicalType.__set_JSON(tparquet::JsonType()); + invalid_typed_values.push_back(json); + + tparquet::SchemaElement float16; + float16.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + float16.__set_type_length(2); + float16.__set_logicalType(tparquet::LogicalType()); + float16.logicalType.__set_FLOAT16(tparquet::Float16Type()); + invalid_typed_values.push_back(float16); + + tparquet::SchemaElement mismatched_integer; + mismatched_integer.__set_type(tparquet::Type::INT64); + mismatched_integer.__set_logicalType(tparquet::LogicalType()); + mismatched_integer.logicalType.__set_INTEGER(tparquet::IntType()); + mismatched_integer.logicalType.INTEGER.__set_bitWidth(16); + mismatched_integer.logicalType.INTEGER.__set_isSigned(true); + invalid_typed_values.push_back(mismatched_integer); + + tparquet::SchemaElement mismatched_decimal; + mismatched_decimal.__set_type(tparquet::Type::INT32); + mismatched_decimal.__set_logicalType(tparquet::LogicalType()); + mismatched_decimal.logicalType.__set_DECIMAL(tparquet::DecimalType()); + mismatched_decimal.logicalType.DECIMAL.__set_precision(10); + mismatched_decimal.logicalType.DECIMAL.__set_scale(2); + invalid_typed_values.push_back(mismatched_decimal); + + tparquet::SchemaElement bad_uuid; + bad_uuid.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + bad_uuid.__set_type_length(15); + bad_uuid.__set_logicalType(tparquet::LogicalType()); + bad_uuid.logicalType.__set_UUID(tparquet::UUIDType()); + invalid_typed_values.push_back(bad_uuid); + + for (auto& typed_value : invalid_typed_values) { + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift( + shredded_primitive_variant_schema(std::move(typed_value))); + EXPECT_FALSE(status.ok()) << "unsupported Variant typed_value pair was accepted"; + } +} + +TEST(ParquetSchemaTest, NativeVariantRejectsRepeatedOuterGroup) { + auto schema = unshredded_variant_schema(); + schema[1].__set_repetition_type(tparquet::FieldRepetitionType::REPEATED); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("repeated"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantAcceptsOmittedShreddedWrapperChildren) { + NativeFieldDescriptor descriptor; + + auto value_only_object = shredded_object_variant_schema(); + value_only_object[5].__set_num_children(1); + value_only_object.pop_back(); + ASSERT_TRUE(descriptor.parse_from_thrift(value_only_object).ok()); + + ASSERT_TRUE(descriptor.parse_from_thrift(shredded_array_variant_schema(true, false)).ok()); + ASSERT_TRUE(descriptor.parse_from_thrift(shredded_array_variant_schema(false, true)).ok()); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsNanosBeforeProjectionChoice) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIMESTAMP(tparquet::TimestampType()); + typed_value.logicalType.TIMESTAMP.__set_isAdjustedToUTC(false); + typed_value.logicalType.TIMESTAMP.__set_unit(tparquet::TimeUnit()); + typed_value.logicalType.TIMESTAMP.unit.__set_NANOS(tparquet::NanoSeconds()); + schema.push_back(std::move(typed_value)); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("TIMESTAMP(NANOS)"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsUnsupportedTimeAnnotations) { + NativeFieldDescriptor descriptor; + const auto adjusted_status = + descriptor.parse_from_thrift(shredded_time_variant_schema(true, false)); + EXPECT_TRUE(adjusted_status.is()) << adjusted_status; + EXPECT_NE(adjusted_status.to_string().find("isAdjustedToUTC"), std::string::npos); + + const auto millis_status = + descriptor.parse_from_thrift(shredded_time_variant_schema(false, true)); + EXPECT_TRUE(millis_status.is()) << millis_status; + EXPECT_NE(millis_status.to_string().find("TIME(MILLIS)"), std::string::npos); +} + TEST(ParquetSchemaTest, NativeMetadataAcceptsRequiredRootWithoutColumns) { tparquet::SchemaElement root; root.__set_name("schema"); @@ -84,9 +486,12 @@ TEST(ParquetSchemaTest, NativeMetadataTreePreservesNestedFieldNamesAndIds) { std::vector> fields; ASSERT_TRUE(build_parquet_column_schema(native_schema, &fields).ok()); ASSERT_EQ(fields.size(), 1); + EXPECT_FALSE(fields[0]->contains_variant); EXPECT_EQ(fields[0]->name, "protocol"); EXPECT_EQ(fields[0]->parquet_field_id, 10); ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_FALSE(fields[0]->children[0]->contains_variant); + EXPECT_FALSE(fields[0]->children[1]->contains_variant); EXPECT_EQ(fields[0]->children[0]->name, "minReaderVersion"); EXPECT_EQ(fields[0]->children[0]->leaf_column_id, 0); EXPECT_EQ(fields[0]->children[1]->name, "minWriterVersion"); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 740892367bb04e..6b48b05d61cd2f 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -37,10 +37,13 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_time.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" +#include "exprs/vcompound_pred.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" #include "exprs/vslot_ref.h" #include "format_v2/file_reader.h" #include "format_v2/parquet/parquet_column_schema.h" @@ -165,6 +168,41 @@ class MetadataInt32GreaterThanExpr final : public VExpr { const std::string _expr_name = "MetadataInt32GreaterThanExpr"; }; +class MetadataSlotInt32GreaterThanExpr final : public VExpr { +public: + MetadataSlotInt32GreaterThanExpr(int slot_index, int32_t value) + : VExpr(std::make_shared(), false), + _slot_index(slot_index), + _value(value) {} + + const std::string& expr_name() const override { return _expr_name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("MetadataSlotInt32GreaterThanExpr is metadata-only"); + } + bool can_evaluate_zonemap_filter() const override { return true; } + void collect_slot_column_ids(std::set& column_ids) const override { + column_ids.insert(_slot_index); + } + ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override { + const auto zone_map = ctx.zone_map(_slot_index); + if (zone_map == nullptr) { + return unsupported_zonemap_filter(ctx); + } + if (!zone_map->has_not_null) { + return ZoneMapFilterResult::kNoMatch; + } + return zone_map->max_value <= Field::create_field(_value) + ? ZoneMapFilterResult::kNoMatch + : ZoneMapFilterResult::kMayMatch; + } + +private: + int _slot_index; + int32_t _value; + const std::string _expr_name = "MetadataSlotInt32GreaterThanExpr"; +}; + class MetadataBoundsProbeExpr final : public VExpr { public: explicit MetadataBoundsProbeExpr(bool require_false_boolean = false) @@ -196,6 +234,107 @@ class MetadataBoundsProbeExpr final : public VExpr { bool _require_false_boolean; const std::string _expr_name = "MetadataBoundsProbeExpr"; }; + +class VariantPathTestExpr final : public VExpr { +public: + VariantPathTestExpr(std::string name, DataTypePtr type, + TExprNodeType::type node_type = TExprNodeType::FUNCTION_CALL) + : VExpr(std::move(type), false), _name(std::move(name)) { + set_node_type(node_type); + } + + const std::string& expr_name() const override { return _name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("VariantPathTestExpr is metadata-only"); + } + +private: + std::string _name; +}; + +VExprContextSPtr variant_path_gt_conjunct(int32_t literal_value, + bool add_narrowing_intermediate_cast = false, + bool decimal_comparison = false) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + DataTypePtr comparison_type = decimal_comparison + ? DataTypePtr(std::make_shared(38, 9)) + : DataTypePtr(std::make_shared()); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + if (add_narrowing_intermediate_cast) { + auto narrowing = std::make_shared( + "CAST", make_nullable(std::make_shared()), TExprNodeType::CAST_EXPR); + narrowing->add_child(element_at); + cast->add_child(narrowing); + } else { + cast->add_child(element_at); + } + auto literal = decimal_comparison + ? VLiteral::create_shared( + comparison_type, + Field::create_field(Decimal128V3( + static_cast<__int128>(literal_value) * 1'000'000'000))) + : VLiteral::create_shared(comparison_type, + Field::create_field(literal_value)); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +VExprContextSPtr variant_path_float_gt_conjunct(float literal_value) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + auto comparison_type = std::make_shared(); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared(comparison_type, + Field::create_field(literal_value)); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +VExprContextSPtr variant_path_string_gt_conjunct(std::string literal_value) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + auto comparison_type = std::make_shared(); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared( + comparison_type, Field::create_field(std::move(literal_value))); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + VExprContextSPtrs bloom_conjuncts(DataTypePtr data_type, std::vector values) { return {VExprContext::create_shared( std::make_shared(0, std::move(data_type), std::move(values)))}; @@ -331,8 +470,9 @@ TEST(NativeParquetStatisticsTest, InvalidTimeAndPaddedBooleanPageBoundsCannotPru page_indexes.emplace(0, std::move(page_index)); std::vector selected_ranges; std::map skip_plans; + tparquet::RowGroup row_group; EXPECT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, + metadata, row_group, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, nullptr) .ok()); return selected_ranges; @@ -362,6 +502,141 @@ TEST(NativeParquetStatisticsTest, InvalidTimeAndPaddedBooleanPageBoundsCannotPru EXPECT_EQ(bool_ranges[0].start, 0); EXPECT_EQ(bool_ranges[0].length, 1); } + +TEST(NativeParquetStatisticsTest, MultiColumnOrUnionsPageIndexRanges) { + auto encode_int32 = [](int32_t value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto make_schema = [](int local_id, int leaf_column_id) { + auto column = std::make_unique(); + column->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + column->local_id = local_id; + column->leaf_column_id = leaf_column_id; + column->type = std::make_shared(); + column->type_descriptor.doris_type = column->type; + column->type_descriptor.physical_type = tparquet::Type::INT32; + return column; + }; + auto make_page_index = [&](const std::vector& values) { + format::parquet::NativeParquetPageIndex page_index; + std::vector encoded; + encoded.reserve(values.size()); + for (const auto value : values) { + encoded.push_back(encode_int32(value)); + } + page_index.column_index.__set_min_values(encoded); + page_index.column_index.__set_max_values(encoded); + page_index.column_index.__set_null_pages(std::vector(values.size(), false)); + page_index.column_index.__set_null_counts(std::vector(values.size(), 0)); + std::vector locations; + for (size_t page_idx = 0; page_idx < values.size(); ++page_idx) { + tparquet::PageLocation location; + location.__set_offset(static_cast(page_idx * 100)); + location.__set_compressed_page_size(100); + location.__set_first_row_index(static_cast(page_idx * 10)); + locations.push_back(location); + } + page_index.offset_index.__set_page_locations(std::move(locations)); + return page_index; + }; + + std::vector> schema; + schema.push_back(make_schema(0, 0)); + schema.push_back(make_schema(1, 1)); + tparquet::ColumnOrder order; + order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder()); + tparquet::FileMetaData metadata; + metadata.__set_column_orders({order, order}); + + auto make_compound_expr = [](TExprOpcode::type opcode, VExprSPtr left, VExprSPtr right) { + TExprNode compound_node; + compound_node.__set_node_type(TExprNodeType::COMPOUND_PRED); + compound_node.__set_opcode(opcode); + compound_node.__set_type(std::make_shared()->to_thrift()); + compound_node.__set_num_children(2); + compound_node.__set_is_nullable(false); + auto compound = VCompoundPred::create_shared(compound_node); + compound->add_child(std::move(left)); + compound->add_child(std::move(right)); + return compound; + }; + auto make_compound = [&](TExprOpcode::type opcode) { + return VExprContext::create_shared(make_compound_expr( + opcode, std::make_shared(0, 50), + std::make_shared(1, 50))); + }; + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.local_positions.emplace(format::LocalColumnId(1), format::LocalIndex(1)); + request.predicate_columns = {format::LocalColumnIndex::top_level(format::LocalColumnId(0)), + format::LocalColumnIndex::top_level(format::LocalColumnId(1))}; + request.conjuncts = {make_compound(TExprOpcode::COMPOUND_OR)}; + + std::unordered_map page_indexes; + page_indexes.emplace(0, make_page_index({100, 0, 0})); + page_indexes.emplace(1, make_page_index({0, 0, 100})); + std::vector selected_ranges; + std::map skip_plans; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 2); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 10); + EXPECT_EQ(selected_ranges[1].start, 20); + EXPECT_EQ(selected_ranges[1].length, 10); + + // A compound predicate after the safety fence must not participate in metadata pruning. + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 30); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + + page_indexes.erase(1); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 30); + + request.conjuncts = {make_compound(TExprOpcode::COMPOUND_AND)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 10); + + page_indexes.emplace(1, make_page_index({0, 0, 100})); + auto first_branch = make_compound_expr( + TExprOpcode::COMPOUND_AND, std::make_shared(0, 50), + std::make_shared(1, 50)); + auto second_branch = make_compound_expr( + TExprOpcode::COMPOUND_AND, std::make_shared(0, -1), + std::make_shared(1, 50)); + request.conjuncts = {VExprContext::create_shared(make_compound_expr( + TExprOpcode::COMPOUND_OR, std::move(first_branch), std::move(second_branch)))}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 20); + EXPECT_EQ(selected_ranges[0].length, 10); +} + TEST(ParquetBloomFilterPruningTest, NativeUint32BloomUsesPhysicalInt32Hash) { const auto column_schema = uint32_parquet_bloom_schema(); format::parquet::native::BlockSplitBloomFilter bloom_filter; @@ -650,8 +925,8 @@ TEST(NativeParquetStatisticsTest, TypeDefinedBoundsRequireSupportedColumnOrder) std::vector selected_ranges; std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr) .ok()); EXPECT_EQ(selected_ranges.size(), 1); @@ -664,8 +939,8 @@ TEST(NativeParquetStatisticsTest, TypeDefinedBoundsRequireSupportedColumnOrder) .ok()); EXPECT_TRUE(selected_row_groups.empty()); ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr) .ok()); EXPECT_TRUE(selected_ranges.empty()); } @@ -737,8 +1012,8 @@ TEST(NativeParquetStatisticsTest, ZonemapPruningIgnoresDisabledSessionSwitch) { std::vector selected_ranges; std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr, nullptr, &state) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr, nullptr, &state) .ok()); EXPECT_TRUE(selected_ranges.empty()); } @@ -780,8 +1055,8 @@ TEST(NativeParquetStatisticsTest, ContradictoryAllNullPageCountsDisablePruning) std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 10, &selected_ranges, - &skip_plans, nullptr) + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 10, + &selected_ranges, &skip_plans, nullptr) .ok()); // ColumnIndex is optional. An impossible all-null claim must fall back to reading the // ten-row data page instead of proving that no value can satisfy the predicate. @@ -791,5 +1066,342 @@ TEST(NativeParquetStatisticsTest, ContradictoryAllNullPageCountsDisablePruning) } } +TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) { + auto encode_int32 = [](int32_t value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto primitive = [](std::string name, int local_id, int leaf_id) { + auto schema = std::make_unique(); + schema->name = std::move(name); + schema->local_id = local_id; + schema->leaf_column_id = leaf_id; + schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + schema->type = make_nullable(std::make_shared()); + schema->type_descriptor.doris_type = schema->type; + schema->type_descriptor.physical_type = tparquet::Type::INT32; + return schema; + }; + auto bytes = [&](std::string name, int local_id, int leaf_id) { + auto schema = primitive(std::move(name), local_id, leaf_id); + schema->type = make_nullable(std::make_shared()); + schema->type_descriptor.doris_type = schema->type; + schema->type_descriptor.physical_type = tparquet::Type::BYTE_ARRAY; + return schema; + }; + + auto variant = std::make_unique(); + variant->name = "v"; + variant->local_id = 0; + variant->kind = format::parquet::ParquetColumnSchemaKind::VARIANT; + variant->contains_variant = true; + variant->type = make_nullable(std::make_shared()); + variant->children.push_back(bytes("metadata", 0, 0)); + variant->children.push_back(bytes("value", 1, 1)); + auto typed_object = std::make_unique(); + typed_object->name = "typed_value"; + typed_object->local_id = 2; + typed_object->kind = format::parquet::ParquetColumnSchemaKind::STRUCT; + auto field = std::make_unique(); + field->name = "col"; + field->local_id = 0; + field->kind = format::parquet::ParquetColumnSchemaKind::STRUCT; + field->children.push_back(bytes("value", 0, 2)); + field->children.push_back(primitive("typed_value", 1, 3)); + typed_object->children.push_back(std::move(field)); + variant->children.push_back(std::move(typed_object)); + std::vector> schema; + schema.push_back(std::move(variant)); + + auto chunk = [&](tparquet::Type::type type, int64_t num_values, int64_t null_count, + std::optional min_value = std::nullopt, + std::optional max_value = std::nullopt) { + tparquet::Statistics statistics; + statistics.__set_null_count(null_count); + if (min_value.has_value() && max_value.has_value()) { + statistics.__set_min_value(encode_int32(*min_value)); + statistics.__set_max_value(encode_int32(*max_value)); + } + tparquet::ColumnMetaData metadata; + metadata.__set_type(type); + metadata.__set_num_values(num_values); + metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk result; + result.__set_meta_data(std::move(metadata)); + return result; + }; + tparquet::RowGroup row_group; + row_group.__set_num_rows(100); + row_group.__set_columns({chunk(tparquet::Type::BYTE_ARRAY, 100, 0), + chunk(tparquet::Type::BYTE_ARRAY, 100, 100), + chunk(tparquet::Type::BYTE_ARRAY, 100, 100), + chunk(tparquet::Type::INT32, 100, 0, 1, 200)}); + tparquet::ColumnOrder order; + order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder()); + tparquet::FileMetaData metadata; + metadata.__set_row_groups({row_group}); + metadata.__set_column_orders({order, order, order, order}); + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.predicate_columns = {format::LocalColumnIndex::top_level(format::LocalColumnId(0))}; + request.conjuncts = {variant_path_gt_conjunct(50)}; + + auto footer_only_metadata = metadata; + footer_only_metadata.row_groups[0].columns[3].meta_data.statistics.__set_max_value( + encode_int32(2)); + std::vector selected_row_groups; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + + // The same predicate can be localized after an earlier unsafe conjunct. Metadata pruning must + // preserve that earlier expression's row-level error instead of skipping the whole row group. + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + + auto leaf_projection = format::LocalColumnIndex::partial_local(0); + auto typed_object_projection = format::LocalColumnIndex::partial_local(2); + auto field_projection = format::LocalColumnIndex::partial_local(0); + field_projection.children.push_back(format::LocalColumnIndex::local(1)); + typed_object_projection.children.push_back(std::move(field_projection)); + leaf_projection.children.push_back(std::move(typed_object_projection)); + request.predicate_columns = {std::move(leaf_projection)}; + for (int leaf = 0; leaf < 4; ++leaf) { + footer_only_metadata.row_groups[0].columns[leaf].meta_data.__set_total_compressed_size( + (leaf + 1) * 10); + } + format::parquet::ParquetPruningStats leaf_pruning_stats; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + &leaf_pruning_stats, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + EXPECT_EQ(leaf_pruning_stats.filtered_bytes, 40); + + request.conjuncts = {variant_path_gt_conjunct(50, false, true)}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + request.conjuncts = {variant_path_gt_conjunct(50)}; + + // Missing typed statistics provide no proof and must retain the row group. + footer_only_metadata.row_groups[0].columns[3].meta_data.__isset.statistics = false; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + // A populated fallback for the same path invalidates both footer and page pruning. + footer_only_metadata = metadata; + footer_only_metadata.row_groups[0].columns[3].meta_data.statistics.__set_max_value( + encode_int32(2)); + footer_only_metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + // A contradictory non-repeated value count cannot prove that every row lacks fallback bytes. + footer_only_metadata.row_groups[0].columns[2].meta_data.__set_num_values(99); + footer_only_metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex typed_pages; + typed_pages.column_index.__set_min_values({encode_int32(1), encode_int32(100)}); + typed_pages.column_index.__set_max_values({encode_int32(2), encode_int32(200)}); + typed_pages.column_index.__set_null_pages({false, false}); + typed_pages.column_index.__set_null_counts({0, 0}); + tparquet::PageLocation first; + first.__set_offset(0); + first.__set_compressed_page_size(10); + first.__set_first_row_index(0); + tparquet::PageLocation second; + second.__set_offset(10); + second.__set_compressed_page_size(10); + second.__set_first_row_index(50); + typed_pages.offset_index.__set_page_locations({first, second}); + std::unordered_map page_indexes; + page_indexes.emplace(3, std::move(typed_pages)); + + std::vector selected_ranges; + std::map skip_plans; + format::parquet::ParquetPruningStats pruning_stats; + ASSERT_TRUE(format::parquet::can_use_parquet_page_index(request, nullptr)); + TQueryOptions query_options; + query_options.__set_enable_expr_zonemap_filter(false); + RuntimeState generic_zonemap_disabled {query_options, TQueryGlobals()}; + EXPECT_TRUE(format::parquet::can_use_parquet_page_index(request, &generic_zonemap_disabled)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, &pruning_stats, nullptr, + &generic_zonemap_disabled) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 50); + EXPECT_EQ(selected_ranges[0].length, 50); + EXPECT_EQ(pruning_stats.page_index_read_calls, 1); + EXPECT_EQ(pruning_stats.filtered_page_rows, 50); + + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + + // Direct Variant numeric comparisons coerce integral literals to a wide DECIMAL domain. + request.conjuncts = {variant_path_gt_conjunct(50, false, true)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 50); + EXPECT_EQ(selected_ranges[0].length, 50); + + // Bounds for the raw INT32 typed leaf are not valid for CAST(CAST(v['col'] AS TINYINT) AS INT). + request.conjuncts = {variant_path_gt_conjunct(50, true)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + + // Raw binary and UUID bounds are physical bytes, while the residual Variant-to-STRING cast + // compares their rendered values. Those domains differ, so neither footer nor page metadata + // may exclude a row solely from the raw byte interval. + auto assert_binary_identity_does_not_prune = [&](bool is_uuid) { + auto* binary_leaf = schema[0]->children[2]->children[0]->children[1].get(); + binary_leaf->type = make_nullable(std::make_shared()); + binary_leaf->type_descriptor = {}; + binary_leaf->type_descriptor.doris_type = binary_leaf->type; + binary_leaf->type_descriptor.physical_type = + is_uuid ? tparquet::Type::FIXED_LEN_BYTE_ARRAY : tparquet::Type::BYTE_ARRAY; + binary_leaf->type_descriptor.fixed_length = is_uuid ? 16 : -1; + binary_leaf->type_descriptor.is_string_like = true; + binary_leaf->type_descriptor.is_uuid = is_uuid; + + auto& binary_chunk = metadata.row_groups[0].columns[3].meta_data; + binary_chunk.__set_type(binary_leaf->type_descriptor.physical_type); + binary_chunk.statistics.__set_min_value("a"); + binary_chunk.statistics.__set_max_value("b"); + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(100); + request.conjuncts = {variant_path_string_gt_conjunct("z")}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex binary_pages; + binary_pages.column_index.__set_min_values({"a"}); + binary_pages.column_index.__set_max_values({"b"}); + binary_pages.column_index.__set_null_pages({false}); + binary_pages.column_index.__set_null_counts({0}); + tparquet::PageLocation binary_location; + binary_location.__set_offset(0); + binary_location.__set_compressed_page_size(10); + binary_location.__set_first_row_index(0); + binary_pages.offset_index.__set_page_locations({binary_location}); + page_indexes.clear(); + page_indexes.emplace(3, std::move(binary_pages)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + }; + assert_binary_identity_does_not_prune(false); + assert_binary_identity_does_not_prune(true); + + // Parquet floating min/max omits NaN values. Without an explicit no-NaN proof, [0, NaN] + // cannot be represented by max=0 and must not prune a Variant comparison that retains NaN. + auto encode_float = [](float value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto* float_leaf = schema[0]->children[2]->children[0]->children[1].get(); + float_leaf->type = make_nullable(std::make_shared()); + float_leaf->type_descriptor.doris_type = float_leaf->type; + float_leaf->type_descriptor.physical_type = tparquet::Type::FLOAT; + auto& float_chunk = metadata.row_groups[0].columns[3].meta_data; + float_chunk.__set_type(tparquet::Type::FLOAT); + float_chunk.statistics.__set_min_value(encode_float(0.0F)); + float_chunk.statistics.__set_max_value(encode_float(0.0F)); + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(100); + request.conjuncts = {variant_path_float_gt_conjunct(1.0F)}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, false, nullptr, + nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex float_pages; + float_pages.column_index.__set_min_values({encode_float(0.0F)}); + float_pages.column_index.__set_max_values({encode_float(0.0F)}); + float_pages.column_index.__set_null_pages({false}); + float_pages.column_index.__set_null_counts({0}); + tparquet::PageLocation float_location; + float_location.__set_offset(0); + float_location.__set_compressed_page_size(10); + float_location.__set_first_row_index(0); + float_pages.offset_index.__set_page_locations({float_location}); + page_indexes.clear(); + page_indexes.emplace(3, std::move(float_pages)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + + // A fallback value in the same row group may have a different Variant type. In that case the + // typed bounds cannot prove anything about the SQL comparison, so all pages must be read. + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + request.conjuncts = {variant_path_gt_conjunct(50)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); +} + } // namespace } // namespace doris diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp new file mode 100644 index 00000000000000..51a50319285c5b --- /dev/null +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -0,0 +1,1822 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/parquet/reader/variant_column_reader.h" + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "common/exception.h" +#include "core/assert_cast.h" +#include "core/column/column_array.h" +#include "core/column/column_decimal.h" +#include "core/column/column_map.h" +#include "core/column/column_nullable.h" +#include "core/column/column_string.h" +#include "core/column/column_struct.h" +#include "core/column/variant_v2/column_variant_v2.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_date_or_datetime_v2.h" +#include "core/data_type/data_type_decimal.h" +#include "core/data_type/data_type_map.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_number.h" +#include "core/data_type/data_type_string.h" +#include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_timestamptz.h" +#include "core/data_type/data_type_variant_v2.h" +#include "core/value/timestamptz_value.h" +#include "core/value/variant/variant_batch_builder.h" +#include "core/value/variant/variant_parquet_encoding.h" +#include "exprs/function/function_variant_element_v2.h" +#include "format_v2/parquet/parquet_column_schema.h" + +namespace doris::format::parquet { +namespace { + +MutableColumnPtr nullable_strings(const std::vector& values, + const std::vector& nulls) { + auto data = ColumnString::create(); + auto null_map = ColumnUInt8::create(); + for (size_t row = 0; row < values.size(); ++row) { + data->insert_data(values[row].data, values[row].size); + null_map->get_data().push_back(nulls[row]); + } + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +ParquetColumnSchema unshredded_schema() { + ParquetColumnSchema schema; + schema.name = "payload"; + schema.kind = ParquetColumnSchemaKind::VARIANT; + schema.contains_variant = true; + schema.type = make_nullable(std::make_shared()); + const auto binary = make_nullable(std::make_shared()); + schema.variant_physical_type = make_nullable(std::make_shared( + DataTypes {binary, binary}, Strings {"metadata", "value"})); + + auto metadata = std::make_unique(); + metadata->name = "metadata"; + metadata->kind = ParquetColumnSchemaKind::PRIMITIVE; + metadata->type = binary; + auto value = std::make_unique(); + value->name = "value"; + value->kind = ParquetColumnSchemaKind::PRIMITIVE; + value->type = binary; + schema.children.push_back(std::move(metadata)); + schema.children.push_back(std::move(value)); + return schema; +} + +ParquetColumnSchema shredded_int64_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + typed->type = make_nullable(std::make_shared()); + typed->type_descriptor.integer_bit_width = 64; + schema.children.push_back(std::move(typed)); + const auto binary = make_nullable(std::make_shared()); + schema.variant_physical_type = make_nullable(std::make_shared( + DataTypes {binary, binary, make_nullable(std::make_shared())}, + Strings {"metadata", "value", "typed_value"})); + return schema; +} + +ParquetColumnSchema shredded_primitive_schema(DataTypePtr type) { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + typed->type = make_nullable(std::move(type)); + schema.children.push_back(std::move(typed)); + return schema; +} + +ParquetColumnSchema shredded_object_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + + auto field = std::make_unique(); + field->name = "a"; + field->kind = ParquetColumnSchemaKind::STRUCT; + auto field_typed = std::make_unique(); + field_typed->name = "typed_value"; + field_typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + field_typed->type = make_nullable(std::make_shared()); + field_typed->type_descriptor.integer_bit_width = 64; + field->children.push_back(std::move(field_typed)); + typed->children.push_back(std::move(field)); + schema.children.push_back(std::move(typed)); + return schema; +} + +ParquetColumnSchema shredded_named_object_schema(std::string field_name) { + auto schema = shredded_object_schema(); + schema.children.back()->children[0]->name = std::move(field_name); + return schema; +} + +ParquetColumnSchema shredded_binary_object_schema() { + auto schema = shredded_object_schema(); + auto* leaf = schema.children.back()->children[0]->children[0].get(); + leaf->type = make_nullable(std::make_shared()); + return schema; +} + +ParquetColumnSchema shredded_array_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::LIST; + auto element = std::make_unique(); + element->name = "element"; + element->kind = ParquetColumnSchemaKind::STRUCT; + auto element_typed = std::make_unique(); + element_typed->name = "typed_value"; + element_typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + element_typed->type = make_nullable(std::make_shared()); + element_typed->type_descriptor.integer_bit_width = 64; + element->children.push_back(std::move(element_typed)); + typed->children.push_back(std::move(element)); + schema.children.push_back(std::move(typed)); + return schema; +} + +ParquetColumnSchema shredded_mixed_array_schema() { + auto schema = shredded_array_schema(); + auto* element = schema.children.back()->children[0].get(); + auto value = std::make_unique(); + value->name = "value"; + value->kind = ParquetColumnSchemaKind::PRIMITIVE; + value->type = make_nullable(std::make_shared()); + element->children.insert(element->children.begin(), std::move(value)); + return schema; +} + +MutableColumnPtr shredded_int64_physical(const std::vector& values) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + std::vector metadata_rows(values.size(), metadata); + std::vector empty_values(values.size(), {ignored.data(), 0}); + std::vector present(values.size(), 0); + std::vector absent(values.size(), 1); + MutableColumns fields; + fields.push_back(nullable_strings(metadata_rows, present)); + fields.push_back(nullable_strings(empty_values, absent)); + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().resize_fill(values.size(), 0); + fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().resize_fill(values.size(), 0); + return ColumnNullable::create(std::move(structure), std::move(root_nulls)); +} + +MutableColumnPtr shredded_primitive_physical(MutableColumnPtr typed) { + const size_t rows = typed->size(); + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings(std::vector(rows, metadata), + std::vector(rows, 0))); + fields.push_back(nullable_strings(std::vector(rows, {ignored.data(), 0}), + std::vector(rows, 1))); + fields.push_back(std::move(typed)); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), + ColumnUInt8::create(rows, 0)); +} + +MutableColumnPtr projected_shredded_object_physical(const std::vector& values, + const IColumn** decoded_leaf = nullptr) { + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().resize_fill(values.size(), 0); + MutableColumnPtr leaf = ColumnNullable::create(std::move(integers), std::move(integer_nulls)); + if (decoded_leaf != nullptr) { + *decoded_leaf = leaf.get(); + } + + MutableColumns wrapper_fields; + wrapper_fields.push_back(std::move(leaf)); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back( + ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); +} + +MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls = {0}); +MutableColumnPtr nullable_int64(const std::vector& values, + const std::vector& nulls); + +MutableColumnPtr complete_shredded_object_physical(std::string_view residual_key, + int64_t residual_value, int64_t typed_value) { + VariantBatchBuilder builder; + auto row = builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef(residual_key.data(), residual_key.size())); + row.add_int(residual_value); + object.finish(); + row.finish(); + VariantBatchBuilder batch = builder.finish_batch(); + const VariantRef residual = batch.value_at(0); + + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({typed_value}, {0})); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(wrapper_fields)), + ColumnUInt8::create(1, 0))); + MutableColumns root_fields; + root_fields.push_back( + nullable_strings({StringRef(residual.metadata.data, residual.metadata.size)}, {0})); + root_fields.push_back( + nullable_strings({StringRef(residual.value.data, residual.value.size)}, {0})); + root_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + return root_wrapper(std::move(root_fields)); +} + +MutableColumnPtr projected_two_field_object_physical(const std::vector& first, + const std::vector& second) { + DORIS_CHECK(first.size() == second.size()); + auto wrapper = [](const std::vector& values) { + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto leaf = + ColumnNullable::create(std::move(integers), ColumnUInt8::create(values.size(), 0)); + MutableColumns fields; + fields.push_back(std::move(leaf)); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), + ColumnUInt8::create(values.size(), 0)); + }; + + MutableColumns object_fields; + object_fields.push_back(wrapper(first)); + object_fields.push_back(wrapper(second)); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(first.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(first.size(), 0)); +} + +MutableColumnPtr projected_wide_object_physical(size_t field_count, int64_t value) { + MutableColumns object_fields; + object_fields.reserve(field_count); + for (size_t field = 0; field < field_count; ++field) { + auto integers = ColumnInt64::create(); + integers->insert_value(value + field); + MutableColumns wrapper_fields; + wrapper_fields.push_back( + ColumnNullable::create(std::move(integers), ColumnUInt8::create(1, 0))); + object_fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(wrapper_fields)), ColumnUInt8::create(1, 0))); + } + MutableColumns root_fields; + root_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + return ColumnNullable::create(ColumnStruct::create(std::move(root_fields)), + ColumnUInt8::create(1, 0)); +} + +std::string materialization_error(const ParquetColumnSchema& schema, ColumnPtr physical) { + auto output = make_nullable(std::make_shared())->create_column(); + const Status status = materialize_variant_rows(schema, std::move(physical), output); + if (!status.ok()) { + return status.to_string(); + } + try { + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + (void)variants.get_value_ref(0); + } catch (const Exception& exception) { + return exception.what(); + } + return {}; +} + +MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls) { + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(root_nulls.begin(), root_nulls.end()); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), std::move(null_map)); +} + +MutableColumnPtr nullable_int64(const std::vector& values, + const std::vector& nulls) { + auto data = ColumnInt64::create(); + data->get_data().assign(values.begin(), values.end()); + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(nulls.begin(), nulls.end()); + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +template +MutableColumnPtr nullable_fixed(std::initializer_list values, + std::initializer_list nulls) { + auto data = ColumnType::create(); + for (const Value& value : values) { + data->insert_value(value); + } + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(nulls.begin(), nulls.end()); + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +template +MutableColumnPtr nullable_decimal(uint32_t scale, std::initializer_list values) { + auto data = ColumnType::create(0, scale); + for (const Value& value : values) { + data->insert_value(value); + } + return ColumnNullable::create(std::move(data), ColumnUInt8::create(values.size(), 0)); +} + +} // namespace + +TEST(VariantColumnReaderTest, UnshreddedRowsPreserveSqlNullAndVariantNull) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata, metadata, metadata}, {0, 0, 0})); + fields.push_back(nullable_strings( + {{int_seven.data(), int_seven.size()}, {ignored.data(), 0}, {ignored.data(), 0}}, + {0, 1, 1})); + auto physical_struct = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().assign({0, 1, 0}); + auto physical = ColumnNullable::create(std::move(physical_struct), std::move(root_nulls)); + + auto output_type = make_nullable(std::make_shared()); + auto output = output_type->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(output->size(), 3); + + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0, 1, 0})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_TRUE(variants.is_shredded()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); + EXPECT_TRUE(variants.get_value_ref(2).is_null()); +} + +TEST(VariantColumnReaderTest, RequiredPhysicalGroupAppendsToNullableExternalSlot) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical = ColumnStruct::create(std::move(fields)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, ShreddedIntegerKeepsDeclaredPhysicalWidth) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + auto integers = ColumnInt64::create(); + integers->get_data().push_back(42); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(structure), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_int64_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 42); + EXPECT_EQ(variants.get_value_ref(0).primitive_id(), VariantPrimitiveId::INT64); +} + +TEST(VariantColumnReaderTest, ReconstructsShreddedPrimitiveTypeMatrix) { + auto decode = [](ParquetColumnSchema schema, MutableColumnPtr typed, + const std::function& verify) { + auto output = make_nullable(std::make_shared())->create_column(); + const Status status = materialize_variant_rows( + schema, shredded_primitive_physical(std::move(typed)), output); + ASSERT_TRUE(status.ok()) << status; + verify(assert_cast( + assert_cast(*output).get_nested_column())); + }; + + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({0, 1}, {0, 0}), [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).primitive_id(), VariantPrimitiveId::FALSE_VALUE); + EXPECT_EQ(values.get_value_ref(1).primitive_id(), VariantPrimitiveId::TRUE_VALUE); + }); + + auto verify_integer = [&](DataTypePtr type, MutableColumnPtr typed, int width, int64_t first, + int64_t second) { + auto schema = shredded_primitive_schema(std::move(type)); + schema.children.back()->type_descriptor.integer_bit_width = width; + decode(std::move(schema), std::move(typed), [&](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_int(), first); + EXPECT_EQ(values.get_value_ref(1).get_int(), second); + }); + }; + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 8, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 16, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 32, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 64, std::numeric_limits::min(), std::numeric_limits::max()); + + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({std::numeric_limits::quiet_NaN(), + std::numeric_limits::infinity()}, + {0, 0}), + [](const auto& values) { + EXPECT_TRUE(std::isnan(values.get_value_ref(0).get_float())); + EXPECT_TRUE(std::isinf(values.get_value_ref(1).get_float())); + }); + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({-std::numeric_limits::infinity(), 1.25}, + {0, 0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_double(), + -std::numeric_limits::infinity()); + EXPECT_EQ(values.get_value_ref(1).get_double(), 1.25); + }); + + { + auto schema = shredded_primitive_schema(std::make_shared(9, 2)); + schema.children.back()->type_descriptor.decimal_precision = 9; + schema.children.back()->type_descriptor.decimal_scale = 2; + decode(std::move(schema), + nullable_decimal(2, {Decimal32 {12345}, Decimal32 {-1}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), (VariantDecimal {12345, 2, 4})); + EXPECT_EQ(values.get_value_ref(1).get_decimal(), (VariantDecimal {-1, 2, 4})); + }); + } + { + auto schema = shredded_primitive_schema(std::make_shared(18, 3)); + schema.children.back()->type_descriptor.decimal_precision = 18; + schema.children.back()->type_descriptor.decimal_scale = 3; + decode(std::move(schema), + nullable_decimal( + 3, {Decimal64 {123456789}, Decimal64 {-123456789}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), + (VariantDecimal {123456789, 3, 8})); + EXPECT_EQ(values.get_value_ref(1).get_decimal(), + (VariantDecimal {-123456789, 3, 8})); + }); + } + { + auto schema = shredded_primitive_schema(std::make_shared(38, 4)); + schema.children.back()->type_descriptor.decimal_precision = 38; + schema.children.back()->type_descriptor.decimal_scale = 4; + decode(std::move(schema), + nullable_decimal( + 4, {Decimal128V3 {static_cast(1234567890123456789LL)}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), + (VariantDecimal {1234567890123456789LL, 4, 16})); + }); + } + + const auto date = DateV2Value::create_from_olap_date( + (static_cast(1970) << 9) | (static_cast(1) << 5) | 2); + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed>({date}, {0}), + [](const auto& values) { EXPECT_EQ(values.get_value_ref(0).get_date(), 1); }); + + auto datetime = DateV2Value::create_from_olap_datetime(19700101000001ULL); + datetime.set_microsecond(234567); + { + auto schema = shredded_primitive_schema(std::make_shared(6)); + schema.children.back()->type_descriptor.time_unit = ParquetTimeUnit::MICROS; + schema.children.back()->type_descriptor.timestamp_is_adjusted_to_utc = false; + decode(std::move(schema), + nullable_fixed>({datetime}, {0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_timestamp_ntz_micros(), 1234567); + }); + } + TimestampTzValue timestamp; + timestamp.unchecked_set_time(1970, 1, 1, 0, 0, 2, 345678); + { + auto schema = shredded_primitive_schema(std::make_shared(6)); + schema.children.back()->type_descriptor.time_unit = ParquetTimeUnit::MICROS; + schema.children.back()->type_descriptor.timestamp_is_adjusted_to_utc = true; + decode(std::move(schema), + nullable_fixed({timestamp}, {0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_timestamp_micros(), 2345678); + }); + } + + auto verify_bytes = [&](bool string_annotation, bool uuid) { + const std::array bytes {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}; + auto schema = shredded_primitive_schema(std::make_shared()); + schema.children.back()->type_descriptor.is_string_annotation = string_annotation; + schema.children.back()->type_descriptor.is_uuid = uuid; + auto strings = ColumnString::create(); + if (uuid) { + strings->insert_data(reinterpret_cast(bytes.data()), bytes.size()); + } else { + strings->insert_data("bytes", 5); + } + auto typed = ColumnNullable::create(std::move(strings), ColumnUInt8::create(1, 0)); + decode(std::move(schema), std::move(typed), [&](const auto& values) { + if (uuid) { + EXPECT_EQ(values.get_value_ref(0).get_uuid(), bytes); + } else if (string_annotation) { + EXPECT_EQ(values.get_value_ref(0).get_string(), StringRef("bytes")); + } else { + EXPECT_EQ(values.get_value_ref(0).get_binary(), StringRef("bytes")); + } + }); + }; + verify_bytes(false, false); + verify_bytes(true, false); + verify_bytes(false, true); +} + +TEST(VariantColumnReaderTest, RejectsInvalidShreddedUuidWidth) { + auto schema = shredded_primitive_schema(std::make_shared()); + schema.children.back()->type_descriptor.is_uuid = true; + auto strings = ColumnString::create(); + strings->insert_data("short", 5); + auto typed = ColumnNullable::create(std::move(strings), ColumnUInt8::create(1, 0)); + const std::string error = + materialization_error(schema, shredded_primitive_physical(std::move(typed))); + EXPECT_NE(error.find("UUID has 5 bytes instead of 16"), std::string::npos) << error; +} + +TEST(VariantColumnReaderTest, DifferentMetadataDictionariesRemainIndependent) { + VariantBatchBuilder first_builder; + auto first_row = first_builder.begin_row(); + auto first_object = first_row.start_object(); + first_object.add_key(StringRef("alpha")); + first_row.add_int(1); + first_object.finish(); + first_row.finish(); + auto first = first_builder.finish_batch(); + + VariantBatchBuilder second_builder; + auto second_row = second_builder.begin_row(); + auto second_object = second_row.start_object(); + second_object.add_key(StringRef("beta")); + second_row.add_int(2); + second_object.finish(); + second_row.finish(); + auto second = second_builder.finish_batch(); + + const VariantRef first_value = first.value_at(0); + const VariantRef second_value = second.value_at(0); + MutableColumns fields; + fields.push_back(nullable_strings({{first_value.metadata.data, first_value.metadata.size}, + {second_value.metadata.data, second_value.metadata.size}}, + {0, 0})); + fields.push_back(nullable_strings({{first_value.value.data, first_value.value.size}, + {second_value.value.data, second_value.value.size}}, + {0, 0})); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().assign({0, 0}); + auto physical = ColumnNullable::create(std::move(structure), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("alpha"), &field)); + EXPECT_EQ(field.get_int(), 1); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("beta"), &field)); + EXPECT_EQ(field.get_int(), 2); +} + +TEST(VariantColumnReaderTest, AppendsCompleteShreddedStatesWithDifferentSchemasAndMetadata) { + auto output = make_nullable(std::make_shared())->create_column(); + auto first_schema = shredded_named_object_schema("a"); + ASSERT_TRUE(materialize_variant_rows(first_schema, + complete_shredded_object_physical("left", 1, 11), output) + .ok()); + auto second_schema = shredded_named_object_schema("b"); + ASSERT_TRUE(materialize_variant_rows(second_schema, + complete_shredded_object_physical("right", 2, 22), output) + .ok()); + + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + ASSERT_EQ(variants.size(), 2); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("left"), &field)); + EXPECT_EQ(field.get_int(), 1); + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_int(), 11); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("right"), &field)); + EXPECT_EQ(field.get_int(), 2); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("b"), &field)); + EXPECT_EQ(field.get_int(), 22); +} + +TEST(VariantColumnReaderTest, ShreddedObjectFieldMayOmitResidualValueColumn) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integer = ColumnInt64::create(); + integer->get_data().push_back(9); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integer), std::move(integer_nulls))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(0); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_int(), 9); + EXPECT_EQ(field.primitive_id(), VariantPrimitiveId::INT64); +} + +TEST(VariantColumnReaderTest, ShreddedTypedPathReusesDecodedLeafColumn) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integers = ColumnInt64::create(); + integers->get_data().push_back(9); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + MutableColumnPtr typed_leaf = + ColumnNullable::create(std::move(integers), std::move(integer_nulls)); + const IColumn* const decoded_typed_leaf = typed_leaf.get(); + + MutableColumns wrapper_fields; + wrapper_fields.push_back(std::move(typed_leaf)); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(0); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + ASSERT_TRUE(variants.is_shredded()); + + const std::array shredded_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(shredded_path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->column.get(), decoded_typed_leaf); + + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + ColumnPtr extracted; + ASSERT_TRUE( + extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) + .ok()); + + const auto& extracted_nullable = assert_cast(*extracted); + const auto& extracted_variant = + assert_cast(extracted_nullable.get_nested_column()); + ASSERT_TRUE(extracted_variant.is_typed()); + EXPECT_EQ(&extracted_variant.typed_column(), decoded_typed_leaf); + const auto& extracted_typed = + assert_cast(extracted_variant.typed_column()); + EXPECT_EQ(assert_cast(extracted_typed.get_nested_column()).get_data()[0], + 9); + EXPECT_TRUE(variants.is_shredded()); +} + +TEST(VariantColumnReaderTest, AppendsProjectedShreddedBatchesWithoutMaterializing) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back( + format::LocalColumnIndex::partial_local(schema.children[2]->local_id)); + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(schema.children[2]->children[0]->local_id)); + projection.children.back().children.back().children.push_back(format::LocalColumnIndex::local( + schema.children[2]->children[0]->children[0]->local_id)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + ASSERT_EQ(plan.variant_state_schema.use_count(), 1); + + auto output = make_nullable(std::make_shared())->create_column(); + const IColumn* first_decoded_leaf = nullptr; + ASSERT_TRUE( + materialize_variant_columns( + plan, projected_shredded_object_physical({10, 20}, &first_decoded_leaf), output) + .ok()); + EXPECT_EQ(plan.variant_state_schema.use_count(), 2); + const auto append_status = + materialize_variant_columns(plan, projected_shredded_object_physical({30}), output); + ASSERT_TRUE(append_status.ok()) << append_status; + + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + ASSERT_TRUE(variants.is_shredded()); + ASSERT_EQ(variants.size(), 3); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->column.get(), first_decoded_leaf); + const auto& values = assert_cast( + assert_cast(*match->column).get_nested_column()); + EXPECT_EQ(values.get_data(), ColumnInt64::Container({10, 20, 30})); + + IColumn::Filter filter {1, 0, 1}; + ColumnPtr filtered = output->filter(filter, 2); + EXPECT_EQ(filtered->size(), 2); + EXPECT_EQ(plan.variant_state_schema.use_count(), 3); +} + +TEST(VariantColumnReaderTest, WideProjectionSharesSchemaAcrossBatchesAndSelections) { + constexpr size_t width = 64; + constexpr size_t batch_count = 16; + auto schema = unshredded_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + typed->local_id = 2; + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(typed->local_id)); + for (size_t field = 0; field < width; ++field) { + auto wrapper = std::make_unique(); + wrapper->name = "field_" + std::to_string(field); + wrapper->kind = ParquetColumnSchemaKind::STRUCT; + wrapper->local_id = static_cast(field); + auto leaf = std::make_unique(); + leaf->name = "typed_value"; + leaf->kind = ParquetColumnSchemaKind::PRIMITIVE; + leaf->local_id = 0; + leaf->type = make_nullable(std::make_shared()); + leaf->type_descriptor.integer_bit_width = 64; + wrapper->children.push_back(std::move(leaf)); + typed->children.push_back(std::move(wrapper)); + + auto wrapper_projection = format::LocalColumnIndex::partial_local(static_cast(field)); + wrapper_projection.children.push_back(format::LocalColumnIndex::local(0)); + projection.children.back().children.push_back(std::move(wrapper_projection)); + } + schema.children.push_back(std::move(typed)); + + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto output = make_nullable(std::make_shared())->create_column(); + for (size_t batch = 0; batch < batch_count; ++batch) { + ASSERT_TRUE(materialize_variant_columns( + plan, projected_wide_object_physical(width, batch * width), output) + .ok()); + } + ASSERT_EQ(output->size(), batch_count); + ASSERT_EQ(plan.variant_state_schema.use_count(), 2); + + // Holding derived slices makes schema ownership observable: every state must retain the same + // reader-scoped schema instead of allocating a width-sized clone for each row selection. + std::vector slices; + slices.reserve(batch_count); + for (size_t row = 0; row < batch_count; ++row) { + slices.push_back(output->cut(row, 1)); + } + EXPECT_EQ(plan.variant_state_schema.use_count(), 2 + batch_count); +} + +TEST(VariantColumnReaderTest, RetainedSchemaFollowsDecodedProjectionOrder) { + auto schema = unshredded_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + typed->local_id = 2; + auto make_wrapper = [](std::string name, int local_id) { + auto wrapper = std::make_unique(); + wrapper->name = std::move(name); + wrapper->kind = ParquetColumnSchemaKind::STRUCT; + wrapper->local_id = local_id; + auto leaf = std::make_unique(); + leaf->name = "typed_value"; + leaf->kind = ParquetColumnSchemaKind::PRIMITIVE; + leaf->local_id = 0; + leaf->type = make_nullable(std::make_shared()); + leaf->type_descriptor.integer_bit_width = 64; + wrapper->children.push_back(std::move(leaf)); + return wrapper; + }; + typed->children.push_back(make_wrapper("z", 0)); + typed->children.push_back(make_wrapper("a", 1)); + schema.children.push_back(std::move(typed)); + + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + for (int local_id : {1, 0}) { + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(local_id)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + } + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, projected_two_field_object_physical({11}, {22}), + output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const std::array a_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const std::array z_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("z")}}; + const auto a = variants.find_shredded_typed_value(a_path); + const auto z = variants.find_shredded_typed_value(z_path); + ASSERT_TRUE(a.has_value()); + ASSERT_TRUE(z.has_value()); + EXPECT_EQ(assert_cast( + assert_cast(*a->column).get_nested_column()) + .get_data()[0], + 11); + EXPECT_EQ(assert_cast( + assert_cast(*z->column).get_nested_column()) + .get_data()[0], + 22); +} + +TEST(VariantColumnReaderTest, AmbiguousTypedIdentityRequiresCanonicalMaterialization) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_strings({StringRef("abc")}, {0})); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(1, 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), ColumnUInt8::create(1, 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto physical = ColumnNullable::create(std::move(root), ColumnUInt8::create(1, 0)); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_binary_object_schema(), *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + EXPECT_FALSE(variants.find_shredded_typed_value(path).has_value()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_binary(), StringRef("abc")); +} + +TEST(VariantColumnReaderTest, MaterializedCacheParticipatesInMemoryAccounting) { + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({42, 43}), + output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const size_t physical_bytes = variants.byte_size(); + const size_t physical_allocated = variants.allocated_bytes(); + + EXPECT_EQ(variants.get_value_ref(0).get_int(), 42); + EXPECT_GT(variants.byte_size(), physical_bytes); + EXPECT_GT(variants.allocated_bytes(), physical_allocated); +} + +TEST(VariantColumnReaderTest, MaterializedShreddedCopiesDetachBeforeMutation) { + auto first_output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({10, 20}), + first_output) + .ok()); + const auto& first = assert_cast( + assert_cast(*first_output).get_nested_column()); + EXPECT_EQ(first.get_value_ref(0).get_int(), 10); + + auto cloned = first.clone_resized(first.size()); + EXPECT_NO_THROW(cloned->pop_back(1)); + ASSERT_EQ(cloned->size(), 1); + EXPECT_EQ(assert_cast(*cloned).get_value_ref(0).get_int(), 10); + + auto second_output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({30}), + second_output) + .ok()); + const auto& second = assert_cast( + assert_cast(*second_output).get_nested_column()); + auto appended = ColumnVariantV2::create(); + appended->insert_range_from(first, 0, first.size()); + EXPECT_NO_THROW(appended->insert_range_from(second, 0, second.size())); + ASSERT_EQ(appended->size(), 3); + EXPECT_EQ(appended->get_value_ref(2).get_int(), 30); +} + +TEST(VariantColumnReaderTest, MissingShreddedObjectWrapperMeansAbsentField) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integer = ColumnInt64::create(); + integer->get_data().push_back(0); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(1); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integer), std::move(integer_nulls))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(1); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).num_elements(), 0); +} + +TEST(VariantColumnReaderTest, MaterializesShreddedArrayElements) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integers = ColumnInt64::create(); + integers->get_data().assign({3, 4}); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().assign({0, 0}); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto wrappers = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().assign({0, 0}); + auto elements = ColumnNullable::create(std::move(wrappers), std::move(wrapper_nulls)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto array = ColumnArray::create(std::move(elements), std::move(offsets)); + auto array_nulls = ColumnUInt8::create(); + array_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(array), std::move(array_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_array_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + const VariantRef value = variants.get_value_ref(0); + ASSERT_EQ(value.num_elements(), 2); + EXPECT_EQ(value.array_at(0).get_int(), 3); + EXPECT_EQ(value.array_at(1).get_int(), 4); +} + +TEST(VariantColumnReaderTest, RejectsCorruptShreddedWrappersWithoutCrashing) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const std::array invalid_value {static_cast(0xff)}; + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + const StringRef residual_int(int_seven.data(), int_seven.size()); + auto expect_error = [](const std::string& error, std::string_view expected) { + EXPECT_NE(error.find(expected), std::string::npos) << error; + }; + std::string_view current_case; + + try { + { + current_case = "null metadata"; + SCOPED_TRACE("null metadata"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {1})); + fields.push_back(nullable_strings({residual_int}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "null metadata"); + } + { + current_case = "wrapper without carriers"; + SCOPED_TRACE("wrapper without carriers"); + auto schema = unshredded_schema(); + schema.children.pop_back(); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + expect_error(materialization_error(schema, root_wrapper(std::move(fields))), + "neither value nor typed_value"); + } + { + current_case = "scalar with residual"; + SCOPED_TRACE("scalar with residual"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(nullable_int64({8}, {0})); + expect_error( + materialization_error(shredded_int64_schema(), root_wrapper(std::move(fields))), + "scalar typed_value cannot have residual"); + } + { + current_case = "object with scalar residual"; + SCOPED_TRACE("object with scalar residual"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({9}, {0})); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(wrapper_fields)), ColumnUInt8::create(1, 0))); + fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_object_schema(), + root_wrapper(std::move(fields))), + "non-object residual"); + } + { + current_case = "object field count mismatch"; + SCOPED_TRACE("object field count mismatch"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + MutableColumns unexpected_object_fields; + unexpected_object_fields.push_back(nullable_int64({1}, {0})); + unexpected_object_fields.push_back(nullable_int64({2}, {0})); + fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(unexpected_object_fields)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_object_schema(), + root_wrapper(std::move(fields))), + "physical field count mismatch"); + } + { + current_case = "array with residual"; + SCOPED_TRACE("array with residual"); + MutableColumns empty_wrapper_fields; + empty_wrapper_fields.push_back(nullable_int64({}, {})); + auto empty_elements = ColumnNullable::create( + ColumnStruct::create(std::move(empty_wrapper_fields)), ColumnUInt8::create()); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(0); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(empty_elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error( + materialization_error(shredded_array_schema(), root_wrapper(std::move(fields))), + "array typed_value cannot have residual"); + } + { + current_case = "null array element wrapper"; + SCOPED_TRACE("null array element wrapper"); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({0}, {1})); + auto wrappers = ColumnStruct::create(std::move(wrapper_fields)); + auto elements = ColumnNullable::create(std::move(wrappers), ColumnUInt8::create(1, 1)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error( + materialization_error(shredded_array_schema(), root_wrapper(std::move(fields))), + "array element wrapper is null"); + } + { + current_case = "missing array element"; + SCOPED_TRACE("missing array element"); + MutableColumns element_fields; + element_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + element_fields.push_back(nullable_int64({0}, {1})); + auto elements = ColumnNullable::create(ColumnStruct::create(std::move(element_fields)), + ColumnUInt8::create(1, 0)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_mixed_array_schema(), + root_wrapper(std::move(fields))), + "array element is missing"); + } + { + current_case = "root field count mismatch"; + SCOPED_TRACE("root field count mismatch"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(nullable_int64({8}, {0})); + fields.push_back(nullable_int64({9}, {0})); + expect_error( + materialization_error(shredded_int64_schema(), root_wrapper(std::move(fields))), + "physical field count mismatch"); + } + { + current_case = "invalid metadata"; + SCOPED_TRACE("invalid metadata"); + MutableColumns fields; + fields.push_back(nullable_strings({StringRef("bad")}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "metadata"); + } + { + current_case = "invalid residual value"; + SCOPED_TRACE("invalid residual value"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{invalid_value.data(), invalid_value.size()}}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "Variant"); + } + } catch (const std::exception& error) { + FAIL() << "Unexpected exception in " << current_case << ": " << error.what(); + } +} + +TEST(VariantColumnReaderTest, ImmediateCorruptionLeavesDestinationUnchanged) { + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE( + materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({7}), output) + .ok()); + MutableColumns invalid_fields; + invalid_fields.push_back(nullable_strings( + {{VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()}}, {0})); + const Status status = materialize_variant_rows(shredded_int64_schema(), + root_wrapper(std::move(invalid_fields)), output); + EXPECT_FALSE(status.ok()); + ASSERT_EQ(output->size(), 1); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, LazyNestedCorruptionLeavesDestinationUnchanged) { + const std::array invalid_value {static_cast(0xff)}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + auto corrupt_variant = [&]() { + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{invalid_value.data(), invalid_value.size()}}, {0})); + return root_wrapper(std::move(fields)); + }; + auto label_schema = []() { + auto schema = std::make_unique(); + schema->name = "label"; + schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + schema->type = make_nullable(std::make_shared()); + return schema; + }; + auto make_plan = [](const ParquetColumnSchema& root) { + auto build = [&](auto&& self, const ParquetColumnSchema* schema) + -> std::unique_ptr { + auto node = std::make_unique(); + node->schema = schema; + node->contains_variant = schema->kind == ParquetColumnSchemaKind::VARIANT; + for (const auto& child_schema : schema->children) { + auto child = self(self, child_schema.get()); + node->contains_variant = node->contains_variant || child->contains_variant; + node->children.push_back(std::move(child)); + } + return node; + }; + return build(build, &root); + }; + auto make_struct_schema = [&](ParquetColumnSchema variant_schema) { + ParquetColumnSchema root; + root.name = "row"; + root.kind = ParquetColumnSchemaKind::STRUCT; + root.children.push_back(label_schema()); + root.children.push_back(std::make_unique(std::move(variant_schema))); + return root; + }; + auto make_struct_physical = [&](std::string_view label, MutableColumnPtr variant) { + MutableColumns fields; + fields.push_back(nullable_strings({StringRef(label.data(), label.size())}, {0})); + fields.push_back(std::move(variant)); + return ColumnStruct::create(std::move(fields)); + }; + const auto element_type = std::make_shared( + DataTypes {make_nullable(std::make_shared()), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}); + + { + auto output = element_type->create_column(); + auto valid_schema = make_struct_schema(shredded_int64_schema()); + auto valid_plan = make_plan(valid_schema); + ASSERT_TRUE(materialize_variant_columns( + *valid_plan, + *make_struct_physical("before", shredded_int64_physical({7})), output) + .ok()); + + auto corrupt_schema = make_struct_schema(unshredded_schema()); + auto corrupt_plan = make_plan(corrupt_schema); + const Status status = materialize_variant_columns( + *corrupt_plan, *make_struct_physical("after", corrupt_variant()), output); + EXPECT_FALSE(status.ok()); + + const auto& structure = assert_cast(*output); + const auto& label = assert_cast(structure.get_column(0)); + EXPECT_EQ(label.size(), 1); + EXPECT_EQ(label.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(label.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& payload = assert_cast(structure.get_column(1)); + EXPECT_EQ(payload.size(), 1); + EXPECT_EQ(payload.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(payload.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } + + { + auto output = std::make_shared(element_type)->create_column(); + auto valid_element_schema = make_struct_schema(shredded_int64_schema()); + ParquetColumnSchema valid_schema; + valid_schema.name = "rows"; + valid_schema.kind = ParquetColumnSchemaKind::LIST; + valid_schema.children.push_back( + std::make_unique(std::move(valid_element_schema))); + auto valid_plan = make_plan(valid_schema); + auto valid_offsets = ColumnArray::ColumnOffsets::create(); + valid_offsets->insert_value(1); + auto valid_physical = + ColumnArray::create(make_struct_physical("before", shredded_int64_physical({7})), + std::move(valid_offsets)); + ASSERT_TRUE(materialize_variant_columns(*valid_plan, *valid_physical, output).ok()); + + auto corrupt_element_schema = make_struct_schema(unshredded_schema()); + ParquetColumnSchema corrupt_schema; + corrupt_schema.name = "rows"; + corrupt_schema.kind = ParquetColumnSchemaKind::LIST; + corrupt_schema.children.push_back( + std::make_unique(std::move(corrupt_element_schema))); + auto corrupt_plan = make_plan(corrupt_schema); + auto corrupt_offsets = ColumnArray::ColumnOffsets::create(); + corrupt_offsets->insert_value(1); + auto corrupt_physical = ColumnArray::create( + make_struct_physical("after", corrupt_variant()), std::move(corrupt_offsets)); + const Status status = materialize_variant_columns(*corrupt_plan, *corrupt_physical, output); + EXPECT_FALSE(status.ok()); + + const auto& array = assert_cast(*output); + EXPECT_EQ(array.get_offsets(), (ColumnArray::Offsets64 {1})); + const auto& element = assert_cast(array.get_data()); + EXPECT_EQ(element.get_null_map_data(), (NullMap {0})); + const auto& structure = assert_cast(element.get_nested_column()); + const auto& label = assert_cast(structure.get_column(0)); + EXPECT_EQ(label.size(), 1); + EXPECT_EQ(label.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(label.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& payload = assert_cast(structure.get_column(1)); + EXPECT_EQ(payload.size(), 1); + EXPECT_EQ(payload.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(payload.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } + + { + auto output = + std::make_shared(make_nullable(std::make_shared()), + make_nullable(std::make_shared())) + ->create_column(); + auto make_map_schema = [&](ParquetColumnSchema variant_schema) { + ParquetColumnSchema root; + root.name = "entries"; + root.kind = ParquetColumnSchemaKind::MAP; + root.children.push_back(label_schema()); + root.children.push_back( + std::make_unique(std::move(variant_schema))); + return root; + }; + auto make_map_physical = [&](std::string_view key, MutableColumnPtr variant) { + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + return ColumnMap::create(nullable_strings({StringRef(key.data(), key.size())}, {0}), + std::move(variant), std::move(offsets)); + }; + + auto valid_schema = make_map_schema(shredded_int64_schema()); + auto valid_plan = make_plan(valid_schema); + ASSERT_TRUE(materialize_variant_columns( + *valid_plan, *make_map_physical("before", shredded_int64_physical({7})), + output) + .ok()); + auto corrupt_schema = make_map_schema(unshredded_schema()); + auto corrupt_plan = make_plan(corrupt_schema); + const Status status = materialize_variant_columns( + *corrupt_plan, *make_map_physical("after", corrupt_variant()), output); + EXPECT_FALSE(status.ok()); + + const auto& map = assert_cast(*output); + EXPECT_EQ(map.get_offsets(), (ColumnArray::Offsets64 {1})); + const auto& keys = assert_cast(map.get_keys()); + EXPECT_EQ(keys.size(), 1); + EXPECT_EQ(keys.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(keys.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& values = assert_cast(map.get_values()); + EXPECT_EQ(values.size(), 1); + EXPECT_EQ(values.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(values.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } +} + +TEST(VariantColumnReaderTest, MaterializesMixedRootArraysAndNullKinds) { + VariantBatchBuilder residual_builder; + { + auto row = residual_builder.begin_row(); + row.add_null(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef("x")); + row.add_int(2); + object.finish(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + auto array = row.start_array(); + row.add_int(3); + row.add_int(4); + array.finish(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + row.add_string(StringRef("tail")); + row.finish(); + } + VariantBatchBuilder residuals = residual_builder.finish_batch(); + const VariantRef first = residuals.value_at(0); + std::vector residual_values; + for (size_t row = 0; row < residuals.num_rows(); ++row) { + residual_values.push_back(residuals.value_at(row).value); + } + residual_values.insert(residual_values.begin() + 1, StringRef {}); + + MutableColumns element_fields; + element_fields.push_back(nullable_strings(residual_values, {0, 1, 0, 0, 0})); + element_fields.push_back(nullable_int64({0, 1, 0, 0, 0}, {1, 0, 1, 1, 1})); + auto elements = ColumnNullable::create(ColumnStruct::create(std::move(element_fields)), + ColumnUInt8::create(5, 0)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->get_data().assign({0, 5, 5, 5}); + auto arrays = ColumnArray::create(std::move(elements), std::move(offsets)); + + const StringRef metadata(first.metadata.data, first.metadata.size); + const std::array ignored {0}; + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata, metadata, metadata, metadata}, {0, 0, 0, 0})); + root_fields.push_back(nullable_strings( + {{ignored.data(), 0}, {ignored.data(), 0}, {ignored.data(), 0}, {ignored.data(), 0}}, + {1, 1, 1, 1})); + auto typed_nulls = ColumnUInt8::create(4, 0); + typed_nulls->get_data()[2] = 1; + typed_nulls->get_data()[3] = 1; + root_fields.push_back(ColumnNullable::create(std::move(arrays), std::move(typed_nulls))); + auto physical = root_wrapper(std::move(root_fields), {0, 0, 0, 1}); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_mixed_array_schema(), *physical, output).ok()); + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0, 0, 0, 1})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).num_elements(), 0); + const VariantRef mixed = variants.get_value_ref(1); + ASSERT_EQ(mixed.num_elements(), 5); + EXPECT_TRUE(mixed.array_at(0).is_null()); + EXPECT_EQ(mixed.array_at(1).get_int(), 1); + VariantRef object_field; + ASSERT_TRUE(mixed.array_at(2).object_find(StringRef("x"), &object_field)); + EXPECT_EQ(object_field.get_int(), 2); + EXPECT_EQ(mixed.array_at(3).array_at(1).get_int(), 4); + EXPECT_EQ(mixed.array_at(4).get_string(), StringRef("tail")); + EXPECT_TRUE(variants.get_value_ref(2).is_null()); +} + +TEST(VariantColumnReaderTest, MaterializesVariantNestedInStruct) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + auto physical = ColumnStruct::create(std::move(root_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + root_schema.children.push_back(std::make_unique(unshredded_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + auto child_plan = std::make_unique(); + child_plan->schema = root_schema.children[0].get(); + child_plan->contains_variant = true; + plan.children.push_back(std::move(child_plan)); + + auto output = std::make_shared( + DataTypes {make_nullable(std::make_shared())}, + Strings {"payload"}) + ->create_column(); + const auto status = materialize_variant_columns(plan, *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& output_struct = assert_cast(*output); + const auto& nullable = assert_cast(output_struct.get_column(0)); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, MaterializesPhysicallyShreddedVariantInStructArrayAndMap) { + auto make_plan_child = [](const ParquetColumnSchema* schema) { + auto child = std::make_unique(); + child->schema = schema; + child->contains_variant = schema->kind == ParquetColumnSchemaKind::VARIANT; + return child; + }; + + { + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + MutableColumns physical_fields; + physical_fields.push_back(shredded_int64_physical({11})); + auto physical = ColumnStruct::create(std::move(physical_fields)); + auto output = std::make_shared( + DataTypes {make_nullable(std::make_shared())}, + Strings {"v"}) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_column(0)) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 11); + } + + { + ParquetColumnSchema root_schema; + root_schema.name = "items"; + root_schema.kind = ParquetColumnSchemaKind::LIST; + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto physical = ColumnArray::create(shredded_int64_physical({12, 13}), std::move(offsets)); + auto output = std::make_shared( + make_nullable(std::make_shared())) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_data()) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 12); + EXPECT_EQ(variants.get_value_ref(1).get_int(), 13); + } + + { + ParquetColumnSchema root_schema; + root_schema.name = "entries"; + root_schema.kind = ParquetColumnSchemaKind::MAP; + auto key_schema = std::make_unique(); + key_schema->name = "key"; + key_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + key_schema->type = std::make_shared(); + root_schema.children.push_back(std::move(key_schema)); + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + plan.children.push_back(make_plan_child(root_schema.children[1].get())); + auto keys = ColumnString::create(); + keys->insert_data("a", 1); + keys->insert_data("b", 1); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto physical = ColumnMap::create(std::move(keys), shredded_int64_physical({14, 15}), + std::move(offsets)); + auto output = + std::make_shared(std::make_shared(), + make_nullable(std::make_shared())) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_values()) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 14); + EXPECT_EQ(variants.get_value_ref(1).get_int(), 15); + } +} + +TEST(VariantColumnReaderTest, ProjectedShreddedStateRejectsRootMaterialization) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({17}), output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + EXPECT_THROW((void)variants.get_value_ref(0), Exception); +} + +TEST(VariantColumnReaderTest, AlignsNestedPrimitiveNullabilityAroundVariant) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + MutableColumns physical_fields; + physical_fields.push_back(nullable_strings({StringRef("required")}, {0})); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + physical_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + auto physical = ColumnStruct::create(std::move(physical_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + auto label_schema = std::make_unique(); + label_schema->name = "label"; + label_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + label_schema->type = make_nullable(std::make_shared()); + root_schema.children.push_back(std::move(label_schema)); + root_schema.children.push_back(std::make_unique(unshredded_schema())); + + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + for (const auto& child_schema : root_schema.children) { + auto child_plan = std::make_unique(); + child_plan->schema = child_schema.get(); + child_plan->contains_variant = child_schema->kind == ParquetColumnSchemaKind::VARIANT; + plan.children.push_back(std::move(child_plan)); + } + + auto output = std::make_shared( + DataTypes {std::make_shared(), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}) + ->create_column(); + const auto status = materialize_variant_columns(plan, *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& output_struct = assert_cast(*output); + EXPECT_EQ(output_struct.get_column(0).get_data_at(0).to_string(), "required"); + const auto& nullable = assert_cast(output_struct.get_column(1)); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, NestedMaterializationMovesUnaffectedSiblingBuffers) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto label = ColumnString::create(); + label->insert_data("large-sibling", 13); + const IColumn* decoded_label = label.get(); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + MutableColumns root_fields; + root_fields.push_back(std::move(label)); + root_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + ColumnPtr physical = ColumnStruct::create(std::move(root_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + auto label_schema = std::make_unique(); + label_schema->name = "label"; + label_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + label_schema->type = std::make_shared(); + root_schema.children.push_back(std::move(label_schema)); + root_schema.children.push_back(std::make_unique(unshredded_schema())); + + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + for (const auto& child_schema : root_schema.children) { + auto child_plan = std::make_unique(); + child_plan->schema = child_schema.get(); + child_plan->contains_variant = child_schema->kind == ParquetColumnSchemaKind::VARIANT; + plan.children.push_back(std::move(child_plan)); + } + + auto output = std::make_shared( + DataTypes {std::make_shared(), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}) + ->create_column(); + const IColumn* empty_output = output.get(); + ASSERT_TRUE(materialize_variant_columns(plan, std::move(physical), output).ok()); + EXPECT_NE(output.get(), empty_output); + const auto& output_struct = assert_cast(*output); + EXPECT_EQ(&output_struct.get_column(0), decoded_label); +} + +} // namespace doris::format::parquet diff --git a/be/test/format_v2/table/iceberg_reader_test.cpp b/be/test/format_v2/table/iceberg_reader_test.cpp index 806cc7e729a467..b9fe813c9c68f0 100644 --- a/be/test/format_v2/table/iceberg_reader_test.cpp +++ b/be/test/format_v2/table/iceberg_reader_test.cpp @@ -59,6 +59,7 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/data_type_timestamptz.h" #include "core/data_type/data_type_varbinary.h" +#include "core/data_type/data_type_variant_v2.h" #include "exec/common/endian.h" #include "exec/scan/access_path_parser.h" #include "exprs/runtime_filter_expr.h" @@ -2627,6 +2628,44 @@ TEST(IcebergV2ReaderTest, IcebergLegacyPlanKeepsAllFieldIdsMappingRule) { TableColumnMappingMode::BY_NAME); } +TEST(IcebergV2ReaderTest, VariantFormatGateUsesPhysicalFileMappings) { + ColumnMapping missing_variant; + missing_variant.table_type = make_nullable(std::make_shared()); + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {missing_variant}) + .ok()); + + ColumnMapping physical_variant = missing_variant; + physical_variant.file_local_id = 0; + const auto orc_status = + doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {physical_variant}); + EXPECT_TRUE(orc_status.is()) << orc_status; + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::PARQUET, {physical_variant}) + .ok()); + + ColumnMapping projected_struct; + projected_struct.table_type = make_nullable(std::make_shared( + DataTypes {make_nullable(std::make_shared()), + make_nullable(std::make_shared())}, + Strings {"label", "payload"})); + projected_struct.file_local_id = 0; + ColumnMapping label; + label.table_type = make_nullable(std::make_shared()); + label.file_local_id = 1; + projected_struct.child_mappings = {label, missing_variant}; + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {projected_struct}) + .ok()); + + projected_struct.child_mappings[1] = physical_variant; + const auto nested_orc_status = + doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {projected_struct}); + EXPECT_TRUE(nested_orc_status.is()) << nested_orc_status; +} + TEST(IcebergV2ReaderTest, IcebergTableReaderDoesNotPushDownAggregateWithPositionDelete) { const auto test_dir = std::filesystem::temp_directory_path() / "doris_iceberg_aggregate_position_delete_test"; diff --git a/be/test/format_v2/table_reader_request_test.cpp b/be/test/format_v2/table_reader_request_test.cpp index 3845e086cea1b1..a58cc82f45d6cb 100644 --- a/be/test/format_v2/table_reader_request_test.cpp +++ b/be/test/format_v2/table_reader_request_test.cpp @@ -71,6 +71,32 @@ TEST(FileScanRequestBuilderTest, PredicateColumnRemovesDuplicateNonPredicateColu EXPECT_EQ(request.non_predicate_columns[0].column_id(), LocalColumnId(2)); } +TEST(FileScanRequestBuilderTest, DeferredComplexOutputSurvivesLaterPredicateMerge) { + FileScanRequest request; + FileScanRequestBuilder builder(&request); + + auto predicate = LocalColumnIndex::partial_local(5); + predicate.children.push_back(LocalColumnIndex::local(0)); + ASSERT_TRUE(builder.add_predicate_column(std::move(predicate)).ok()); + + auto output = LocalColumnIndex::partial_local(5); + output.children.push_back(LocalColumnIndex::local(0)); + output.children.push_back(LocalColumnIndex::local(1)); + ASSERT_TRUE(builder.add_deferred_non_predicate_column(std::move(output)).ok()); + + auto delete_predicate = LocalColumnIndex::partial_local(5); + delete_predicate.children.push_back(LocalColumnIndex::local(2)); + ASSERT_TRUE(builder.add_predicate_column(std::move(delete_predicate)).ok()); + + ASSERT_EQ(request.predicate_columns.size(), 1); + EXPECT_EQ(request.predicate_columns[0].children.size(), 2); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_EQ(request.non_predicate_columns[0].children.size(), 2); + EXPECT_EQ(request.local_positions.at(LocalColumnId(5)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(5)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(5))); +} + // Scenario: TableReader's format-specific customization path delegates to FileScanRequestBuilder // and preserves the same predicate/non-predicate de-duplication rule. TEST(TableReaderRequestTest, AppendPredicateColumnKeepsOtherNonPredicateColumns) { diff --git a/be/test/format_v2/table_reader_test.cpp b/be/test/format_v2/table_reader_test.cpp index 219f2bff269e38..19d2f977cb2255 100644 --- a/be/test/format_v2/table_reader_test.cpp +++ b/be/test/format_v2/table_reader_test.cpp @@ -1569,6 +1569,48 @@ TEST(TableReaderTest, ConstantPruningStopsAtUnsafeSlotlessPredicate) { ASSERT_TRUE(reader.close().ok()); } +TEST(TableReaderTest, MetadataPruningBoundaryKeepsUnsafeSlotlessBarrier) { + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + bool predicate_executed = false; + auto fake_state = std::make_shared(); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE( + reader + .init({ + .projected_columns = projected_columns, + .conjuncts = + {prepared_conjunct( + &state, + std::make_shared( + &predicate_executed)), + prepared_conjunct(&state, + table_int32_greater_than_expr(0, 0, 10))}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }) + .ok()); + + SplitReadOptions split; + split.current_range.__set_path("fake-table-reader-input"); + ASSERT_TRUE(reader.prepare_split(split).ok()); + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + ASSERT_NE(fake_state->last_request, nullptr); + ASSERT_EQ(fake_state->last_request->conjuncts.size(), 1); + EXPECT_EQ(fake_state->last_request->metadata_pruning_safe_conjunct_count, 0); + ASSERT_TRUE(reader.close().ok()); +} + TEST(TableReaderTest, CanUseInjectedFileReaderForStandaloneUnitTest) { std::vector file_schema; file_schema.push_back(make_file_column(0, "id", std::make_shared())); diff --git a/docs/file-scanner-v2-parquet-scan-design.md b/docs/file-scanner-v2-parquet-scan-design.md index b81a779ac5edcc..29dfe66c6fac63 100644 --- a/docs/file-scanner-v2-parquet-scan-design.md +++ b/docs/file-scanner-v2-parquet-scan-design.md @@ -197,9 +197,9 @@ flowchart LR does not repeatedly interpret table-schema evolution. 3. **Capability checks:** ZoneMap, Dictionary, and Bloom use only expressions they can interpret safely. All others remain row-level residual predicates. -4. **Prefer safe single-column predicates:** Single-column predicates can drive indexes and staged - filtering. Multi-column, stateful, or error-sensitive expressions retain whole-expression - evaluation. +4. **Prefer safe single-column row filters:** Single-column predicates can drive staged dictionary + or raw filtering. Multi-column AND/OR trees may still combine conservative Row Group and Page + Index candidate ranges, but the complete expression remains in whole-expression row evaluation. 5. **Runtime Filters can refresh:** ScannerScheduler refreshes late Runtime Filters before reading. TableReader handles partition-range pruning during Split preparation, and passes file-pushable parts as localized conjuncts. @@ -285,9 +285,11 @@ sequenceDiagram ### How the plan drives physical skips ColumnIndex provides min/max/null semantics for each page. OffsetIndex maps pages to Row Group row -numbers and file offsets. Candidate ranges from multiple predicate columns are intersected into -`selected_ranges`; a `page_skip_plan` is then built for each leaf so its column reader can skip pages -that do not overlap surviving rows. +numbers and file offsets. Candidate ranges follow the predicate tree: AND nodes intersect child +ranges and OR nodes union them into `selected_ranges`. A missing or unusable AND child contributes +no pruning, while a missing or unusable OR branch retains the complete Row Group range. A +`page_skip_plan` is then built for each leaf so its column reader can skip pages that do not overlap +surviving rows. > `selected_ranges` represents logical row ranges, while `page_skip_plan` represents physical page > reads. Keeping them separate allows the scheduler to advance by row batch while each column skips @@ -826,7 +828,8 @@ split safely, or read anomalies must never change query semantics. | Bloom missing, disabled, or unreadable | Skip Bloom pruning and continue with later scan stages | | Incomplete dictionary page, mixed non-dictionary encoding, complex/repeated column | Disable dictionary pruning and Dictionary-ID Filter; use actual values | | Missing or inconsistent ColumnIndex/OffsetIndex | Disable fine-grained page pruning and read the full candidate range | -| Multi-column, OR, stateful, or error-order-sensitive expression | Preserve whole-expression evaluation to avoid changing SQL short-circuit or error semantics | +| Multi-column AND/OR expression | Combine only conservative metadata candidate ranges; preserve whole-expression row evaluation | +| Stateful or error-order-sensitive expression | Preserve whole-expression evaluation without metadata decomposition | | No stable file-version identity for Page Cache | Disable Parquet Page Cache to prevent stale-byte reads | | Incomplete Condition Cache coverage | Retain and recompute uncovered ranges | diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java index 765baf5393d1b2..5765f68fe7f6f5 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java +++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java @@ -2036,7 +2036,7 @@ public class Config extends ConfigBase { * Max data version of backends serialize block. */ @ConfField(mutable = false) - public static int max_be_exec_version = 11; + public static int max_be_exec_version = 12; /** * Min data version of backends serialize block. diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 024a036df11158..50ac62e871c66a 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -470,6 +470,19 @@ public long streamingSplitEstimate(ConnectorSession session, ConnectorTableHandl return fileCount >= threshold ? fileCount : -1; } + @Override + public boolean canServeMetadataOnlyCount(ConnectorSession session, ConnectorTableHandle handle, + Optional filter) { + IcebergTableHandle iceHandle = (IcebergTableHandle) handle; + if (iceHandle.isSystemTable() || filter.isPresent()) { + // Snapshot summaries describe the whole table and cannot prove a filtered row count. + return false; + } + Table table = resolveTable(session, iceHandle); + TableScan scan = buildScan(table, iceHandle, filter, session); + return getCountFromSnapshot(scan, session) >= 0; + } + /** * Lazy streaming split source (FIX-M3), mirroring legacy {@code IcebergScanNode.doStartSplit}: slice files at * a FIXED size ({@code file_split_size} if set, else {@code max_split_size} — NOT the per-table diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java index 2021e967017aff..7cd13afdb85425 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java @@ -93,8 +93,12 @@ public static ConnectorType fromIcebergType(Type icebergType, fieldIds.add(f.fieldId()); } return ConnectorType.structOf(names, types, nullable, comments).withChildrenFieldIds(fieldIds); + case VARIANT: + // Iceberg owns the Parquet Variant physical encoding, so expose an execution-only + // VariantV2 carrier without changing persisted Doris table metadata semantics. + return ConnectorType.of("VARIANT_COMPUTE_V2"); default: - // Any non-primitive iceberg type Doris cannot represent (VARIANT today; future non-primitive + // Any future non-primitive iceberg type Doris cannot represent // typeIds) degrades to UNSUPPORTED: the table still LOADS and only this column is // present-but-unqueryable. This DIVERGES from legacy fe-core (IcebergUtils.icebergTypeToDorisType // threw IllegalArgumentException at schema-load, failing the whole table). Graceful degradation diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java index 6594c2589d85c0..f25cdff9fd4a67 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java @@ -161,6 +161,7 @@ public IcebergWritePlanProvider(Map properties, @Override public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + validateWriteSchema(handle.getColumns(), handle.isWritesDataFiles()); IcebergTableHandle tableHandle = (IcebergTableHandle) handle.getTableHandle(); IcebergConnectorTransaction transaction = currentTransaction(session); @@ -213,7 +214,7 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl case MERGE: { TDataSink dataSink = new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK); dataSink.setIcebergMergeSink(buildMergeSink(table, tableHandle, rewritableDeletes, - handle.isRequireMergeCardinalityCheck(), schemaContext)); + handle.isWritesDataFiles(), handle.isRequireMergeCardinalityCheck(), schemaContext)); return new ConnectorSinkPlan(dataSink); } case REWRITE: { @@ -231,6 +232,27 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl } } + static void validateWriteSchema(List columns, boolean writesDataFiles) { + if (!writesDataFiles) { + return; + } + if (columns.stream().anyMatch(column -> containsVariant(column.getType()))) { + // Reject the whole data-file write: validating only selected columns would let an + // unchanged Variant target flow through a writer that cannot preserve its physical identity. + throw new DorisConnectorException( + "Iceberg VARIANT columns are read-only and cannot be written"); + } + } + + private static boolean containsVariant(ConnectorType type) { + String typeName = type.getTypeName(); + if ("VARIANT".equalsIgnoreCase(typeName) + || "VARIANT_COMPUTE_V2".equalsIgnoreCase(typeName)) { + return true; + } + return type.getChildren().stream().anyMatch(IcebergWritePlanProvider::containsVariant); + } + @Override public Optional> getWriteColumns(ConnectorSession session, ConnectorTableHandle tableHandle, Optional branchName) { @@ -535,7 +557,8 @@ private TIcebergDeleteSink buildDeleteSink(Table table, IcebergTableHandle table */ private TIcebergMergeSink buildMergeSink(Table table, IcebergTableHandle tableHandle, Map> rewritableDeletes, - boolean requireMergeCardinalityCheck, IcebergWriteSchemaContext schemaContext) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck, + IcebergWriteSchemaContext schemaContext) { TIcebergMergeSink tSink = new TIcebergMergeSink(); tSink.setDbName(tableHandle.getDbName()); tSink.setTbName(tableHandle.getTableName()); @@ -549,6 +572,7 @@ private TIcebergMergeSink buildMergeSink(Table table, IcebergTableHandle tableHa IcebergWriterHelper.shouldCollectColumnStats(schemaContext, schema)); // #66112: UPDATE and SQL MERGE share this sink, but only SQL MERGE has the one-source-row invariant. tSink.setRequireMergeCardinalityCheck(requireMergeCardinalityCheck); + tSink.setWritesDataFiles(writesDataFiles); PartitionSpec partitionSpec = schemaContext.getPartitionSpec(); if (partitionSpec.isPartitioned()) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 426f89c7d427b6..43bd3fd0d64c28 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -1459,6 +1459,22 @@ public void countPushdownFollowsTheSnapshotPin() { Assertions.assertEquals(10L, pinned.get(0).getPushDownRowCount()); } + @Test + public void metadataOnlyCountCapabilityUsesSnapshotSummary() { + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + table.newAppend().appendFile(dataFile( + table.spec(), "s3://b/db/t1/f1.parquet", 1000, null, null)).commit(); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider( + Collections.emptyMap(), opsReturning(table)); + ConnectorSession session = new FakeScanSession("UTC", Collections.emptyMap()); + + Assertions.assertTrue(provider.canServeMetadataOnlyCount( + session, new IcebergTableHandle("db1", "t1"), Optional.empty())); + Assertions.assertFalse(provider.canServeMetadataOnlyCount( + session, IcebergTableHandle.forSystemTable( + "db1", "t1", "snapshots", -1L, null, -1L), Optional.empty())); + } + @Test public void getScanNodePropertiesUnderPinEmitsFullPinnedSchemaDict() throws Exception { // T07 Option A: under a time-travel pin the field-id dict is built from the FULL pinned schema (covering diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java index 97f995ecf60eba..41c7e44f2e2a14 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java @@ -87,9 +87,9 @@ public void flagIndependentPrimitivesMatchLegacy() { @Test public void unknownAndV3TypesDegradeToUnsupportedByDesign() { - // WHY (user decision 2026-07-13, DV-051): iceberg types Doris cannot represent — the v3 primitives - // TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN and the non-primitive VARIANT — must map to - // UNSUPPORTED WITHOUT throwing, so the table still loads and only the exotic column is + // WHY (user decision 2026-07-13, DV-051): iceberg primitive types Doris cannot represent — + // TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN — must map to UNSUPPORTED WITHOUT throwing, + // so the table still loads and only the exotic column is // present-but-unqueryable. This deliberately DIVERGES from legacy fe-core, which threw // IllegalArgumentException("Cannot transform unknown type") at schema-load and failed the whole table. // This test PINS the graceful-degradation choice: MUTATION making either default arm throw -> red, @@ -100,9 +100,7 @@ public void unknownAndV3TypesDegradeToUnsupportedByDesign() { Assertions.assertEquals("UNSUPPORTED", mapOff(Types.GeometryType.crs84()).getTypeName()); Assertions.assertEquals("UNSUPPORTED", mapOff(Types.GeographyType.crs84()).getTypeName()); Assertions.assertEquals("UNSUPPORTED", mapOff(Types.UnknownType.get()).getTypeName()); - // VARIANT is NOT a primitive (falls to the nested-switch default); legacy mapped it to UNSUPPORTED - // too, so this stays parity while the primitives above are the intentional divergence. - Assertions.assertEquals("UNSUPPORTED", mapOff(Types.VariantType.get()).getTypeName()); + Assertions.assertEquals("VARIANT_COMPUTE_V2", mapOff(Types.VariantType.get()).getTypeName()); // The mapping flags do not rescue an unrepresentable type (both arms are flag-independent). Assertions.assertEquals("UNSUPPORTED", mapOn(Types.GeometryType.crs84()).getTypeName()); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java index cc431d2814e8c9..efca95e8e0902a 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java @@ -104,6 +104,19 @@ public class IcebergWritePlanProviderTest { private static final Map NON_REST_PROPS = Collections.singletonMap("iceberg.catalog.type", "hadoop"); + @Test + public void rejectsVariantDataWritesButAllowsDeleteOnlyMerge() { + ConnectorColumn nestedVariant = new ConnectorColumn("payload", + ConnectorType.structOf(Collections.singletonList("nested"), + Collections.singletonList(ConnectorType.of("VARIANT"))), + null, true, null); + Assertions.assertThrows(DorisConnectorException.class, + () -> IcebergWritePlanProvider.validateWriteSchema( + Collections.singletonList(nestedVariant), true)); + Assertions.assertDoesNotThrow(() -> IcebergWritePlanProvider.validateWriteSchema( + Collections.singletonList(nestedVariant), false)); + } + private static InMemoryCatalog freshCatalog() { InMemoryCatalog catalog = new InMemoryCatalog(); catalog.initialize("test", Collections.emptyMap()); diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java index 3987325ae56949..dc7adcfa07d42e 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java @@ -100,6 +100,14 @@ default boolean isRequireMergeCardinalityCheck() { return false; } + /** + * Whether this write can emit data files. A delete-only MERGE returns false so a connector may + * allow position-delete output even when the table has read-only column types. + */ + default boolean isWritesDataFiles() { + return true; + } + /** * The named table branch this write targets ({@code INSERT INTO t@branch(name)}), or * {@link Optional#empty()} when the write goes to the table's default ref. Threaded from the diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java index d1e5569b3fec90..2a5caa2ffcaa33 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java @@ -352,6 +352,17 @@ default long streamingSplitEstimate( return -1; } + /** + * Whether this connector can answer the current table-level COUNT(*) without decoding data files. + * The default is false; connectors may use snapshot metadata to prove the stronger condition. + */ + default boolean canServeMetadataOnlyCount( + ConnectorSession session, + ConnectorTableHandle handle, + Optional filter) { + return false; + } + /** * Builds a lazy {@link ConnectorSplitSource} for streaming split generation. Called once, on a * background task, only when {@link #streamingSplitEstimate} returned a non-negative value. The diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index cfe2fc607999af..0a6afdaa5bc5b4 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -18,6 +18,8 @@ package org.apache.doris.connector.spi; import org.apache.doris.connector.spi.handle.ConnectorColumnHandle; +import org.apache.doris.connector.spi.handle.ConnectorWriteHandle; +import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.spi.write.ConnectorWritePlanProvider; import org.junit.jupiter.api.Assertions; @@ -71,9 +73,8 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException Assertions.assertNotNull(in, "missing connector plugin API version resource"); version.load(in); } - // ConnectorWritePlanProvider and ConnectorColumnHandle gained public default methods in this - // surface revision. A plugin built against major 3 must be refused rather than silently run - // against an expanded contract it did not compile against. + // API 4 covers both the SPI namespace migration and the expanded column/write contracts. + // Older plugins must be rejected instead of failing later on missing types or methods. Assertions.assertEquals("4.0", version.getProperty("api.version")); } @@ -83,6 +84,8 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException ConnectorContext.class, Connector.class, ConnectorColumnHandle.class, + ConnectorScanPlanProvider.class, + ConnectorWriteHandle.class, ConnectorWritePlanProvider.class, org.apache.doris.extension.spi.Plugin.class, org.apache.doris.extension.spi.PluginFactory.class, diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt index 7e2854758a24e1..14174dbe23aa78 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt @@ -45,6 +45,38 @@ org.apache.doris.connector.spi.ConnectorProvider#validatePropertiesForUpdate(jav org.apache.doris.connector.spi.handle.ConnectorColumnHandle#equals(java.lang.Object):boolean org.apache.doris.connector.spi.handle.ConnectorColumnHandle#hashCode():int org.apache.doris.connector.spi.handle.ConnectorColumnHandle#withProjectedFieldIds(java.util.Set):org.apache.doris.connector.spi.handle.ConnectorColumnHandle +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getBranchName():java.util.Optional +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getColumns():java.util.List +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getSortInfo():org.apache.doris.thrift.TSortInfo +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getStaticPartitionSpec():java.util.Map +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getTableHandle():org.apache.doris.connector.spi.handle.ConnectorTableHandle +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getWriteOperation():org.apache.doris.connector.spi.handle.WriteOperation +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isOverwrite():boolean +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isRequireMergeCardinalityCheck():boolean +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isWritesDataFiles():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#adjustFileCompressType(org.apache.doris.thrift.TFileCompressType):org.apache.doris.thrift.TFileCompressType +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#appendExplainInfo(java.lang.StringBuilder,java.lang.String,java.util.Map):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#canServeMetadataOnlyCount(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#classifyColumn(java.lang.String):org.apache.doris.connector.spi.scan.ConnectorColumnCategory +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#collectScanProfiles(org.apache.doris.connector.spi.ConnectorSession):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getDeleteFiles(org.apache.doris.thrift.TTableFormatFileDesc):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getMustReadColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):java.util.Set +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getScanNodeProperties(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional):java.util.Map +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getScanNodePropertiesResult(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional):org.apache.doris.connector.spi.scan.ScanNodePropertiesResult +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#ignorePartitionPruneShortCircuit():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#planScan(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.scan.ConnectorScanRequest):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#planScanForPartitionBatch(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.scan.ConnectorScanRequest,java.util.List):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#populateScanLevelParams(org.apache.doris.thrift.TFileScanRangeParams,java.util.Map):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#releaseReadTransaction(java.lang.String):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#scannedPartitionCount(java.util.List):java.util.OptionalLong +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#streamingSplitEstimate(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional,boolean):long +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#streamSplits(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional,long):org.apache.doris.connector.spi.scan.ConnectorSplitSource +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsBatchScan(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsFileCache():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableIncrementalRead(java.lang.String):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableOptions(java.lang.String):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableTimeTravel():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsTableSample():boolean org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#appendExplainInfo(java.lang.StringBuilder,java.lang.String,org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorWriteHandle):void org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#getSyntheticWriteColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):java.util.List org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#getWriteColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional):java.util.Optional diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java index 2fb365e9e4db41..b378ee5f426fa2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java @@ -199,7 +199,9 @@ private StorageBindings initStorageAdapters() { throw new IllegalStateException( "Duplicate storage type: " + a.getType()); }, LinkedHashMap::new)); - local = new StorageBindings(ordered, byType); + // Consumers share the published map without locking, so prevent caller-specific + // mutations from changing the catalog-wide snapshot after publication. + local = new StorageBindings(ordered, Collections.unmodifiableMap(byType)); this.storageBindings = local; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java index a81f27720c10dc..30909a05c14c19 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java @@ -318,6 +318,11 @@ private static Type convertScalarType(String typeName, int precision, int scale) return ScalarType.createVarbinaryType(ScalarType.MAX_VARBINARY_LENGTH); case "JSONB": return ScalarType.createType("JSON"); + case "VARIANT_COMPUTE_V2": + // This carrier is execution-only: connector schemas use it for native external + // Variant encodings, while persisted Doris table metadata keeps regular Variant rules. + return new org.apache.doris.catalog.VariantType( + new ArrayList<>(), 0, false, 10000, 0, false, 0L, 64, false, true); case "UNSUPPORTED": return Type.UNSUPPORTED; default: diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index ec13b7e04b8a10..3437b453bf7573 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -26,9 +26,15 @@ import org.apache.doris.analysis.TableScanParams; import org.apache.doris.analysis.ToSqlParams; import org.apache.doris.analysis.TupleDescriptor; +import org.apache.doris.catalog.ArrayType; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.MapType; +import org.apache.doris.catalog.StructType; import org.apache.doris.catalog.TableIf; +import org.apache.doris.catalog.Type; +import org.apache.doris.catalog.VariantType; +import org.apache.doris.common.Config; import org.apache.doris.common.UserException; import org.apache.doris.common.profile.RuntimeProfile; import org.apache.doris.common.profile.SummaryProfile; @@ -124,6 +130,7 @@ * */ public class PluginDrivenScanNode extends FileQueryScanNode { + private static final int SUPPORT_ICEBERG_VARIANT_EXEC_VERSION = 12; private static final Logger LOG = LogManager.getLogger(PluginDrivenScanNode.class); @@ -192,6 +199,77 @@ public PluginDrivenScanNode(PlanNodeId id, TupleDescriptor desc, this.currentHandle = tableHandle; } + @Override + protected void doInitialize() throws UserException { + super.doInitialize(); + // Compatibility must inspect the snapshot-specific handle: latest metadata may answer + // COUNT(*) while an older time-travel snapshot still requires a Variant data scan. + pinMvccSnapshot(); + checkVariantBackendCompatibilityForCurrentScan(backendPolicy.getBackends()); + } + + void checkVariantBackendCompatibilityForCurrentScan(Iterable backends) + throws UserException { + boolean metadataCountProven = false; + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); + if (isTableLevelCountStarPushdown() && conjuncts.isEmpty() && scanProvider != null) { + metadataCountProven = onPluginClassLoader(scanProvider, + () -> canServeMetadataOnlyCount(scanProvider, connectorSession, currentHandle)); + } + checkVariantBackendCompatibility( + !metadataCountProven && projectsComputeVariant(desc), backends); + } + + static boolean canServeMetadataOnlyCount(ConnectorScanPlanProvider scanProvider, + ConnectorSession session, ConnectorTableHandle handle) { + return scanProvider.canServeMetadataOnlyCount(session, handle, Optional.empty()); + } + + static boolean projectsComputeVariant(TupleDescriptor tuple) { + // Nested-column pruning updates the effective slot type but deliberately keeps the original + // Column metadata; compatibility must follow the payload this scan actually projects. + return tuple.getSlots().stream().anyMatch(slot -> containsComputeVariant(slot.getType())); + } + + private static boolean containsComputeVariant(Type type) { + if (type instanceof VariantType) { + return ((VariantType) type).isComputeV2(); + } + if (type instanceof ArrayType) { + return containsComputeVariant(((ArrayType) type).getItemType()); + } + if (type instanceof MapType) { + MapType map = (MapType) type; + return containsComputeVariant(map.getKeyType()) || containsComputeVariant(map.getValueType()); + } + if (type instanceof StructType) { + return ((StructType) type).getFields().stream() + .anyMatch(field -> containsComputeVariant(field.getType())); + } + return false; + } + + static void checkVariantBackendCompatibility(boolean projectsVariant, Iterable backends) + throws UserException { + if (!projectsVariant) { + return; + } + if (Config.be_exec_version < SUPPORT_ICEBERG_VARIANT_EXEC_VERSION) { + // The query-wide execution version covers every eligible backend, including ordinary + // rolling-upgrade nodes that are not marked as cloud smooth-upgrade sources. + throw new UserException("Iceberg Variant requires backend execution version " + + SUPPORT_ICEBERG_VARIANT_EXEC_VERSION + " or newer during rolling upgrade"); + } + for (Backend backend : backends) { + if (backend.isSmoothUpgradeSrc()) { + // Old backends cannot distinguish the logical Variant from its physical carrier, + // so scheduling this projection there could corrupt the result shape. + throw new UserException("Iceberg Variant is unavailable while backend " + + backend.getId() + " is a smooth upgrade source"); + } + } + } + // Lazily resolves this node's ConnectorMetadata through the per-statement funnel and caches it, so the // per-method resolvers below share one instance for the statement instead of rebuilding it each time. private ConnectorMetadata metadata() { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index 5af8be9c147bbf..5a9d2dee752b01 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -555,7 +555,7 @@ public PlanFragment visitPhysicalExternalRowLevelDeleteSink( // TIcebergDeleteSink dialect. No output-expr / materialized-name loop is needed: the row id reaches // BE as the __DORIS_ICEBERG_ROWID_COL__ block column (a real hidden column), and viceberg_delete_sink // resolves it by block-name, not by output-expr name. - rootFragment.setSink(buildPluginRowLevelDmlSink(deleteSink, WriteOperation.DELETE, false)); + rootFragment.setSink(buildPluginRowLevelDmlSink(deleteSink, WriteOperation.DELETE, false, false)); return rootFragment; } @@ -592,7 +592,7 @@ public PlanFragment visitPhysicalExternalRowLevelMergeSink( // SQL MERGE INTO carries the cardinality requirement onto the write handle; UPDATE shares this // sink dialect but has no such rule, so it threads false (see PhysicalExternalRowLevelMergeSink). rootFragment.setSink(buildPluginRowLevelDmlSink(mergeSink, WriteOperation.MERGE, - mergeSink.isRequireMergeCardinalityCheck())); + mergeSink.isWritesDataFiles(), mergeSink.isRequireMergeCardinalityCheck())); return rootFragment; } @@ -609,7 +609,7 @@ public PlanFragment visitPhysicalExternalRowLevelMergeSink( */ private PluginDrivenTableSink buildPluginRowLevelDmlSink( PhysicalBaseExternalTableSink sink, WriteOperation writeOperation, - boolean requireMergeCardinalityCheck) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck) { PluginDrivenExternalTable targetTable = (PluginDrivenExternalTable) sink.getTargetTable(); PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) targetTable.getCatalog(); @@ -652,7 +652,8 @@ private PluginDrivenTableSink buildPluginRowLevelDmlSink( // writeSortInfo == null: a row-level DML has no engine-resolved write sort (MERGE's sort lives in the // connector's TIcebergMergeSink.sort_fields, DELETE is unsorted). return new PluginDrivenTableSink(targetTable, writePlanProvider, connSession, - providerTableHandle, connectorColumns, null, writeOperation, requireMergeCardinalityCheck); + providerTableHandle, connectorColumns, null, writeOperation, + writesDataFiles, requireMergeCardinalityCheck); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java index c00d9e7fd619a6..7aa7e357229a0c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java @@ -39,6 +39,7 @@ public Rule build() { sink.getTargetTable(), sink.getCols(), sink.getOutputExprs(), + sink.isWritesDataFiles(), sink.isRequireMergeCardinalityCheck(), Optional.empty(), sink.getLogicalProperties(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java index 0633637b3377aa..915ef5f7eba26f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java @@ -137,6 +137,15 @@ public Void visitSlotReference(SlotReference slotReference, CollectorContext con path, context.bottomFilter, ColumnAccessPathType.DATA)); return null; } + if (dataType instanceof VariantType) { + // A root Variant consumer must dominate any predicate-only leaf path. Otherwise the + // scanner can legally project a shredded leaf that cannot serve the root expression. + int slotId = slotReference.getExprId().asInt(); + slotToAccessPaths.put(slotId, new CollectAccessPathResult( + ImmutableList.of(slotReference.getName()), + context.bottomFilter, ColumnAccessPathType.DATA)); + return null; + } if (dataType instanceof NestedColumnPrunable) { context.accessPathBuilder.addPrefix(slotReference.getName().toLowerCase()); ImmutableList path = Utils.fastToImmutableList(context.accessPathBuilder.accessPath); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java index 7e7674d20586c7..c9cd9d9046fcd4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java @@ -25,6 +25,7 @@ import org.apache.doris.nereids.trees.expressions.Expression; import org.apache.doris.nereids.trees.expressions.NamedExpression; import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.SlotReference; import org.apache.doris.nereids.trees.expressions.functions.Function; import org.apache.doris.nereids.trees.expressions.functions.generator.Explode; import org.apache.doris.nereids.trees.expressions.functions.generator.ExplodeMap; @@ -249,6 +250,12 @@ public Void visitLogicalProject(LogicalProject project, Statemen List outerPath = outerSlotAccessPath.getPath(); List replaceSlotNamePath = new ArrayList<>(); replaceSlotNamePath.add(innerSlot.getName()); + if (outerPath.size() == 1 && innerSlot instanceof SlotReference + && ((SlotReference) innerSlot).hasSubColPath()) { + // A whole access to a derived subcolumn slot is whole only relative to that + // slot; preserve its physical leaf path when propagating to the scan slot. + replaceSlotNamePath.addAll(((SlotReference) innerSlot).getSubPath()); + } replaceSlotNamePath.addAll(outerPath.subList(1, outerPath.size())); allSlotToAccessPaths.put( innerSlot.getExprId().asInt(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java index 6ae6ef2b1abf9d..3faf0d581a5c72 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java @@ -724,6 +724,11 @@ public void setAccessByPath(List path, int accessIndex, ColumnAccessPath // Any other sub-path on a string column means full data is needed. accessAll = true; return; + } else if (type.isVariantType()) { + // Variant object keys stay in the serialized access path. Keeping the terminal type + // here lets BE project a shredded leaf without inventing static schema fields. + accessAll = true; + return; } else if (isRoot) { children.get(path.get(accessIndex).toLowerCase()).setAccessByPath(path, accessIndex + 1, pathType); return; diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java index 5e3b601df33958..ca8a3b5ba4c7cd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java @@ -64,6 +64,7 @@ import org.apache.doris.nereids.types.MapType; import org.apache.doris.nereids.types.NestedColumnPrunable; import org.apache.doris.nereids.types.StructType; +import org.apache.doris.nereids.types.VariantType; import org.apache.doris.nereids.util.MoreFieldsThread; import com.google.common.collect.ImmutableCollection; @@ -669,6 +670,10 @@ private void replaceAccessPathToFieldId(List originPath, int index, Data break; } } + } else if (type instanceof VariantType) { + // Variant object keys are data, not Iceberg schema field IDs. Replacing them with + // the root ID destroys the physical shredding path before it reaches the scanner. + return; } else { originPath.set(index, String.valueOf(column.getUniqueId())); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java index 853d2275f467cd..fb98fca5cfe7ac 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java @@ -411,6 +411,8 @@ LogicalPlan buildMergePlan(ConnectContext ctx, ExternalTable icebergTable) { icebergTable, ConnectorWriteSchemaUtils.pinAndGet(ctx, icebergTable), outputExprs, + matchedClauses.stream().anyMatch(clause -> !clause.isDelete()) + || !notMatchedClauses.isEmpty(), true, Optional.empty(), Optional.empty(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java index 59686f0012171c..041375512ca7e7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java @@ -138,6 +138,7 @@ LogicalPlan buildMergePlan(ConnectContext ctx, LogicalPlan logicalQuery, icebergTable, writeColumns, outputExprs, + true, false, Optional.empty(), Optional.empty(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java index d2e55779ef5b48..017bfaccdb2666 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java @@ -45,6 +45,8 @@ public class LogicalExternalRowLevelMergeSink extends L implements Sink, PropagateFuncDeps { private final ExternalDatabase database; private final ExternalTable targetTable; + // Delete-only MERGE emits position deletes but never invokes the data-file writer. + private final boolean writesDataFiles; // True for SQL MERGE INTO, false for UPDATE. MERGE must reject a target row matched by more than one // source row (SQL cardinality rule), which the BE sink can only do when the plan keeps the merge // distribution; UPDATE has no such rule. Read by RequestPropertyDeriver (which otherwise drops the @@ -67,12 +69,29 @@ public LogicalExternalRowLevelMergeSink(ExternalDatabase database, Optional groupExpression, Optional logicalProperties, CHILD_TYPE child) { + this(database, targetTable, cols, outputExprs, true, requireMergeCardinalityCheck, + groupExpression, logicalProperties, child); + } + + /** + * Constructor that records whether the merge writes replacement data files. + */ + public LogicalExternalRowLevelMergeSink(ExternalDatabase database, + ExternalTable targetTable, + List cols, + List outputExprs, + boolean writesDataFiles, + boolean requireMergeCardinalityCheck, + Optional groupExpression, + Optional logicalProperties, + CHILD_TYPE child) { super(PlanType.LOGICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK, outputExprs, groupExpression, logicalProperties, cols, child); this.database = Objects.requireNonNull(database, "database != null in LogicalExternalRowLevelMergeSink"); this.targetTable = Objects.requireNonNull(targetTable, "targetTable != null in LogicalExternalRowLevelMergeSink"); + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } @@ -81,19 +100,19 @@ public Plan withChildAndUpdateOutput(Plan child) { .map(NamedExpression.class::cast) .collect(ImmutableList.toImmutableList()); return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, output, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child); + writesDataFiles, requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child); } @Override public Plan withChildren(List children) { Preconditions.checkArgument(children.size() == 1, "LogicalExternalRowLevelMergeSink only accepts one child"); return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), children.get(0)); + writesDataFiles, requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), children.get(0)); } public LogicalExternalRowLevelMergeSink withOutputExprs(List outputExprs) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child()); + writesDataFiles, requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child()); } public ExternalDatabase getDatabase() { @@ -104,6 +123,10 @@ public ExternalTable getTargetTable() { return targetTable; } + public boolean isWritesDataFiles() { + return writesDataFiles; + } + public boolean isRequireMergeCardinalityCheck() { return requireMergeCardinalityCheck; } @@ -123,12 +146,14 @@ public boolean equals(Object o) { return Objects.equals(database, that.database) && Objects.equals(targetTable, that.targetTable) && Objects.equals(cols, that.cols) + && writesDataFiles == that.writesDataFiles && requireMergeCardinalityCheck == that.requireMergeCardinalityCheck; } @Override public int hashCode() { - return Objects.hash(super.hashCode(), database, targetTable, cols, requireMergeCardinalityCheck); + return Objects.hash(super.hashCode(), database, targetTable, cols, writesDataFiles, + requireMergeCardinalityCheck); } @Override @@ -138,6 +163,7 @@ public String toString() { "database", database.getFullName(), "targetTable", targetTable.getName(), "cols", cols, + "writesDataFiles", writesDataFiles, "requireMergeCardinalityCheck", requireMergeCardinalityCheck); } @@ -149,13 +175,15 @@ public R accept(PlanVisitor visitor, C context) { @Override public Plan withGroupExpression(Optional groupExpression) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, Optional.of(getLogicalProperties()), child()); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, Optional.of(getLogicalProperties()), child()); } @Override public Plan withGroupExprLogicalPropChildren(Optional groupExpression, Optional logicalProperties, List children) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, logicalProperties, children.get(0)); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, logicalProperties, children.get(0)); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java index 9634c33e2458f6..4b27e865c44c06 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java @@ -59,6 +59,7 @@ */ public class PhysicalExternalRowLevelMergeSink extends PhysicalBaseExternalTableSink { + private final boolean writesDataFiles; // True for SQL MERGE INTO, false for UPDATE; see LogicalExternalRowLevelMergeSink. private final boolean requireMergeCardinalityCheck; @@ -73,10 +74,27 @@ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, Optional groupExpression, LogicalProperties logicalProperties, CHILD_TYPE child) { - this(database, targetTable, cols, outputExprs, requireMergeCardinalityCheck, + this(database, targetTable, cols, outputExprs, true, requireMergeCardinalityCheck, groupExpression, logicalProperties, PhysicalProperties.GATHER, null, child); } + /** + * Constructor that records whether the merge writes replacement data files. + */ + public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, + ExternalTable targetTable, + List cols, + List outputExprs, + boolean writesDataFiles, + boolean requireMergeCardinalityCheck, + Optional groupExpression, + LogicalProperties logicalProperties, + CHILD_TYPE child) { + this(database, targetTable, cols, outputExprs, writesDataFiles, + requireMergeCardinalityCheck, groupExpression, logicalProperties, + PhysicalProperties.GATHER, null, child); + } + /** * Constructor */ @@ -90,11 +108,34 @@ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, PhysicalProperties physicalProperties, Statistics statistics, CHILD_TYPE child) { + this(database, targetTable, cols, outputExprs, true, requireMergeCardinalityCheck, + groupExpression, logicalProperties, physicalProperties, statistics, child); + } + + /** + * Constructor with explicit data-file and physical-property settings. + */ + public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, + ExternalTable targetTable, + List cols, + List outputExprs, + boolean writesDataFiles, + boolean requireMergeCardinalityCheck, + Optional groupExpression, + LogicalProperties logicalProperties, + PhysicalProperties physicalProperties, + Statistics statistics, + CHILD_TYPE child) { super(PlanType.PHYSICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK, database, targetTable, cols, outputExprs, groupExpression, logicalProperties, physicalProperties, statistics, child); + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } + public boolean isWritesDataFiles() { + return writesDataFiles; + } + public boolean isRequireMergeCardinalityCheck() { return requireMergeCardinalityCheck; } @@ -103,7 +144,7 @@ public boolean isRequireMergeCardinalityCheck() { public Plan withChildren(List children) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, - cols, outputExprs, requireMergeCardinalityCheck, groupExpression, + cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), physicalProperties, statistics, children.get(0)); } @@ -116,7 +157,8 @@ public R accept(PlanVisitor visitor, C context) { public Plan withGroupExpression(Optional groupExpression) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), child()); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, getLogicalProperties(), child()); } @Override @@ -124,13 +166,14 @@ public Plan withGroupExprLogicalPropChildren(Optional groupExpr Optional logicalProperties, List children) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, logicalProperties.get(), children.get(0)); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, logicalProperties.get(), children.get(0)); } @Override public PhysicalPlan withPhysicalPropertiesAndStats(PhysicalProperties physicalProperties, Statistics statistics) { return new PhysicalExternalRowLevelMergeSink<>( - database, targetTable, cols, outputExprs, requireMergeCardinalityCheck, + database, targetTable, cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), physicalProperties, statistics, child()); } @@ -146,12 +189,13 @@ public boolean equals(Object o) { return false; } PhysicalExternalRowLevelMergeSink that = (PhysicalExternalRowLevelMergeSink) o; - return requireMergeCardinalityCheck == that.requireMergeCardinalityCheck; + return writesDataFiles == that.writesDataFiles + && requireMergeCardinalityCheck == that.requireMergeCardinalityCheck; } @Override public int hashCode() { - return Objects.hash(super.hashCode(), requireMergeCardinalityCheck); + return Objects.hash(super.hashCode(), writesDataFiles, requireMergeCardinalityCheck); } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java index e9e539ea018e2f..b821728e9f195e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java @@ -18,6 +18,7 @@ package org.apache.doris.planner; import org.apache.doris.common.AnalysisException; +import org.apache.doris.common.Config; import org.apache.doris.connector.spi.ConnectorColumn; import org.apache.doris.connector.spi.ConnectorSession; import org.apache.doris.connector.spi.handle.ConnectorTableHandle; @@ -51,6 +52,7 @@ * specific {@code T*TableSink} dialect lives entirely inside the connector.

*/ public class PluginDrivenTableSink extends BaseExternalTableDataSink { + private static final int SUPPORT_ICEBERG_VARIANT_EXEC_VERSION = 12; private final PluginDrivenExternalTable targetTable; // Plan-provider mode (W5): the connector builds its own opaque TDataSink via planWrite(). @@ -70,6 +72,7 @@ public class PluginDrivenTableSink extends BaseExternalTableDataSink { // the INSERT TIcebergTableSink. Threaded onto the write handle so planWrite's buildWriteContext // reads it via ConnectorWriteHandle.getWriteOperation(). private final WriteOperation writeOperation; + private final boolean writesDataFiles; // SQL MERGE INTO only: the statement must reject a target row matched by more than one source row. // Carried from PhysicalExternalRowLevelMergeSink onto the write handle so the connector can stamp the // enforcement flag onto its BE sink; false for UPDATE and for every non-row-level write. @@ -121,6 +124,18 @@ public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, ConnectorWritePlanProvider writePlanProvider, ConnectorSession connectorSession, ConnectorTableHandle tableHandle, List connectorColumns, TSortInfo writeSortInfo, WriteOperation writeOperation, boolean requireMergeCardinalityCheck) { + this(targetTable, writePlanProvider, connectorSession, tableHandle, connectorColumns, + writeSortInfo, writeOperation, true, requireMergeCardinalityCheck); + } + + /** + * Plan-provider mode with explicit data-file and merge-cardinality requirements. + */ + public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, + ConnectorWritePlanProvider writePlanProvider, ConnectorSession connectorSession, + ConnectorTableHandle tableHandle, List connectorColumns, + TSortInfo writeSortInfo, WriteOperation writeOperation, boolean writesDataFiles, + boolean requireMergeCardinalityCheck) { super(); this.targetTable = targetTable; this.writePlanProvider = writePlanProvider; @@ -129,6 +144,7 @@ public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, this.connectorColumns = connectorColumns; this.writeSortInfo = writeSortInfo; this.writeOperation = writeOperation == null ? WriteOperation.INSERT : writeOperation; + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } @@ -162,7 +178,7 @@ public String getExplainString(String prefix, TExplainLevel explainLevel) { // EXPLAIN), so the connector derives the detail from the write handle. ConnectorWriteHandle handle = new PluginDrivenWriteHandle( tableHandle, connectorColumns, false, Collections.emptyMap(), null, Optional.empty(), - writeOperation, requireMergeCardinalityCheck); + writeOperation, writesDataFiles, requireMergeCardinalityCheck); writePlanProvider.appendExplainInfo(sb, prefix, connectorSession, handle); return sb.toString(); } @@ -178,6 +194,13 @@ public String getExplainString(String prefix, TExplainLevel explainLevel) { @Override public void bindDataSink(Optional insertCtx) throws AnalysisException { + if (writeOperation == WriteOperation.MERGE && !writesDataFiles + && Config.be_exec_version < SUPPORT_ICEBERG_VARIANT_EXEC_VERSION) { + // Older BEs ignore writes_data_files and instantiate the omitted data writer, so reject + // the all-or-nothing query before a Variant schema reaches any rolling-upgrade backend. + throw new AnalysisException("Delete-only Iceberg MERGE with Variant is unavailable " + + "during rolling upgrade"); + } boolean overwrite = false; Map writeContext = Collections.emptyMap(); Optional branchName = Optional.empty(); @@ -189,7 +212,7 @@ public void bindDataSink(Optional insertCtx) } ConnectorWriteHandle handle = new PluginDrivenWriteHandle( tableHandle, connectorColumns, overwrite, writeContext, writeSortInfo, branchName, - writeOperation, requireMergeCardinalityCheck); + writeOperation, writesDataFiles, requireMergeCardinalityCheck); ConnectorSinkPlan sinkPlan = writePlanProvider.planWrite(connectorSession, handle); this.tDataSink = sinkPlan.getDataSink(); } @@ -210,12 +233,13 @@ private static final class PluginDrivenWriteHandle implements ConnectorWriteHand private final TSortInfo sortInfo; private final Optional branchName; private final WriteOperation writeOperation; + private final boolean writesDataFiles; private final boolean requireMergeCardinalityCheck; private PluginDrivenWriteHandle(ConnectorTableHandle tableHandle, List columns, boolean overwrite, Map writeContext, TSortInfo sortInfo, Optional branchName, WriteOperation writeOperation, - boolean requireMergeCardinalityCheck) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck) { this.tableHandle = tableHandle; this.columns = columns; this.overwrite = overwrite; @@ -223,9 +247,15 @@ private PluginDrivenWriteHandle(ConnectorTableHandle tableHandle, List { + initializationStarted.countDown(); + awaitInitialization(allowInitialization); + return Collections.emptyMap(); + }); + + ExecutorService executor = Executors.newSingleThreadExecutor(); + AtomicReference> readerResult = new AtomicReference<>(); + Thread concurrentReader = new Thread( + () -> readerResult.set(catalogProperty.getStorageAdaptersMap())); + try { + Future> initializer = + executor.submit(catalogProperty::getStorageAdaptersMap); + Assert.assertTrue(initializationStarted.await(5, TimeUnit.SECONDS)); + + concurrentReader.start(); + Assert.assertTrue(waitUntilBlockedOrTerminated(concurrentReader, 5, TimeUnit.SECONDS)); + Assert.assertEquals("The reader must block until initialization publishes the completed map", + Thread.State.BLOCKED, concurrentReader.getState()); + + allowInitialization.countDown(); + Map initialized = initializer.get(5, TimeUnit.SECONDS); + concurrentReader.join(TimeUnit.SECONDS.toMillis(5)); + Assert.assertFalse(concurrentReader.isAlive()); + Assert.assertSame(initialized, readerResult.get()); + } finally { + allowInitialization.countDown(); + concurrentReader.interrupt(); + executor.shutdownNow(); + } + } + + @Test + public void testStorageAdaptersCacheIsImmutable() { + CatalogProperty catalogProperty = new CatalogProperty( + null, Collections.singletonMap("fs.defaultFS", "hdfs://test-ns")); + catalogProperty.setPluginDerivedStorageDefaultsSupplier(Collections::emptyMap); + + Map storageAdapters = catalogProperty.getStorageAdaptersMap(); + Assert.assertThrows(UnsupportedOperationException.class, storageAdapters::clear); + } + + private static boolean waitUntilBlockedOrTerminated(Thread thread, long timeout, TimeUnit timeUnit) { + long deadline = System.nanoTime() + timeUnit.toNanos(timeout); + while (thread.isAlive() && thread.getState() != Thread.State.BLOCKED + && System.nanoTime() < deadline) { + Thread.yield(); + } + return !thread.isAlive() || thread.getState() == Thread.State.BLOCKED; + } + + private static void awaitInitialization(CountDownLatch allowInitialization) { + try { + if (!allowInitialization.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("Timed out waiting to continue storage adapter initialization"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("Interrupted while initializing storage adapters", e); + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java index 272e0f7557ec7e..f92b93df99dacf 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java @@ -126,6 +126,13 @@ void testUnsupportedTypeConversion() { Assertions.assertTrue(back.isUnsupported()); } + @Test + void testComputeVariantCarrierConversion() { + Type type = ConnectorColumnConverter.convertType(ConnectorType.of("VARIANT_COMPUTE_V2")); + Assertions.assertTrue(type instanceof org.apache.doris.catalog.VariantType); + Assertions.assertTrue(((org.apache.doris.catalog.VariantType) type).isComputeV2()); + } + @Test void testUnknownTypeDefaultsToUnsupported() { ConnectorType ct = ConnectorType.of("GEOMETRY", -1, -1); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java new file mode 100644 index 00000000000000..e25cd39b7ef699 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java @@ -0,0 +1,91 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.scan; + +import org.apache.doris.common.Config; +import org.apache.doris.common.UserException; +import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.handle.ConnectorTableHandle; +import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; +import org.apache.doris.system.Backend; + +import org.junit.Assert; +import org.junit.Test; + +import java.util.Collections; +import java.util.Optional; + +/** Tests the mixed-version safety gate for plugin-driven Variant scans. */ +public class PluginDrivenScanNodeCompatibilityTest { + + private static final int VARIANT_EXEC_VERSION = 12; + + @Test + public void computeVariantRejectsSmoothUpgradeSourceBackend() { + Backend backend = new Backend(7L, "127.0.0.1", 9050); + backend.setSmoothUpgradeSrc(true); + + UserException exception = Assert.assertThrows(UserException.class, + () -> PluginDrivenScanNode.checkVariantBackendCompatibility( + true, Collections.singletonList(backend))); + Assert.assertTrue(exception.getMessage().contains("backend 7")); + } + + @Test + public void compatibilityCheckIgnoresScansWithoutComputeVariant() throws UserException { + Backend backend = new Backend(7L, "127.0.0.1", 9050); + backend.setSmoothUpgradeSrc(true); + + PluginDrivenScanNode.checkVariantBackendCompatibility( + false, Collections.singletonList(backend)); + } + + @Test + public void computeVariantRejectsOldQueryWideExecutionVersion() { + int original = Config.be_exec_version; + try { + Config.be_exec_version = VARIANT_EXEC_VERSION - 1; + Backend backend = new Backend(8L, "127.0.0.1", 9050); + + UserException exception = Assert.assertThrows(UserException.class, + () -> PluginDrivenScanNode.checkVariantBackendCompatibility( + true, Collections.singletonList(backend))); + Assert.assertTrue(exception.getMessage().contains("execution version")); + } finally { + Config.be_exec_version = original; + } + } + + @Test + public void metadataCountCapabilityUsesPinnedHandle() { + ConnectorSession session = org.mockito.Mockito.mock(ConnectorSession.class); + ConnectorTableHandle latest = new ConnectorTableHandle() { }; + ConnectorTableHandle pinned = new ConnectorTableHandle() { }; + ConnectorScanPlanProvider provider = + org.mockito.Mockito.mock(ConnectorScanPlanProvider.class); + org.mockito.Mockito.doAnswer(invocation -> invocation.getArgument(1) == latest) + .when(provider).canServeMetadataOnlyCount(org.mockito.Mockito.same(session), + org.mockito.Mockito.any(ConnectorTableHandle.class), + org.mockito.Mockito.eq(Optional.empty())); + + Assert.assertTrue(PluginDrivenScanNode.canServeMetadataOnlyCount( + provider, session, latest)); + Assert.assertFalse(PluginDrivenScanNode.canServeMetadataOnlyCount( + provider, session, pinned)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java index 3c92606e4e3fcc..ca9c59067c625b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java @@ -45,6 +45,9 @@ import org.apache.doris.nereids.types.DataType; import org.apache.doris.nereids.types.NestedColumnPrunable; import org.apache.doris.nereids.types.NullType; +import org.apache.doris.nereids.types.StructField; +import org.apache.doris.nereids.types.StructType; +import org.apache.doris.nereids.types.VariantType; import org.apache.doris.nereids.util.MemoPatternMatchSupported; import org.apache.doris.nereids.util.PlanChecker; import org.apache.doris.planner.OlapScanNode; @@ -1050,6 +1053,22 @@ public void testDataTypeAccessTree() { ); } + @Test + public void testDataTypeAccessTreeKeepsVariantTerminalPath() { + StructType type = new StructType(ImmutableList.of( + new StructField("payload", VariantType.INSTANCE, true, ""))); + SlotReference slot = new SlotReference("info", type); + DataTypeAccessTree tree = DataTypeAccessTree.ofRoot(slot, ColumnAccessPathType.DATA); + + tree.setAccessByPath(ImmutableList.of("info", "payload", "typed_col"), 0, + ColumnAccessPathType.DATA); + + DataType prunedType = tree.pruneDataType().get(); + Assertions.assertInstanceOf(StructType.class, prunedType); + Assertions.assertEquals(VariantType.INSTANCE, + ((StructType) prunedType).getFields().get(0).getDataType()); + } + @Test public void testWithVariant() throws Exception { connectContext.getSessionVariable().enableDecimal256 = true; diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java index 5268569e0b8861..cba617e96ed631 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java @@ -105,6 +105,25 @@ public void testVariantOrPredicatePaths() throws Exception { ); } + @Test + public void testWholeVariantOutputDominatesPredicateLeafProjection() throws Exception { + String rootOutputSql = "select v from variant_tbl where v['n'] > 1"; + assertAllAccessPathsContain( + rootOutputSql, + ImmutableList.of(path("v")), + ImmutableList.of() + ); + assertPredicateAccessPathsEqual(rootOutputSql, ImmutableList.of(path("v", "n"))); + + String predicateOnlySql = "select count(*) from variant_tbl where v['n'] > 1"; + assertAllAccessPathsContain( + predicateOnlySql, + ImmutableList.of(path("v", "n")), + ImmutableList.of(path("v")) + ); + assertPredicateAccessPathsEqual(predicateOnlySql, ImmutableList.of(path("v", "n"))); + } + @Test public void testVariantIfExpressionPaths() throws Exception { assertVariantSubColumnSlots( diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java index b91a71b62ce111..527f5d4582f966 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java @@ -18,6 +18,7 @@ package org.apache.doris.planner; import org.apache.doris.common.AnalysisException; +import org.apache.doris.common.Config; import org.apache.doris.connector.spi.ConnectorColumn; import org.apache.doris.connector.spi.ConnectorSession; import org.apache.doris.connector.spi.handle.ConnectorTableHandle; @@ -225,6 +226,40 @@ null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), Assert.assertEquals(WriteOperation.DELETE, provider.seenHandle.getWriteOperation()); } + @Test + public void bindDataSinkThreadsDeleteOnlyMergeToHandle() throws AnalysisException { + RecordingWritePlanProvider provider = new RecordingWritePlanProvider( + new ConnectorSinkPlan(new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK))); + PluginDrivenTableSink sink = new PluginDrivenTableSink( + null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), + null, WriteOperation.MERGE, false, true); + sink.bindDataSink(Optional.empty()); + + // Delete-only MERGE must bypass data-file validation while retaining cardinality enforcement. + Assert.assertFalse(provider.seenHandle.isWritesDataFiles()); + Assert.assertTrue(provider.seenHandle.isRequireMergeCardinalityCheck()); + } + + @Test + public void deleteOnlyMergeRejectsOldQueryWideExecutionVersion() { + int original = Config.be_exec_version; + try { + Config.be_exec_version = 11; + RecordingWritePlanProvider provider = new RecordingWritePlanProvider( + new ConnectorSinkPlan(new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK))); + PluginDrivenTableSink sink = new PluginDrivenTableSink( + null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), + null, WriteOperation.MERGE, false, true); + + AnalysisException exception = Assert.assertThrows(AnalysisException.class, + () -> sink.bindDataSink(Optional.empty())); + Assert.assertTrue(exception.getMessage().contains("rolling upgrade")); + Assert.assertNull(provider.seenHandle); + } finally { + Config.be_exec_version = original; + } + } + @Test public void getExplainStringThreadsWriteOperationToHandle() { // WHY: EXPLAIN of a post-flip MERGE/DELETE builds a (degraded) handle for appendExplainInfo; the diff --git a/gensrc/thrift/DataSinks.thrift b/gensrc/thrift/DataSinks.thrift index 9efe29a51eb421..7295da1a6874fd 100644 --- a/gensrc/thrift/DataSinks.thrift +++ b/gensrc/thrift/DataSinks.thrift @@ -539,6 +539,8 @@ struct TIcebergMergeSink { 14: optional bool collect_column_stats; // Unset preserves old-FE UPDATE behavior; execution version gates SQL MERGE validation. 15: optional bool require_merge_cardinality_check; + // Unset preserves old-FE UPDATE behavior, which always writes replacement data rows. + 16: optional bool writes_data_files; // delete side (position delete only) 20: optional TFileContent delete_type diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet new file mode 100644 index 00000000000000..f45415374f2f45 Binary files /dev/null and b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet differ diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json new file mode 100644 index 00000000000000..1f00c2ff298f7c --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json @@ -0,0 +1,74 @@ +{ + "format-version" : 3, + "table-uuid" : "270e9d8f-db6e-4522-9dce-2b9872f7159b", + "location" : "s3a://warehouse/wh/iceberg_variant_read_db/variant_page_pruning", + "last-sequence-number" : 1, + "last-updated-ms" : 1785245120440, + "last-column-id" : 2, + "current-schema-id" : 0, + "schemas" : [ { + "type" : "struct", + "schema-id" : 0, + "fields" : [ { + "id" : 1, + "name" : "id", + "required" : false, + "type" : "int" + }, { + "id" : 2, + "name" : "v", + "required" : false, + "type" : "variant" + } ] + } ], + "default-spec-id" : 0, + "partition-specs" : [ { + "spec-id" : 0, + "fields" : [ ] + } ], + "last-partition-id" : 999, + "default-sort-order-id" : 0, + "sort-orders" : [ { + "order-id" : 0, + "fields" : [ ] + } ], + "properties" : { + "write.format.default" : "parquet", + "write.parquet.shred-variants" : "true", + "schema.name-mapping.default" : "[ {\n \"field-id\" : 1,\n \"names\" : [ \"id\" ]\n}, {\n \"field-id\" : 2,\n \"names\" : [ \"v\" ]\n} ]" + }, + "current-snapshot-id" : 5420489606554005823, + "next-row-id" : 4096, + "refs" : { + "main" : { + "snapshot-id" : 5420489606554005823, + "type" : "branch" + } + }, + "snapshots" : [ { + "sequence-number" : 1, + "snapshot-id" : 5420489606554005823, + "timestamp-ms" : 1785245120440, + "summary" : { + "operation" : "append", + "added-data-files" : "1", + "added-records" : "4096", + "added-files-size" : "34528", + "total-records" : "4096", + "total-files-size" : "34528", + "total-data-files" : "1", + "total-delete-files" : "0" + }, + "manifest-list" : "s3a://warehouse/wh/iceberg_variant_read_db/variant_page_pruning/metadata/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro", + "schema-id" : 0, + "first-row-id" : 0, + "added-rows" : 4096 + } ], + "statistics" : [ ], + "partition-statistics" : [ ], + "snapshot-log" : [ { + "timestamp-ms" : 1785245120440, + "snapshot-id" : 5420489606554005823 + } ], + "metadata-log" : [ ] +} diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/b7958052-e154-425f-8850-f0011d0272c5-m0.avro b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/b7958052-e154-425f-8850-f0011d0272c5-m0.avro new file mode 100644 index 00000000000000..40c0bda5b56964 Binary files /dev/null and b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/b7958052-e154-425f-8850-f0011d0272c5-m0.avro differ diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro new file mode 100644 index 00000000000000..76465f90e3ae1e Binary files /dev/null and b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro differ diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_varbinary.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_varbinary.out index 65a20f0fb7cb7d..49f6fa445a6b02 100644 Binary files a/regression-test/data/external_table_p0/iceberg/test_iceberg_varbinary.out and b/regression-test/data/external_table_p0/iceberg/test_iceberg_varbinary.out differ diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out new file mode 100644 index 00000000000000..6abb509f237f85 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out @@ -0,0 +1,181 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !variant_evolution_initial_snapshot -- +1 initial 10 v1 + +-- !variant_evolution_initial_tag -- +1 initial 10 v1 + +-- !variant_evolution_initial_time -- +1 initial 10 v1 + +-- !variant_root_projection -- +1 false {"arr":[1,2],"n":10,"name":"alice","nested":{"city":"hz"},"ok":true,"ratio":1.5} +10 false {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 false {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +2 false {"arr":[3,4],"n":20,"name":"bob","nested":{"city":"sh"},"ok":false,"ratio":2.5} +3 false {"n":30,"name":"same","ok":true} +4 false null +5 true \N +6 false 42 +7 false root-string +8 false {"n":30,"name":"same","ok":true} +9 false {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_root_array_projection -- +1 false [] \N \N \N \N +2 false [null,1,{"x":2},[3,4],"tail"] null 1 2 4 +3 false [{"nested":[null,{"y":5}]}] {"nested":[null,{"y":5}]} \N \N \N +4 false null \N \N \N \N +5 true \N \N \N \N \N + +-- !variant_path_expressions -- +1 ALICE 11 1.5 true 3 hz +10 DAVE 51 5.5 false 15 sz +11 NULL 61 6.5 true 19 null +2 BOB 21 2.5 false 7 sh +9 CAROL 41 4.5 true 11 bj + +-- !variant_filter -- +11 null 60 +3 same 30 +8 same 30 +9 carol 40 + +-- !variant_cross_file_leaf_projection -- +1 10 +10 50 +11 60 +2 20 +3 30 +4 \N +5 \N +6 \N +7 \N +8 30 +9 40 + +-- !variant_multi_file_serial -- +2 20 \N 2 \N {"b":2,"shared":20,"z":200} +3 30 3 \N \N {"a":3,"shared":30,"z":300} +4 40 \N \N \N {"c":4,"shared":40} +5 50 \N 5 500 {"b":5,"new_field":{"k":500},"shared":50} + +-- !variant_multi_file_parallel -- +2 20 \N 2 \N {"b":2,"shared":20,"z":200} +3 30 3 \N \N {"a":3,"shared":30,"z":300} +4 40 \N \N \N {"c":4,"shared":40} +5 50 \N 5 500 {"b":5,"new_field":{"k":500},"shared":50} + +-- !variant_type_matrix -- +true -128 -32768 2147483647 -9223372036854775808 true true -1234567890.1234 1970-01-02 1970-01-01T00:00:01.234567 "YmluYXJ5" false + +-- !variant_multi_row_group_result -- +192 8000 8191 1554336 + +-- !variant_deletion_vector_current -- +2048 0 4094 4192256 + +-- !variant_deletion_vector_before_delete -- +4096 0 4095 8386560 + +-- !variant_equality_delete_current -- +1 10 keep-one {"label":"keep-one","n":10} +3 30 keep-three {"label":"keep-three","n":30} + +-- !variant_equality_delete_before_delete -- +1 10 keep-one {"label":"keep-one","n":10} +2 20 delete {"label":"delete","n":20} +3 30 keep-three {"label":"keep-three","n":30} + +-- !variant_implicit_filter -- +10 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +9 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_shredded_only_time_travel -- +1095 3001 4095 3885060 + +-- !variant_mixed_before_delete -- +1096 3001 5000 3890060 + +-- !variant_page_pruning_result -- +1094 3001 4094 + +-- !variant_aggregate -- +false 2 70 4 +true 5 170 4.17 + +-- !variant_join -- +10 fifty dave +2 twenty bob + +-- !variant_null_count_distinct -- +11 10 1 9 + +-- !variant_count_pushdown -- +10 11 + +-- !variant_canonical_group -- +{"n":30,"name":"same","ok":true} 2 + +-- !variant_nested_projection -- +1 first {"deep":{"name":"inside"},"x":11} {"kind":"open","score":101} {"enabled":true,"score":1001} +2 second \N null {"enabled":false,"score":2002} + +-- !variant_nested_filter -- +1 + +-- !variant_signed_selector -- +\N 41 + +-- !variant_nested_expressions -- +1 11 inside open \N true 1002 +2 \N \N \N 202 false 2003 + +-- !variant_evolution_renamed_snapshot -- +1 initial 10 v1 +2 renamed 20 v2 + +-- !variant_evolution_renamed_tag -- +1 initial 10 v1 +2 renamed 20 v2 + +-- !variant_evolution_added_reordered -- +1 initial \N v1 +2 renamed \N v2 +3 with-aux 300 v3 + +-- !variant_evolution_dropped -- +1 \N v1 +2 \N v2 +3 300 v3 +4 400 v4 + +-- !variant_evolution_drop_readd -- +1 \N \N \N v1 +2 \N \N \N v2 +3 300 \N \N v3 +4 400 \N \N v4 +5 500 readded 50 v5 + +-- !variant_delete_only_merge -- +0 + +-- !variant_position_delete_alignment -- +10 dave 50 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +9 carol 40 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_before_position_delete -- +10 dave 50 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 null 60 {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +9 carol 40 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_orc_missing_column -- +1 \N + +-- !variant_orc_count_star -- +1 + +-- !variant_mixed_format -- +1 {"format":"parquet"} +2 \N diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy index a5f091298a7a2d..c6d87e32693c45 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy @@ -22,6 +22,7 @@ suite("test_iceberg_varbinary", "p0,external") { logger.info("disable iceberg test.") return } + sql "SET ENABLE_VARIANT_V2=true" String catalog_name_no_mapping = "test_iceberg_no_mapping" String catalog_name_with_mapping = "test_iceberg_with_mapping" @@ -29,13 +30,18 @@ suite("test_iceberg_varbinary", "p0,external") { String rest_port = context.config.otherConfigs.get("iceberg_rest_uri_port") String minio_port = context.config.otherConfigs.get("iceberg_minio_port") String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + // A container-advertised REST URI may differ from the host port-forward address. + String restUri = context.config.otherConfigs.get("iceberg_rest_uri") + if (restUri == null) { + restUri = "http://${externalEnvIp}:${rest_port}" + } sql """drop catalog if exists ${catalog_name_no_mapping}""" sql """ CREATE CATALOG ${catalog_name_no_mapping} PROPERTIES ( 'type'='iceberg', 'iceberg.catalog.type'='rest', - 'uri' = 'http://${externalEnvIp}:${rest_port}', + 'uri' = '${restUri}', "s3.access_key" = "admin", "s3.secret_key" = "password", "s3.endpoint" = "http://${externalEnvIp}:${minio_port}", @@ -50,7 +56,7 @@ suite("test_iceberg_varbinary", "p0,external") { CREATE CATALOG ${catalog_name_with_mapping} PROPERTIES ( 'type'='iceberg', 'iceberg.catalog.type'='rest', - 'uri' = 'http://${externalEnvIp}:${rest_port}', + 'uri' = '${restUri}', "s3.access_key" = "admin", "s3.secret_key" = "password", "s3.endpoint" = "http://${externalEnvIp}:${minio_port}", @@ -166,10 +172,8 @@ suite("test_iceberg_varbinary", "p0,external") { qt_select23 """ select id from test_variant_repro; """ - test { - sql """ - select * from test_variant_repro; - """ - exception "UNSUPPORTED" - } + sql """set enable_file_scanner_v2=true""" + qt_select_variant """ + select id, cast(v as string) from test_variant_repro order by id; + """ } diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy new file mode 100644 index 00000000000000..ffc99236a1976f --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -0,0 +1,1057 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import java.util.regex.Matcher +import java.util.regex.Pattern +import com.amazonaws.auth.AWSStaticCredentialsProvider +import com.amazonaws.auth.BasicAWSCredentials +import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration +import com.amazonaws.services.s3.AmazonS3ClientBuilder +import org.apache.doris.regression.action.ProfileAction + +suite("test_iceberg_variant_read", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + sql "SET ENABLE_VARIANT_V2=true" + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String restUri = context.config.otherConfigs.get("iceberg_rest_uri") + if (restUri == null) { + restUri = "http://${externalEnvIp}:${restPort}" + } + String catalogName = "test_iceberg_variant_read" + String dbName = "iceberg_variant_read_db" + String fixtureKey = "doris-regression/iceberg-variant/iceberg_variant_shredded.parquet" + File shreddedFixture = new File(context.dataPath, "iceberg_variant_shredded.parquet") + File shreddedTableFixture = new File(context.dataPath, "iceberg_variant_shredded_table") + String shreddedMetadataName = + "00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json" + assertTrue(shreddedFixture.isFile(), "Missing shredded Variant Parquet fixture") + assertTrue(shreddedTableFixture.isDirectory(), "Missing shredded Variant table fixture") + def credentials = new BasicAWSCredentials("admin", "password") + def endpoint = new EndpointConfiguration( + "http://${externalEnvIp}:${minioPort}", "us-east-1") + def minioClient = AmazonS3ClientBuilder.standard() + .withEndpointConfiguration(endpoint) + .withPathStyleAccessEnabled(true) + .withCredentials(new AWSStaticCredentialsProvider(credentials)) + .build() + def executeCommand = { String command, int timeoutSeconds = 300 -> + StringBuilder stdout = new StringBuilder() + StringBuilder stderr = new StringBuilder() + def process = new ProcessBuilder("/bin/bash", "-c", command).start() + process.consumeProcessOutput(stdout, stderr) + process.waitForOrKill(timeoutSeconds * 1000) + assertEquals(0, process.exitValue(), + "Command failed\nstdout:\n${stdout}\nstderr:\n${stderr}") + return stdout.toString() + } + String dockerCommand = context.config.otherConfigs.get("externalDockerCommand") ?: "docker" + String sparkContainer = context.config.otherConfigs.get("icebergSparkContainer") + if (sparkContainer == null || sparkContainer.isEmpty()) { + String containers = executeCommand( + "${dockerCommand} ps --format '{{.ID}}\t{{.Names}}'", 30) + def matches = [] + containers.readLines().each { String line -> + String containerId = line.split(/\t/, 2)[0] + String probe = "${dockerCommand} exec ${containerId} bash -lc " + + "'test -f /mnt/SUCCESS && command -v spark-sql >/dev/null'" + try { + executeCommand(probe, 30) + matches.add(containerId) + } catch (Throwable ignored) { + // Only the Spark service contains the Iceberg writer dependencies. + } + } + assertEquals(1, matches.size(), "Expected exactly one usable Spark Iceberg container") + sparkContainer = matches[0] + } + def runInSparkContainer = { String command -> + executeCommand("${dockerCommand} exec ${sparkContainer} bash -lc '${command}'", 300) + } + + def latestSnapshotId = { String tableName -> + List> rows = spark_iceberg """ + SELECT snapshot_id + FROM demo.${dbName}.${tableName}.snapshots + ORDER BY committed_at DESC + LIMIT 1 + """ + assertEquals(1, rows.size()) + return rows[0][0].toString() + } + + spark_iceberg_multi """ + CREATE NAMESPACE IF NOT EXISTS demo.${dbName}; + DROP TABLE IF EXISTS demo.${dbName}.variant_values; + CREATE TABLE demo.${dbName}.variant_values ( + id INT, + v VARIANT + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.merge.mode'='merge-on-read' + ); + INSERT INTO demo.${dbName}.variant_values VALUES + (1, parse_json('{"name":"alice","n":10,"ratio":1.5,"ok":true,"arr":[1,2],"nested":{"city":"hz"}}')), + (2, parse_json('{"name":"bob","n":20,"ratio":2.5,"ok":false,"arr":[3,4],"nested":{"city":"sh"}}')), + (3, parse_json('{"name":"same","n":30,"ok":true}')), + (4, parse_json('null')), + (5, NULL), + (6, parse_json('42')), + (7, parse_json('"root-string"')); + ALTER TABLE demo.${dbName}.variant_values SET TBLPROPERTIES ( + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_values + VALUES + (8, parse_json('{"ok":true,"n":30,"name":"same"}')), + (9, parse_json('{"name":"carol","n":40,"ratio":4.5,"ok":true,"arr":[5,6],"nested":{"city":"bj"},"new_key":"new"}')), + (10, parse_json('{"name":"dave","n":50,"ratio":5.5,"ok":false,"arr":[7,8],"nested":{"city":"sz"}}')), + (11, parse_json('{"name":null,"n":60,"ratio":6.5,"ok":true,"arr":[9,10],"nested":{"city":null}}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_root_arrays; + CREATE TABLE demo.${dbName}.variant_root_arrays (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_root_arrays VALUES + (1, parse_json('[]')), + (2, parse_json('[null,1,{"x":2},[3,4],"tail"]')), + (3, parse_json('[{"nested":[null,{"y":5}]}]')), + (4, parse_json('null')), + (5, NULL); + + DROP TABLE IF EXISTS demo.${dbName}.variant_multi_file; + CREATE TABLE demo.${dbName}.variant_multi_file (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false' + ); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (1, parse_json('{"a":1,"shared":10}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES ( + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='1' + ); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (2, parse_json('{"b":2,"shared":20,"z":200}')); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (3, parse_json('{"z":300,"shared":30,"a":3}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES + ('write.parquet.shred-variants'='false'); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (4, parse_json('{"c":4,"shared":40}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES + ('write.parquet.shred-variants'='false'); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (5, parse_json('{"shared":50,"b":5,"new_field":{"k":500}}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_type_matrix; + CREATE TABLE demo.${dbName}.variant_type_matrix (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_type_matrix SELECT 1, to_variant_object(named_struct( + 'bool_value', true, + 'tiny_value', CAST(-128 AS TINYINT), + 'small_value', CAST(-32768 AS SMALLINT), + 'int_value', CAST(2147483647 AS INT), + 'big_value', CAST('-9223372036854775808' AS BIGINT), + 'float_value', CAST('NaN' AS FLOAT), + 'double_value', CAST('Infinity' AS DOUBLE), + 'decimal_value', CAST('-1234567890.1234' AS DECIMAL(20, 4)), + 'date_value', CAST('1970-01-02' AS DATE), + 'timestamp_value', TIMESTAMP'1970-01-01 00:00:01.234567', + 'binary_value', CAST('binary' AS BINARY), + 'null_value', CAST(NULL AS INT) + )); + + DROP TABLE IF EXISTS demo.${dbName}.variant_multi_row_group; + CREATE TABLE demo.${dbName}.variant_multi_row_group (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='100', + 'write.parquet.row-group-size-bytes'='4096' + ); + SET spark.sql.shuffle.partitions=1; + INSERT INTO demo.${dbName}.variant_multi_row_group + SELECT /*+ COALESCE(1) */ CAST(id AS INT), parse_json(concat( + '{"n":', id, ',"padding":"', repeat('x', 256), '"}')) + FROM range(0, 8192); + + DROP TABLE IF EXISTS demo.${dbName}.variant_deletion_vector; + CREATE TABLE demo.${dbName}.variant_deletion_vector (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='100', + 'write.delete.mode'='merge-on-read', + 'read.parquet.vectorization.enabled'='false', + 'write.parquet.row-group-size-bytes'='4096' + ); + INSERT INTO demo.${dbName}.variant_deletion_vector + SELECT /*+ COALESCE(1) */ CAST(id AS INT), parse_json(concat('{"n":', id, ',"keep":', + IF(id % 2 = 0, 'true', 'false'), '}')) + FROM range(0, 4096); + RESET spark.sql.shuffle.partitions; + + DROP TABLE IF EXISTS demo.${dbName}.variant_equality_delete; + CREATE TABLE demo.${dbName}.variant_equality_delete (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_equality_delete VALUES + (1, parse_json('{"n":10,"label":"keep-one"}')), + (2, parse_json('{"n":20,"label":"delete"}')), + (3, parse_json('{"n":30,"label":"keep-three"}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_page_pruning; + + DROP TABLE IF EXISTS demo.${dbName}.variant_nested; + CREATE TABLE demo.${dbName}.variant_nested ( + id INT, + info STRUCT, + events ARRAY, + attrs MAP + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_nested SELECT + 1, + named_struct('label', 'first', 'payload', parse_json('{"x":11,"deep":{"name":"inside"}}')), + array(parse_json('{"kind":"open","score":101}'), parse_json('2')), + map('primary', parse_json('{"enabled":true,"score":1001}')); + INSERT INTO demo.${dbName}.variant_nested SELECT + 2, + named_struct('label', 'second', 'payload', CAST(NULL AS VARIANT)), + array(parse_json('null'), parse_json('{"kind":"close","score":202}')), + map('primary', parse_json('{"enabled":false,"score":2002}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_signed_selector; + CREATE TABLE demo.${dbName}.variant_signed_selector ( + id INT, + v VARIANT + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_signed_selector + VALUES (1, parse_json('{"-1":41}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_evolution; + CREATE TABLE demo.${dbName}.variant_evolution ( + id INT, + payload VARIANT, + note STRING + ) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_evolution + VALUES (1, parse_json('{"stage":"initial","metric":10}'), 'v1'); + + DROP TABLE IF EXISTS demo.${dbName}.variant_write_guard; + CREATE TABLE demo.${dbName}.variant_write_guard (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_write_guard VALUES (1); + """ + + List> multiFileDataFiles = spark_iceberg """ + SELECT COUNT(*) FROM demo.${dbName}.variant_multi_file.files WHERE content = 0 + """ + assertEquals(1, multiFileDataFiles.size()) + assertTrue(Long.parseLong(multiFileDataFiles[0][0].toString()) > 1, + "The parallel Variant fixture must contain multiple data files") + + List> multiRowGroupFiles = spark_iceberg """ + SELECT COUNT(*) FROM demo.${dbName}.variant_multi_row_group.files WHERE content = 0 + """ + assertEquals(1, multiRowGroupFiles.size()) + assertEquals("1", multiRowGroupFiles[0][0].toString(), + "The multi-row-group fixture must contain exactly one data file") + + String equalityDeleteBaseSnapshot = latestSnapshotId("variant_equality_delete") + String equalityDeleteJava = ''' +import java.util.HashMap; +import java.util.Map; +import org.apache.iceberg.CatalogUtil; +import org.apache.iceberg.DeleteFile; +import org.apache.iceberg.PartitionSpec; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.catalog.Catalog; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.data.GenericRecord; +import org.apache.iceberg.data.Record; +import org.apache.iceberg.data.parquet.GenericParquetWriter; +import org.apache.iceberg.deletes.EqualityDeleteWriter; +import org.apache.iceberg.io.OutputFile; +import org.apache.iceberg.parquet.Parquet; + +public class AppendVariantEqualityDelete { + public static void main(String[] args) throws Exception { + Map props = new HashMap<>(); + props.put("type", "rest"); + props.put("uri", "http://rest:8181"); + props.put("warehouse", "s3://warehouse/wh/"); + props.put("io-impl", "org.apache.iceberg.aws.s3.S3FileIO"); + props.put("s3.endpoint", "http://minio:9000"); + props.put("s3.path-style-access", "true"); + props.put("s3.region", "us-east-1"); + Catalog catalog = CatalogUtil.buildIcebergCatalog("demo", props, null); + Table table = catalog.loadTable(TableIdentifier.of(args[0], args[1])); + Schema equalitySchema = table.schema().select("id"); + int fieldId = table.schema().findField("id").fieldId(); + OutputFile output = table.io().newOutputFile( + table.location() + "/data/variant-equality-delete-" + + System.currentTimeMillis() + ".parquet"); + EqualityDeleteWriter writer = Parquet.writeDeletes(output) + .forTable(table) + .rowSchema(equalitySchema) + .withSpec(PartitionSpec.unpartitioned()) + .createWriterFunc(GenericParquetWriter::create) + .equalityFieldIds(fieldId) + .overwrite() + .buildEqualityWriter(); + GenericRecord record = GenericRecord.create(equalitySchema); + record.setField("id", Integer.valueOf(args[2])); + writer.write(record); + writer.close(); + DeleteFile deleteFile = writer.toDeleteFile(); + table.newRowDelta().addDeletes(deleteFile).commit(); + } +} +''' + String encodedEqualityDeleteJava = + equalityDeleteJava.getBytes("UTF-8").encodeBase64().toString() + runInSparkContainer( + "echo ${encodedEqualityDeleteJava} | base64 -d " + + ">/tmp/AppendVariantEqualityDelete.java && " + + "javac -cp \"/opt/spark/jars/*\" " + + "/tmp/AppendVariantEqualityDelete.java && " + + "java -cp \"/tmp:/opt/spark/jars/*\" AppendVariantEqualityDelete " + + "${dbName} variant_equality_delete 2") + + String writeGuardSourceSnapshot = latestSnapshotId("variant_write_guard") + String deletionVectorBaseSnapshot = latestSnapshotId("variant_deletion_vector") + spark_iceberg """ + ALTER TABLE demo.${dbName}.variant_write_guard ADD COLUMN payload VARIANT + """ + spark_iceberg """ + DELETE FROM demo.${dbName}.variant_deletion_vector WHERE id % 2 = 1 + """ + List> deletionVectorFiles = spark_iceberg """ + SELECT file_format, content_offset, content_size_in_bytes + FROM demo.${dbName}.variant_deletion_vector.files + WHERE content = 1 + """ + assertFalse(deletionVectorFiles.isEmpty(), + "The Variant deletion fixture must expose a live delete file") + deletionVectorFiles.each { List deleteFile -> + assertEquals("PUFFIN", deleteFile[0].toString().toUpperCase(), + "The format-v3 Variant fixture must use PUFFIN deletion vectors") + assertTrue(Long.parseLong(deleteFile[1].toString()) >= 0, + "A PUFFIN deletion vector must expose its content offset") + assertTrue(Long.parseLong(deleteFile[2].toString()) > 0, + "A PUFFIN deletion vector must expose its content size") + } + + // Register a stable Iceberg metadata fixture so the page-pruning case always uses a + // standards-compliant shredded Variant file, independent of the Spark writer version. + minioClient.putObject("warehouse", fixtureKey, shreddedFixture) + shreddedTableFixture.eachFile { File fixtureFile -> + minioClient.putObject("warehouse", + "wh/${dbName}/variant_page_pruning/metadata/${fixtureFile.name}", fixtureFile) + } + minioClient.shutdown() + spark_iceberg """ + CALL demo.system.register_table( + table => '${dbName}.variant_page_pruning', + metadata_file => + 's3a://warehouse/wh/${dbName}/variant_page_pruning/metadata/${shreddedMetadataName}') + """ + String shreddedOnlySnapshot = latestSnapshotId("variant_page_pruning") + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_page_pruning SET TBLPROPERTIES ( + 'read.parquet.vectorization.enabled'='false', + 'write.delete.mode'='merge-on-read' + ); + INSERT INTO demo.${dbName}.variant_page_pruning VALUES + (5000, parse_json('{"n":5000,"padding":"mixed-unshredded"}')); + """ + String mixedBeforeDeleteSnapshot = latestSnapshotId("variant_page_pruning") + // One deletion vector targets the shredded fixture and another targets the appended + // unshredded file, forcing both physical states through the same scan and delete alignment. + spark_iceberg """ + DELETE FROM demo.${dbName}.variant_page_pruning WHERE id IN (4095, 5000) + """ + + sql """drop catalog if exists ${catalogName}""" + sql """ + CREATE CATALOG ${catalogName} PROPERTIES ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='${restUri}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0', + 'meta.cache.iceberg.schema.ttl-second'='0' + ) + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """set enable_file_scanner_v2=true""" + sql """set enable_profile=true""" + sql """set profile_level=2""" + + def profileAction = new ProfileAction(context) + def mergedProfile = { String profile -> + if (!profile.contains("MergedProfile:")) { + return profile + } + String merged = profile.substring(profile.indexOf("MergedProfile:")) + int end = merged.length() + ["DetailProfile(", "Execution Profile:", "Appendix:"].each { String sectionName -> + int sectionIndex = merged.indexOf(sectionName) + if (sectionIndex > 0) { + end = Math.min(end, sectionIndex) + } + } + return merged.substring(0, end) + } + def counterSum = { String profile, String counterName -> + Pattern pattern = Pattern.compile("(?m)^\\s*(?:-\\s*)?" + + Pattern.quote(counterName) + ":\\s+([^\\n]+)") + Matcher matcher = pattern.matcher(mergedProfile(profile)) + long sum = 0 + while (matcher.find()) { + String valueText = matcher.group(1) + // Merged counters may be human-readable; the parenthesized value is the exact sum. + Matcher exact = Pattern.compile("\\(([0-9,]+)\\)").matcher(valueText) + Matcher number = Pattern.compile("([0-9,]+)").matcher(valueText) + if (exact.find()) { + sum += Long.parseLong(exact.group(1).replace(",", "")) + } else if (number.find()) { + sum += Long.parseLong(number.group(1).replace(",", "")) + } + } + return sum + } + def profileInfoValues = { String profile, String infoName -> + Pattern pattern = Pattern.compile( + Pattern.quote(infoName) + ":\\s*\\[([^\\]]*)\\]") + Matcher matcher = pattern.matcher(profile) + if (!matcher.find()) { + return [] + } + return matcher.group(1).split(",").collect { String value -> value.trim() } + .findAll { String value -> !value.isEmpty() } + .collect { String value -> Long.parseLong(value.replace(",", "")) } + } + def getProfileByToken = { String token, List positiveCounters = [] -> + String lastProfile = profileAction.getProfileBySql(token, positiveCounters) + if (positiveCounters.every { String counter -> counterSum(lastProfile, counter) > 0 }) { + return lastProfile + } + return profileAction.waitProfile({ + lastProfile = profileAction.getProfileBySql(token, positiveCounters) + return positiveCounters.every { + String counter -> counterSum(lastProfile, counter) > 0 + } ? lastProfile : "" + }, [], "Completed profile with positive counters ${positiveCounters} for ${token}") + } + + String evolutionInitial = latestSnapshotId("variant_evolution") + sql """ALTER TABLE variant_evolution CREATE TAG variant_initial""" + + order_qt_variant_evolution_initial_snapshot """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionInitial} + ORDER BY id + """ + + order_qt_variant_evolution_initial_tag """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF 'variant_initial' + ORDER BY id + """ + + List> initialTime = sql """ + SELECT date_format(date_add(committed_at, interval 1 second), '%Y-%m-%d %H:%i:%s') + FROM variant_evolution\$snapshots + WHERE snapshot_id = ${evolutionInitial} + """ + assertEquals(1, initialTime.size()) + order_qt_variant_evolution_initial_time """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR TIME AS OF "${initialTime[0][0]}" + ORDER BY id + """ + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution RENAME COLUMN payload TO event; + INSERT INTO demo.${dbName}.variant_evolution + VALUES (2, parse_json('{"stage":"renamed","metric":20}'), 'v2'); + """ + String evolutionRenamed = latestSnapshotId("variant_evolution") + sql """ALTER TABLE variant_evolution CREATE TAG variant_renamed""" + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution ADD COLUMN aux VARIANT; + ALTER TABLE demo.${dbName}.variant_evolution ALTER COLUMN aux FIRST; + INSERT INTO demo.${dbName}.variant_evolution (id, event, note, aux) + VALUES (3, parse_json('{"stage":"with-aux","metric":30}'), 'v3', + parse_json('{"side":300}')); + """ + String evolutionWithAux = latestSnapshotId("variant_evolution") + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution RENAME COLUMN aux TO sidecar; + ALTER TABLE demo.${dbName}.variant_evolution DROP COLUMN event; + INSERT INTO demo.${dbName}.variant_evolution (id, note, sidecar) + VALUES (4, 'v4', parse_json('{"side":400}')); + """ + String evolutionDropped = latestSnapshotId("variant_evolution") + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution ADD COLUMN event VARIANT; + INSERT INTO demo.${dbName}.variant_evolution (id, note, sidecar, event) + VALUES (5, 'v5', parse_json('{"side":500}'), + parse_json('{"stage":"readded","metric":50}')); + + -- Write ORC before evolving the logical schema to Variant. This retains valid ORC files + -- in the snapshots while using Iceberg FileIO instead of Spark native ORC, whose optional + -- S3A implementation may not be installed. + DROP TABLE IF EXISTS demo.${dbName}.variant_orc; + CREATE TABLE demo.${dbName}.variant_orc (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='orc'); + INSERT INTO demo.${dbName}.variant_orc VALUES (1); + ALTER TABLE demo.${dbName}.variant_orc ADD COLUMN v VARIANT; + + DROP TABLE IF EXISTS demo.${dbName}.variant_mixed_format; + CREATE TABLE demo.${dbName}.variant_mixed_format (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='orc'); + INSERT INTO demo.${dbName}.variant_mixed_format VALUES (2); + ALTER TABLE demo.${dbName}.variant_mixed_format ADD COLUMN v VARIANT; + ALTER TABLE demo.${dbName}.variant_mixed_format SET TBLPROPERTIES + ('write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_mixed_format + VALUES (1, parse_json('{"format":"parquet"}')); + """ + String evolutionReadded = latestSnapshotId("variant_evolution") + + // Root projection covers objects, arrays, scalars, Variant null and SQL NULL. + order_qt_variant_root_projection """ + SELECT id, v IS NULL, CAST(v AS STRING) + FROM variant_values + ORDER BY id + """ + + order_qt_variant_root_array_projection """ + SELECT id, + v IS NULL, + CAST(v AS STRING), + CAST(v[1] AS STRING), + CAST(v[2] AS INT), + CAST(v[3]['x'] AS INT), + CAST(v[4][2] AS INT) + FROM variant_root_arrays + ORDER BY id + """ + + order_qt_variant_path_expressions """ + SELECT id, + UPPER(CAST(v['name'] AS STRING)), + CAST(v['n'] AS INT) + 1, + ROUND(CAST(v['ratio'] AS DOUBLE), 1), + CAST(v['ok'] AS BOOLEAN), + ARRAY_SUM(CAST(v['arr'] AS ARRAY)), + CAST(v['nested']['city'] AS STRING) + FROM variant_values + WHERE id IN (1, 2, 9, 10, 11) + ORDER BY id + """ + + order_qt_variant_filter """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT) + FROM variant_values + WHERE CAST(v['n'] AS INT) >= 20 + AND CAST(v['ok'] AS BOOLEAN) = true + ORDER BY id + """ + + // Keep the independent Spark writes on one scanner to exercise metadata dictionaries and + // complete Variant state transitions across file boundaries before batching. + sql "set parallel_pipeline_task_num=1" + sql "set max_file_scanners_concurrency=1" + order_qt_variant_cross_file_leaf_projection """ + SELECT id, CAST(v['n'] AS INT) + FROM variant_values + ORDER BY id + """ + + order_qt_variant_multi_file_serial """ + SELECT id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + sql "set parallel_pipeline_task_num=4" + sql "set max_file_scanners_concurrency=8" + sql "set min_file_scanners_concurrency=4" + order_qt_variant_multi_file_parallel """ + SELECT id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + String parallelScanToken = + "iceberg_variant_parallel_scan_" + UUID.randomUUID().toString() + List> parallelScanRows = sql """ + SELECT '${parallelScanToken}', id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + assertEquals(4, parallelScanRows.size(), + "The parallel Variant query must read rows from multiple data files") + String parallelScanProfile = profileAction.getProfileBySql( + parallelScanToken, ["PerScannerRowsRead"]) + if (profileInfoValues(parallelScanProfile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } <= 1) { + parallelScanProfile = profileAction.waitProfile({ + String profile = profileAction.getProfileBySql( + parallelScanToken, ["PerScannerRowsRead"]) + return profileInfoValues(profile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } > 1 ? profile : "" + }, [], "Completed parallel Variant profile with multiple non-empty scanners") + } + assertTrue(profileInfoValues(parallelScanProfile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } > 1, + "The parallel Variant query did not use multiple non-empty scanners") + sql "set min_file_scanners_concurrency=1" + + order_qt_variant_type_matrix """ + SELECT CAST(v['bool_value'] AS BOOLEAN), + CAST(v['tiny_value'] AS TINYINT), + CAST(v['small_value'] AS SMALLINT), + CAST(v['int_value'] AS INT), + CAST(v['big_value'] AS BIGINT), + ISNAN(CAST(v['float_value'] AS FLOAT)), + ISINF(CAST(v['double_value'] AS DOUBLE)), + CAST(v['decimal_value'] AS DECIMAL(20, 4)), + CAST(v['date_value'] AS DATE), + CAST(v['timestamp_value'] AS DATETIMEV2(6)), + CAST(v['binary_value'] AS STRING), + v['null_value'] IS NULL + FROM variant_type_matrix + """ + + String multiRowGroupColdToken = + "iceberg_variant_multi_row_group_cold_" + UUID.randomUUID().toString() + sql """ + SELECT '${multiRowGroupColdToken}', COUNT(*), MIN(id), MAX(id) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + String multiRowGroupColdProfile = getProfileByToken(multiRowGroupColdToken, + ["RowGroupsTotalNum", "VariantDirectLeafPathMisses", "VariantReconstructedRows", + "FilteredRowsByLazyRead"]).toString() + assertTrue(counterSum(multiRowGroupColdProfile, "RowGroupsTotalNum") > 1, + "The generated Variant file did not contain multiple Parquet row groups") + assertTrue(counterSum(multiRowGroupColdProfile, "VariantDirectLeafPathMisses") > 0, + "The unshredded scan did not record its direct-leaf fallback") + assertTrue(counterSum(multiRowGroupColdProfile, "VariantReconstructedRows") > 0, + "The unshredded scan did not reconstruct Variant rows") + assertTrue(counterSum(multiRowGroupColdProfile, "FilteredRowsByLazyRead") > 0, + "The unshredded Variant predicate did not defer non-predicate columns") + String multiRowGroupWarmToken = + "iceberg_variant_multi_row_group_warm_" + UUID.randomUUID().toString() + sql """ + SELECT '${multiRowGroupWarmToken}', COUNT(*), MIN(id), MAX(id) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + String multiRowGroupWarmProfile = getProfileByToken(multiRowGroupWarmToken, + ["VariantDirectLeafPathMisses"]).toString() + assertTrue(counterSum(multiRowGroupWarmProfile, "VariantDirectLeafPathMisses") > 0, + "The warm unshredded scan did not preserve its direct-leaf fallback") + qt_variant_multi_row_group_result """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + + qt_variant_deletion_vector_current """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_deletion_vector + WHERE v['n'] >= 0 + """ + qt_variant_deletion_vector_before_delete """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_deletion_vector FOR VERSION AS OF ${deletionVectorBaseSnapshot} + WHERE v['n'] >= 0 + """ + order_qt_variant_equality_delete_current """ + SELECT id, CAST(v['n'] AS INT), CAST(v['label'] AS STRING), CAST(v AS STRING) + FROM variant_equality_delete + WHERE v['n'] >= 0 + ORDER BY id + """ + order_qt_variant_equality_delete_before_delete """ + SELECT id, CAST(v['n'] AS INT), CAST(v['label'] AS STRING), CAST(v AS STRING) + FROM variant_equality_delete FOR VERSION AS OF ${equalityDeleteBaseSnapshot} + WHERE v['n'] >= 0 + ORDER BY id + """ + + // Keep the root Variant as output while the scalar comparison exercises the fallback path for + // the unshredded Spark files. + order_qt_variant_implicit_filter """ + SELECT id, CAST(v AS STRING) + FROM variant_values + WHERE v['n'] > 35 + ORDER BY id + """ + + qt_variant_shredded_only_time_travel """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_page_pruning FOR VERSION AS OF ${shreddedOnlySnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + qt_variant_mixed_before_delete """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_page_pruning FOR VERSION AS OF ${mixedBeforeDeleteSnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + + // The complete Variant is the only scanned output column outside the predicate. A positive + // lazy-read count therefore proves Variant output deferral rather than deferral of an id + // sibling, while the row relationship proves reconstruction happens after filtering. + String lazyVariantToken = + "iceberg_variant_lazy_materialization_" + UUID.randomUUID().toString() + List> lazyVariantRows = sql """ + SELECT '${lazyVariantToken}', CAST(v AS STRING) + FROM variant_page_pruning FOR VERSION AS OF ${shreddedOnlySnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + String lazyVariantProfile = getProfileByToken(lazyVariantToken, + ["VariantDirectLeafRows", "VariantReconstructedRows", + "FilteredRowsByLazyRead"]).toString() + long reconstructedVariantRows = + counterSum(lazyVariantProfile, "VariantReconstructedRows") + assertEquals((long) lazyVariantRows.size(), reconstructedVariantRows, + "Complete Variant reconstruction must be limited to selected output rows") + assertTrue(counterSum(lazyVariantProfile, "VariantDirectLeafRows") > + reconstructedVariantRows, + "Variant output was not deferred until after its shredded-leaf predicate") + assertTrue(counterSum(lazyVariantProfile, "FilteredRowsByLazyRead") > 0, + "The shredded predicate did not defer complete Variant output") + + // The query projects the complete Variant while its predicate reads the shredded typed leaf. + // The appended unshredded file must fall back independently in the same scan. + String pagePruningToken = "iceberg_variant_page_pruning_" + UUID.randomUUID().toString() + sql """ + SELECT '${pagePruningToken}', id, CAST(v AS STRING) + FROM variant_page_pruning + WHERE CAST(v['n'] AS INT) > 3000 + ORDER BY id + """ + String pagePruningProfile = getProfileByToken(pagePruningToken, + ["FilteredRowsByPage", "VariantLeafProjections", "VariantDirectLeafPathMisses", + "VariantDirectLeafRows", "VariantReconstructedRows"]).toString() + assertTrue(counterSum(pagePruningProfile, "FilteredRowsByPage") > 0, + "Shredded Variant typed_value did not filter any Parquet page") + // The predicate_access_paths contract keeps the typed leaf eager while the complete Variant + // root is read through the independent deferred-output projection. + assertTrue(counterSum(pagePruningProfile, "VariantLeafProjections") > 0, + "A root Variant output query did not retain its typed predicate leaf projection") + assertTrue(counterSum(pagePruningProfile, "VariantDirectLeafPathMisses") > 0, + "The mixed scan did not fall back for its unshredded Variant file") + assertTrue(counterSum(pagePruningProfile, "VariantDirectLeafRows") > 0, + "The mixed scan did not evaluate rows from the shredded typed leaf") + assertTrue(counterSum(pagePruningProfile, "VariantReconstructedRows") > 0, + "The mixed scan did not reconstruct complete Variant output") + String leafProjectionToken = + "iceberg_variant_leaf_projection_" + UUID.randomUUID().toString() + sql """ + SELECT '${leafProjectionToken}', COUNT(*) + FROM variant_page_pruning + WHERE CAST(v['n'] AS INT) > 3000 + """ + String leafProjectionProfile = getProfileByToken(leafProjectionToken, + ["VariantLeafProjections"]).toString() + assertTrue(counterSum(leafProjectionProfile, "VariantLeafProjections") > 0, + "Variant typed predicate did not retain a physical leaf projection") + qt_variant_page_pruning_result """ + SELECT COUNT(*), MIN(id), MAX(id) + FROM variant_page_pruning + WHERE CAST(v['n'] AS INT) > 3000 + """ + + // A later Variant metadata predicate must not prune away an earlier error-producing conjunct. + test { + sql """ + SELECT COUNT(*) + FROM variant_page_pruning + WHERE assert_true(id != 1, 'variant_metadata_error_barrier') + AND v['n'] > 5000 + """ + exception "variant_metadata_error_barrier" + } + + order_qt_variant_aggregate """ + SELECT CAST(v['ok'] AS BOOLEAN), + COUNT(*), + SUM(CAST(v['n'] AS INT)), + ROUND(AVG(CAST(v['ratio'] AS DOUBLE)), 2) + FROM variant_values + WHERE v['name'] IS NOT NULL + GROUP BY CAST(v['ok'] AS BOOLEAN) + ORDER BY 1 + """ + + order_qt_variant_join """ + WITH thresholds AS ( + SELECT 20 AS n, 'twenty' AS label + UNION ALL + SELECT 50 AS n, 'fifty' AS label + ) + SELECT t.id, d.label, CAST(t.v['name'] AS STRING) + FROM variant_values t + JOIN thresholds d ON CAST(t.v['n'] AS INT) = d.n + ORDER BY t.id + """ + + qt_variant_null_count_distinct """ + SELECT COUNT(*), COUNT(v), SUM(v IS NULL), COUNT(DISTINCT v) + FROM variant_values + """ + + qt_variant_count_pushdown """ + SELECT COUNT(v), COUNT(*) + FROM variant_values + """ + + order_qt_variant_canonical_group """ + SELECT CAST(v AS STRING), COUNT(*) + FROM variant_values + GROUP BY v + HAVING COUNT(*) > 1 + ORDER BY 1 + """ + + order_qt_variant_nested_projection """ + SELECT id, + info.label, + CAST(info.payload AS STRING), + CAST(events[1] AS STRING), + CAST(element_at(attrs, 'primary') AS STRING) + FROM variant_nested + ORDER BY id + """ + + // Spark may leave nested Variant values unshredded even when top-level shredding is enabled. + // Keep the external-table regression focused on correctness; mapper/reader unit tests use a + // physical typed_value fixture to verify nested leaf projection. + order_qt_variant_nested_filter """ + SELECT id + FROM variant_nested + WHERE CAST(info.payload['x'] AS INT) > 0 + ORDER BY id + """ + + // Signed integer selectors are array indexes, even when a shredded object has a key with the + // same serialized token. The ambiguous scanner path must retain enough state for both results. + order_qt_variant_signed_selector """ + SELECT CAST(v[-1] AS INT), CAST(v['-1'] AS INT) + FROM variant_signed_selector + """ + + order_qt_variant_nested_expressions """ + SELECT id, + CAST(info.payload['x'] AS INT), + CAST(info.payload['deep']['name'] AS STRING), + CAST(events[1]['kind'] AS STRING), + CAST(events[2]['score'] AS INT), + CAST(element_at(attrs, 'primary')['enabled'] AS BOOLEAN), + CAST(element_at(attrs, 'primary')['score'] AS INT) + 1 + FROM variant_nested + ORDER BY id + """ + + order_qt_variant_evolution_renamed_snapshot """ + SELECT id, CAST(event['stage'] AS STRING), CAST(event['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionRenamed} + ORDER BY id + """ + + order_qt_variant_evolution_renamed_tag """ + SELECT id, CAST(event['stage'] AS STRING), CAST(event['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF 'variant_renamed' + ORDER BY id + """ + + order_qt_variant_evolution_added_reordered """ + SELECT id, + CAST(event['stage'] AS STRING), + CAST(aux['side'] AS INT), + note + FROM variant_evolution FOR VERSION AS OF ${evolutionWithAux} + ORDER BY id + """ + + order_qt_variant_evolution_dropped """ + SELECT id, CAST(sidecar['side'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionDropped} + ORDER BY id + """ + + test { + sql """ + SELECT event + FROM variant_evolution FOR VERSION AS OF ${evolutionDropped} + """ + exception "event" + } + + order_qt_variant_evolution_drop_readd """ + SELECT id, + CAST(sidecar['side'] AS INT), + CAST(event['stage'] AS STRING), + CAST(event['metric'] AS INT), + note + FROM variant_evolution FOR VERSION AS OF ${evolutionReadded} + ORDER BY id + """ + + test { + sql """SELECT payload FROM variant_evolution""" + exception "payload" + } + + test { + sql """ + INSERT INTO variant_write_guard (id) + SELECT id + FROM variant_write_guard FOR VERSION AS OF ${writeGuardSourceSnapshot} + """ + exception "Iceberg VARIANT columns are read-only and cannot be written" + } + + // A delete-only MERGE emits only position deletes. It must remain available even though + // update/insert actions would route the unchanged Variant through the unsupported data writer. + String beforePositionDeleteSnapshot = latestSnapshotId("variant_values") + sql """ + MERGE INTO variant_values t + USING (SELECT 11 AS id) s + ON t.id = s.id + WHEN MATCHED THEN DELETE + """ + qt_variant_delete_only_merge "SELECT COUNT(*) FROM variant_values WHERE id = 11" + order_qt_variant_position_delete_alignment """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT), CAST(v AS STRING) + FROM variant_values + WHERE v['n'] >= 40 + ORDER BY id + """ + order_qt_variant_before_position_delete """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT), CAST(v AS STRING) + FROM variant_values FOR VERSION AS OF ${beforePositionDeleteSnapshot} + WHERE v['n'] >= 40 + ORDER BY id + """ + String positionDeleteToken = + "iceberg_variant_position_delete_" + UUID.randomUUID().toString() + sql """ + SELECT '${positionDeleteToken}', COUNT(*) + FROM variant_values + WHERE v['n'] >= 40 + """ + String positionDeleteProfile = getProfileByToken(positionDeleteToken, + ["VariantDirectLeafPathMisses", "VariantReconstructedRows"]).toString() + assertTrue(counterSum(positionDeleteProfile, "VariantDirectLeafPathMisses") > 0, + "Position-delete filtering did not preserve the unshredded Variant fallback") + assertTrue(counterSum(positionDeleteProfile, "VariantReconstructedRows") > 0, + "Position-delete filtering did not reconstruct its Variant rows") + + // Files written before the Variant field existed have no physical Variant payload. Schema + // evolution must synthesize NULL instead of rejecting their non-Parquet file format. + order_qt_variant_orc_missing_column """ + SELECT id, CAST(v AS STRING) FROM variant_orc ORDER BY id + """ + qt_variant_orc_count_star "SELECT COUNT(*) FROM variant_orc" + order_qt_variant_mixed_format """ + SELECT id, CAST(v AS STRING) FROM variant_mixed_format ORDER BY id + """ + + sql """set enable_file_scanner_v2=false""" + try { + test { + sql """SELECT CAST(v AS STRING) FROM variant_values ORDER BY id""" + exception "legacy file scanner does not support VARIANT" + } + } finally { + sql """set enable_file_scanner_v2=true""" + } +} diff --git a/regression-test/suites/variant_p0/variant_with_rowstore.groovy b/regression-test/suites/variant_p0/variant_with_rowstore.groovy index db83c8ae1158e4..f7bd05f4922f36 100644 --- a/regression-test/suites/variant_p0/variant_with_rowstore.groovy +++ b/regression-test/suites/variant_p0/variant_with_rowstore.groovy @@ -44,7 +44,9 @@ suite("regression_test_variant_rowstore", "variant_type"){ sql """insert into ${table_name} select * from (select -2, '{"a": 11245, "b" : [123, {"xx" : 1}], "c" : {"c" : 456, "d" : "null", "e" : 7.111}}' as json_str union all select -1, '{"a": 1123}' as json_str union all select *, '{"a" : 1234, "xxxx" : "kaana"}' as json_str from numbers("number" = "4096"))t order by 1 limit 4096 ;""" sql "sync" - qt_sql "select * from ${table_name} order by k limit 10" + // Row-store and column-store Variant readers may emit different insignificant JSON spacing. + // Normalize it so this suite continues to verify that both paths preserve the same values. + qt_sql "select k, replace(cast(v as string), ', ', ',') from ${table_name} order by k limit 10" table_name = "multi_var_rs" @@ -60,7 +62,8 @@ suite("regression_test_variant_rowstore", "variant_type"){ properties("replication_num" = "1", "disable_auto_compaction" = "false", "store_row_column" = "true"); """ sql """insert into ${table_name} select k, cast(v as string), cast(v as string) from var_rowstore""" - qt_sql "select * from ${table_name} order by k limit 10" + qt_sql """select k, replace(cast(v as string), ', ', ','), + replace(cast(v1 as string), ', ', ',') from ${table_name} order by k limit 10""" // Parse url def user = context.config.jdbcUser